Implement test replay functionality

[theosotr]

Given one (or more) schema, cynthia should be able to re-execute a set of given AQL queries.

@StefanosChaliasos

[theosotr]

Structure of .cynthia:

• dbs: Directory of sqlite3 databases.
• schemas: Directory of temporary sql scripts used to initialize backends.
• projects: Directory of temporary projects used to run the queries.
• results: Directory of results
  • results/<db-name>
  • results/<db-name>/schema.sql: SQL script to setup db schema.
  • results/<db-name>/mismatches/
  • results/<db-name>/mismatches/<id>/
  • results/<db-name>/mismatches/<id>/query.aql: AQL query in human readable form
  • results/<db-name>/mismatches/<id>/query.aql.json: AQL query in JSON format
  • results/<db-name>/mismatches/<id>/<orm>_<backend>.out: Output of ORM orm on db backend.
  • results/<db-name>/matches/
  • results/<db-name>/matches/<id>/
  • results/<db-name>/matches/<id>/query.aql: AQL query in human readable form
  • results/<db-name>/matches/<id>/query.aql.json: AQL query in JSON format
  • results/<db-name>/matches/<id>/<orm>_<backend>.out: Output of ORM orm on db backend.
  • results/<db-name>/invalid/
  • results/<db-name>/invalid/<id>/
  • results/<db-name>/invalid/<id>/query.aql: AQL query in human readable form
  • results/<db-name>/invalid/<id>/query.aql.json: AQL query in JSON format
  • results/<db-name>/invalid/<id>/<orm>_<backend>.out: Output of ORM orm on db backend.

Now, only mismatches are kept into the file system. There should be a CLI flag --store-matches to save matches as well. By default, the replay command should only run the queries of mismatches. However, there should be an option --all to replay all tests (mismatches, matches, and invalid).

[theosotr]

I propose the following cynthia subcommands.

• [x] cynthia test: Generate AQL queries and test orms.

• [x] cynthia replay: Run the given AQL queries on the given schemas. If no options are specified, this command inspects the .cynthia directory.

• [x] cynthia generate: Generate random AQL queries and schemas. It does not translate queries into ORM queries.

• [x] cynthia run: It expects an AQL query, a schema, and an ORM, and prints the output of the ORM.

• [x] cynthia clean: Cleans up the .cynthia directory.

[StefanosChaliasos]

We also have the command cynthia select

[theosotr]

We also have the command cynthia select

I suggest it should be part of replay.

[StefanosChaliasos]

Nope

[theosotr]

cynthia replay --schema /script.sql --query /dir/query.aql (similar to select) cynthia replay (inspects .cynthia)

OK?

[StefanosChaliasos]

We need select command for testing purposes. If we remove it, we won't have any way to run some queries that we know if they should fail or succeed (e.g., examples/listing, examples/books). We could write tests, but I guess neither of us is willing to write them.

[StefanosChaliasos]

20