Build a single unit test working with all backends.

[vadz]

It would be nice to have a single unit test which could be used to check that all, or some, tests pass with all the (configured) backends as typically this is what is needed when developing SOCI.

With CATCH support for tags, it should be possible to just run soci_test "[postgresql]" to run all the PostgreSQL tests, for example.

Some questions to consider:

• How to organize the tests for CATCH? We can't have multiple copies of the same test as the names would clash, but how else to do it?
• How to specify (non default) connection strings for more than one backend at a time? Simplest would probably be to use environment variables but this is not very user-friendly under Windows so having special options --backend-dsn=<string> options would probably be better.

[mloskot]

I asked the CATCH forum on Adding your own command line options, let's wait for some responses.

[OniDaito]

Not sure if this is helpful, but internally we went with GoogleTest and a series of scripts that pulled and ran docker containers with the required database files. I don't, at present, have a good way of including this as an option, though I'd like to.

[mloskot]

Technically, it's not difficult to implement what we need here.

I think, the question is rather how to do it in CATCH only, without pulling any extra thirdparty dependencies into SOCI. Otherwise, we could, for instance, get rapidjson on board and make the tests read all connection strings from a (presumably, generated) JSON file.

Replacing CATCH with Google Test is not an option, I'm afraid.

(BTW, do you mean docker containers to run SOCI-based client specifically? I'd be interested to learn more :))

[OniDaito]

I was surprised to see CATCH - it's not one I'd heard of but it doesn't really matter. The problem we found was testing against specific versions of all the databases without having to install them all and such. So I wrote in some scripts that are launched by CTest (which I think is a good thing to use because we are using CMake anyway - you get it for free). The scripts pull in a docker image, loaded with say, postgresql. The container gets fired up and the SOCI test program is run against it.

[mloskot]

@OniDaito testing against specific versions of all the databases without having to install them all is certainly what we've been looking for as well. Would you consider and be allowed to share your scripts?

[OniDaito]

Oh with pleasure! It's just I was chatting with the boss about how we package things. I'd overwritten a lot of the soci test directory to get it all working but on reflection, maybe it's better to have all the docker and script crap as a separate github project? I honestly don't know and although it works, it's messy.

That said, I think it's a good approach and just needs some more thought. What I might do is tidy them up and stick them on pastebin for you to take a peek. Bear in mind though, that I use CTest as the test harness. In fact, using ctest could be a cmake option which, if selected, could pull in docker stuff.

[OniDaito]

http://piratepad.net/Ta3mfJo7u9

Here is an example of the test for postgres. By running CTest all this magically happens :)

It's perhaps not easy to follow but here we go:

• Firstly, the cmakelists.txt in tests is modified so it moves files into the correct directory. It then runs the postgresql.sh
• this script pulls in the two dockerfiles, postgresql.dockerfile and wait.dockerfile
• These containers are created and fired up. Wait is there to halt testing until the postgresql service has actually started.
• The test binaries are run against the containers
• the script records the results, returning the final outcome from the tests (in this case, a pass or fail from googletest - could be any test platform)
• Ctest then carries on or quits if that script reports a fail

[vadz]

I'm not sure what is the real gain compared to the current system which seems to work fine and is more lightweight. Anyhow, whatever we do I'd just like to be able to continue running the tests without bothering with docker and all this, it can be convenient for people who don't have their own database but I need to be able run the tests against the same version of the database that we use in production [too].

[OniDaito]

As I see it, you need to install all the databases and have them running. Do you have a separate test machine, like a VM for that? That is probably the most elegant solution as VMs are fairly cheap to spin up and shut down. In fact, a Vagrant box would be ideal for that and much easier to setup than separate docker images. I'd propose a separate github project for that I think.

[vadz]

I just run test free databases (PostgreSQL, Firebird, SQLite and MySQL) locally, it's not really a problem. For Oracle and MS SQL I use remote servers. And I admit I don't test DB2 (but Travis does so I could too if I really wanted to).

Anyhow, this seems rather orthogonal to the topic of this ticket, which is to have a single test executable instead of multiple ones to be able to run the same test with all databases, however are they configured, at once. It could be nice to have Docker images/VMs/whatever containing everything needed for SOCI testing, but this is a completely separate topic, I don't really know how did we end up discussing this here.

[OniDaito]

Would you envision a single cpp file with set tests that is compiled for each plugin and run once with a separate connection string for each db

[vadz]

This isn't really that different from the current situation, is it?

Again, my motivation is to be able to run some particular test(s) for all the databases more easily. Currently I have to run several different tests with different connection strings to achieve this, it would be nice if I could run a single test (even if it still required using different command line arguments specifying all the connection strings it needs) instead.

[OniDaito]

I think thats why I was wondering about CTest - its one command to run all tests with, as you say, separate connection strings. A one liner to run them all but perhaps I've missed what you mean.

Im keen to figure it out because I spent a lot of time writing and automating the tests for our build so it's something I could easily drop in.

[vadz]

ctest can be used to run all the tests with the connection strings defined at cmake time, but I'm not sure how to use it to run just some of the tests (identified either by the CATCH test name or tag(s)) and/or use different connection strings?

[OniDaito]

I see your point a little better now. Is this something catch doesnt support; command line switch to take a test tag and just run that?

[mloskot]

On 24 April 2015 at 17:18, VZ notifications@github.com wrote:

I'm not sure what is the real gain compared to the current system which seems to work fine and is more lightweight. Anyhow, whatever we do I'd just like to be able to continue running the tests without bothering with docker and all this,

I sense misunderstanding. I'm not proposing any changes in the current CMake configuration or Travis CI setup.

it can be convenient for people who don't have their own database but

This is exactly the reason I'm interested in the docker-based solution. It seems, I could use it in my local development cycle to do early testing, before my changes are pushed pulled by Travis CI.

[OniDaito]

Yeah, testing is hard! :S I think the solution I came up with was not optimal but the idea of a local VM is a good one for local development and testing. But yes, selective control over the tests is something I'd like too! Just not sure how to do it yet.

[mloskot]

On 24 April 2015 at 17:48, VZ notifications@github.com wrote:

I just run test free databases (PostgreSQL, Firebird, SQLite and MySQL) locally, it's not really a problem.

I'd prefer to run databases on-demand

For Oracle and MS SQL I use remote servers. And I admit I don't test DB2 (but Travis does so I could too if I really wanted to).

including those heavy ones too, that's why I prefer generally on-demand solution.

Anyhow, this seems rather orthogonal to the topic of this ticket (...)

Yes, indeed. Testing based on docker is orthogonal to having single executable to run tests. Unless, I can't see the relation.

I don't really know how did we end up discussing this here.

It's my fault, because I hijacked it a bit asking about the docker.

Let's move the Docker discussion to #331

[mloskot]

On 26 April 2015 at 14:43, VZ notifications@github.com wrote:

ctest can be used to run all the tests with the connection strings defined at cmake time, but I'm not sure how to use it to run just some of the tests (identified either by the CATCH test name or tag(s)) and/or use different connection strings?

How about the test executable reading configuration from text file?

[mloskot]

On 27 April 2015 at 10:17, Benjamin Blundell notifications@github.com wrote:

I see your point a little better now. Is this something catch doesnt support; command line switch to take a test tag and just run that?

It can take tags and names of tests, etc. https://github.com/philsquared/Catch/blob/master/docs/command-line.md

But, AFAIK (see the linked thread above to the CATCH forum) currently, we don't know how to pass custom arguments like connection strings. That's why I'm contemplating a configuration file.

[vadz]

Personally I'd be quite happy with using environment variables, e.g. take the connection string value from SOCI_TEST_XXX_DSN if it is defined. Then we could have a single test executable which would run the tests for all the RDBMS by default and we could use it as soci_test [oracle] to only run Oracle tests (as we already can now) or soci_test "Truncation error" to run the given test for all the backends (as we can't easily do now) or, of course, any combination of those such as soci_test {[oracle],[postgresql]}"TruncationError".

[mloskot]

By the way, there is a way to parse custom arguments, see https://groups.google.com/d/msg/catch-forum/FwNEKaj2BHc/Ey_rDYjlyAkJ But, CATCH does not offer any means to make them available to the tests - we need our own singleton with such custom configuration, for instance, see https://groups.google.com/d/msg/catch-forum/FwNEKaj2BHc/24lIXVgImVcJ

I don't mind environment variables. It's just the command line arguments seem a better fit, though harder to implement.

[snikulov]

ctest -N - will give list of available tests. ctest -VV -R regex - will run tests according to regex for example

ctest -VV -R postgesql - will run only postgresql related tests.

Is it what requested?

[mloskot]

@snikulov Not really. ctest -R regex matches name used in CMakes's add_test command, AFAIK.

This idea here is about utilising CATCH capability to index/lookup tests by name, section, tag:

https://github.com/philsquared/Catch/blob/master/docs/test-cases-and-sections.md https://github.com/philsquared/Catch/blob/master/docs/command-line.md#listing-available-tests-tags-or-reporters

One of challenges here is how to pass SOCI specific arguments via the CATCH test trunner.

[snikulov]

@mloskot is it really useful for UT?

[mloskot]

It is not that useful for testing on CI builds. It is useful for deverloper, in edit code - build - test cycle. It has usability potential, so it is nice to have feature.