(integration) tests are flaky

[turbohz]

While implementing https://github.com/Odonno/surrealdb-migrations/pull/56 I noticed that tests will pass sometimes, and then they will not.

When I knew a test should fail, but was passing, I could reliably fix (break again?) the test by restarting the surrealdb instances that are used in the integration tests.

This is probably due to reusing the database, which makes the tests not pure (previous state of the database can influence them).

Ideally each integration test should run in a fresh, uninitialized test database.

From my experience with other languages, they often have hooks to run stuff before/after a test suite or individual tests. I assume something similar can be done in Rust.

[turbohz]

This means, before-integration-tests.sh should not be required to be manually ran.

[turbohz]

I also noticed that sometimes the surrealdb processes crash, or stop for some reason.

[Odonno]

The 2 main goals with tests are:

• Tests should run in less than 10s, so parallelizing a maximum of tests is a priority. The ones that can't be run in parallel are the ones on branch feature. Might be a good idea to split in 2 test libraries. One in parallel and one in serial (one library per limited scope like branches). I can see that.
• Tests should run in CI. So having to launch a limited number of instances of surrealdb is good. And to mimic the workflow of the CI in your local computer, we need the same workflow.

While implementing https://github.com/Odonno/surrealdb-migrations/pull/56 I noticed that tests will pass sometimes, and then they will not.

Never noticed that. I suppose you have no idea to determine the cause? I think it can happen to have ns/db conflict between 2 tests but that should be like one in a zillion.

Ideally each integration test should run in a fresh, uninitialized test database.

Ideally yes. But then it will take a lot of time to run tests. Apart from the "branches" tests, tests are generally using the same surrealdb instance but in separate ns/db so there should never be any conflict. We still need a right cleanup for "branches" test.

From my experience with other languages, they often have hooks to run stuff before/after a test suite or individual tests. I assume something similar can be done in Rust. This means, before-integration-tests.sh should not be required to be manually ran.

Kind of like setupTest in jest or a beforeAll/afterAll in most languages. Yes, I can definitely see that. Not sure if it is possible at the moment. I mean, I did not found a solution for this.

I also noticed that sometimes the surrealdb processes crash, or stop for some reason.

I never experienced failed test or surrealdb process crash. Are you sure nothing else come in the way? Enough CPU? Enough RAM? I suppose you were not able to identify the cause.

[Odonno]

On another note, I am also interested in a migration to nextest. But that should be in its own PR.

[turbohz]

I see, I never considered that critical that the tests ran fast,

Ideally yes. But then it will take a lot of time to run tests. Apart from the "branches" tests, tests are generally using the same surrealdb instance but in separate ns/db so there should never be any conflict.

I fixed an issue where the integration test was using a db named test, instead of a random named one.

Could be just that one, it might well be, yeah.

I never experienced failed test or surrealdb process crash. Are you sure nothing else come in the way? Enough CPU? Enough RAM? I suppose you were not able to identify the cause.

I work on a beefy desktop PC. Since the two process used are put in the background, I could not catch what happened to them, I just got errors when the test tried to connect.

Could be an issue with my setup, or bugs in surrealdb related with the strict mode, which is what I was testing.

I'll close the issue for now, and reopen if I find out anything more concrete.

[Odonno]

Thank you for your input. It emerges with good idea like splitting test library in 2. Will create an issue for this.