search

loculus-project / loculus

An open-source software package to power microbial genomic databases
https://loculus.org
GNU Affero General Public License v3.0
34 stars 1 forks source link

chore(backend): disable parallel test execution to avoid flaky tests #2710

Closed chaoran-chen closed 2 weeks ago

chaoran-chen commented 2 weeks ago

Maybe helps with #2708 (hopefully, let's see)

This reverts the parallel test execution that was introduced in #1914. Some tests expect an empty database, so parallel execution can very likely be a source of flakiness.

I will run backend tests five times: https://github.com/loculus-project/loculus/actions/runs/10716872022/job/29715313279?pr=2710

corneliusroemer commented 2 weeks ago

Ah sorry, makes sense to revert if there's a chance for flakiness. Not that it has happened in practice I think.

Isn't it dangerous to work on one database? Do you assume that every test cleans up after itself?

fengelniederhammer commented 2 weeks ago

Do you assume that every test cleans up after itself?

Yes, there is a clean up step after (or before) every test and thus every test assumes that it's running on a clean state.

corneliusroemer commented 2 weeks ago

Could we easily set up separate databases for each runner? It's not worth if it takes much work, but speeding up tests isn't completely without benefit!

fengelniederhammer commented 2 weeks ago

speeding up tests isn't completely without benefit!

You have to weigh it against initial setup and long-term maintenance. My feeling is that it's not worth the effort yet.

corneliusroemer commented 2 weeks ago

Sure! I just don't know how hard it is to give each runner its own db.

corneliusroemer commented 2 weeks ago

Sure! I just don't know how hard it is to give each runner its own db.

Note I've reworded the PR to not auto close the flakiness because I think the flakiness predates parallelity.

corneliusroemer commented 2 weeks ago

I'm merging this now already because I introduced the change and shouldn't have. No need to discuss further at this point and waste our time ;)