search

aiidateam / aiida-core

The official repository for the AiiDA code
https://aiida-core.readthedocs.io
Other
435 stars 188 forks source link

faster plugin tests #2648

Closed zhubonan closed 5 years ago

zhubonan commented 5 years ago

It is now convenient to run just pytest in the test folder using the fixture defined in aiida.manage.fixtures.fixture_manager. This creates a complete separate aiida profile and database. However, this process is relatively slow (~20 seconds) compared to the old verdi -p <test_profile> devel tests (no delay) method. This is becuase the postgres cluster needs to be created every time be migrated afterwards.

It would be better to allow this fixture_manager to reuse the existing test_profile to speed things up. During plugin development sometimes there can be just a small change made and it is desirable to get the test results faster.

zhubonan commented 5 years ago

Example here:

https://gitlab.com/bz1/aiida-castep/blob/7dd23de764d7f614ff7adb87b133e0b917a39c1c/aiida_castep/tests/dbtests/fixture.py

The test profile will be used if AIIDA_TEST_PROFILE environmental variable is set, otherwise fallback to creating new profiles/db cluster from scratch Not sure how to detect the backend after profile loaded. Only works for django now.

Speed up: 26 s vs 4 s for 8 tests @ltalirz @chrisjsewell

ltalirz commented 5 years ago

@zhubonan Thanks, I'll be looking into this. I'm a little surprised that your numbers with the fixtures are so high - I'm not so familiar with pytest so I will need to check they are set up properly for pytest - with unittest, using a special runner, the postgres cluster should be set up only once for all tests and destroyed afterwards. 20s for this seems way too long.

zhubonan commented 5 years ago

@ltalirz It is only the setup time that is a bit long, so it slows down the processing of fixing bugs (e.g the run test/fix/re-run cycles). Running 8 tests and 40 tests should take rought the same time.

Once the initail setup is complete, the tests actually run fairly fast.

ltalirz commented 5 years ago

ok, thanks. so, good to know that at least the fixtures are doing what they are supposed to.

I agree that there can always be a use case where you want to run just a few tests and you may want to avoid the overhead of creating the postgres cluster (which is never going to be zero). I'll give your implementation a try!