👌 IMPROVE failure logging

[chrisjsewell]

Log run information from mock code execution, and print to stdout on test fixture teardown.

For example:

        check_diff_output(res)

>       raise
E       RuntimeError: No active exception to reraise

tests/mock_code/test_diff.py:98: RuntimeError
------------------------------------------- Captured log setup -------------------------------------------
DEBUG    pgsu:__init__.py:143 Trying to connect via "psycopg2"...
---------------------------------------- Captured stdout teardown ----------------------------------------
AiiDA mock code logging:
2022-11-22 07:23:37.801306:diff: Init mock code
2022-11-22 07:23:37.802971:diff: Hashed paths: ['_aiidasubmit.sh', '_scheduler-stderr.txt', '_scheduler-stdout.txt', 'file1.txt', 'file2.txt', 'patch.diff']
2022-11-22 07:23:37.803233:diff: Cached directory exists: /Users/chrisjsewell/Documents/GitHub/aiida-testing/tests/mock_code/data/mock-diff-4b5ca93b23993979a24f795147487f1c

======================================== short test summary info =========================================
FAILED tests/mock_code/test_diff.py::test_basic - RuntimeError: No active exception to reraise

Also adds --mock-fail-on-missing CLI option. This can be used for CI (e.g. GH Action) runs, where you should likely not be wanting to generate new cache data

[chrisjsewell]

should we add perhaps one or two small tests, one checking the content of the log file, and one checking that execution fails in the expected way when asking to fail on missing executable?

yep absolutely, just wanted to open this prematurely, so you knew I was working on it 😄

[chrisjsewell]

FYI @ltalirz the other thing I was thinking to do, is allow users to specify a function as a string, to be loaded by https://docs.python.org/3/library/importlib.html#importlib.import_module in the mock code (or perhaps using entry points), that could override the default hashing function, e.g. to deal with things like the rounding errors in .xyz files

[ltalirz]

FYI @ltalirz the other thing I was thinking to do, is allow users to specify a function as a string, to be loaded by https://docs.python.org/3/library/importlib.html#importlib.import_module in the mock code (or perhaps using entry points), that could override the default hashing function, e.g. to deal with things like the rounding errors in .xyz files

mock_code has been in use for a while and the issue raised by @mpougin is the first time this has come up to my knowledge.

Yesterday, she posted the following comparison between two XYZ files https://github.com/lsmo-epfl/aiida-lsmo/issues/102#issuecomment-1322512168 which does not look like a rounding error at all.

If we do determine that this is not a bug but indeed related to randomness at the level of the machine epsilon, then I agree we can implement such a functionality (although I suspect its use will be limited to "experts").

[chrisjsewell]

Cheers, the remaining changes look fine; just some minor suggestions (optional). Let me know whether you want to add them

added cheers

[chrisjsewell]

cheers