JOSS Review — Test suite

[TheChymera]

As part of the JOSS Review → https://github.com/openjournals/joss-reviews/issues/5575

This is analogous with the issue raised here https://github.com/CyclotronResearchCentre/bidsme/issues/7

I fully appreciate that it can be difficult to formulate exhaustive or even just comprehensive tests for scientific software, but without a test suite it is impossible for me to determine whether the software can do anything at all.

The other reviewer has mentioned github workflows, which are a very useful feature. However if for some reason you don't want to do that, I'd be satisfied with just a few tests I can run locally, covering some of the functionalities mentioned in the article.

[nbeliy]

The code is tested gitlab-ci but on private gitlab of ULiege, which is the main platform. The CRC GitHub is a sort of front-end for publically aviable projects. But I guess I can replicate the same test on GitHub.

[nbeliy]

Forgot to announce in this issue, I ported the integration test from gitlab to github.

[TheChymera]

The test suite appears to pass on GitHub CI, however I can't seem to run it locally, am I missing something or is there an issue with the documentation of dependencies and testdata?

[deco]~/src/bidsme/bidsme_examples/example1 ❱ bidsme process renamed/ bids/ --plugin resources/plugins/process_plugin.py -b resources/map/bidsmap.yaml
main(76) - INFO
main(77) - INFO -------------- START bidsme ----------------
main(78) - INFO Wed Sep  6 04:21:56 2023
main(79) - INFO programm version: 1.4.0.post4
main(80) - INFO bids version: 1.2.0
bidsme.process(189) - INFO -------------- Processing data -------------
bidsme.process(190) - INFO Source directory: renamed/
bidsme.process(191) - INFO Destination directory: bids/
bidsme.process(213) - WARNING Dataset description file 'dataset_description.json' not found in 'bids/'
bidsme.process(219) - WARNING Dataset readme file 'README' not found in 'bids/'
bidsme.process(234) - INFO loading bidsmap /home/chymera/src/bidsme/bidsme_examples/example1/resources/map/bidsmap.yaml
bidsme.plugins.plugins(79) - INFO Loading module process_plugin from resources/plugins/process_plugin.py
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/main.py(138) in main:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/process.py(265) in process:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/bidsMeta/BidsSession.py(213) in loadSubjectFields:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/bidsMeta/BidsMeta.py(435) in LoadDefinitions:
bidsme.exceptions.exceptions(51) - ERROR 1:FileNotFoundError: [Errno 2] No such file or directory: 'renamed/participants.json'
main(181) - INFO Command: ['process', 'renamed/', 'bids/', '--plugin', 'resources/plugins/process_plugin.py', '-b', 'resources/map/bidsmap.yaml']
main(183) - INFO -------------- FINISHED! -------------------
main(148) - INFO counthangler:{'WARNING': 2, 'ERROR': 5}
main(152) - INFO loghandler:/home/chymera/src/bidsme/bidsme_examples/example1/bids/code/bidsme/process/main.log
main(152) - INFO errorhandler:/home/chymera/src/bidsme/bidsme_examples/example1/bids/code/bidsme/process/main.err
main(185) - INFO Took 0.891542123 seconds
main(186) - INFO --------------------------------------------
[deco]~/src/bidsme/bidsme_examples/example1 ❱ bidsme bidsify renamed/ bids/ --plugin resources/plugins/bidsify_plugin.py -b resources/map/bidsmap.yaml
main(76) - INFO
main(77) - INFO -------------- START bidsme ----------------
main(78) - INFO Wed Sep  6 04:22:09 2023
main(79) - INFO programm version: 1.4.0.post4
main(80) - INFO bids version: 1.2.0
bidsme.bidsify(179) - INFO -------------- Prepearing data -------------
bidsme.bidsify(180) - INFO Source directory: renamed/
bidsme.bidsify(181) - INFO Destination directory: bids/
bidsme.bidsify(203) - WARNING Dataset description file 'dataset_description.json' not found in 'bids/'
bidsme.bidsify(209) - WARNING Dataset readme file 'README' not found in 'bids/'
bidsme.bidsify(224) - INFO loading bidsmap /home/chymera/src/bidsme/bidsme_examples/example1/resources/map/bidsmap.yaml
bidsme.plugins.plugins(79) - INFO Loading module bidsify_plugin from resources/plugins/bidsify_plugin.py
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/main.py(151) in main:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/bidsify.py(255) in bidsify:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/bidsMeta/BidsSession.py(213) in loadSubjectFields:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/bidsMeta/BidsMeta.py(435) in LoadDefinitions:
bidsme.exceptions.exceptions(51) - ERROR 1:FileNotFoundError: [Errno 2] No such file or directory: 'renamed/participants.json'
main(181) - INFO Command: ['bidsify', 'renamed/', 'bids/', '--plugin', 'resources/plugins/bidsify_plugin.py', '-b', 'resources/map/bidsmap.yaml']
main(183) - INFO -------------- FINISHED! -------------------
main(148) - INFO counthangler:{'WARNING': 2, 'ERROR': 5}
main(152) - INFO loghandler:/home/chymera/src/bidsme/bidsme_examples/example1/bids/code/bidsme/bidsify/main.log
main(152) - INFO errorhandler:/home/chymera/src/bidsme/bidsme_examples/example1/bids/code/bidsme/bidsify/main.err
main(185) - INFO Took 0.973882708 seconds
main(186) - INFO --------------------------------------------
[deco]~/src/bidsme/bidsme_examples/example1 ❱ bidsme map renamed/ bids/ --plugin resources/plugins/bidsify_plugin.py -b resources/map/bidsmap.yaml
main(76) - INFO
main(77) - INFO -------------- START bidsme ----------------
main(78) - INFO Wed Sep  6 04:22:20 2023
main(79) - INFO programm version: 1.4.0.post4
main(80) - INFO bids version: 1.2.0
bidsme.mapper(202) - INFO ------------ Generating bidsmap ------------
bidsme.mapper(203) - INFO Current directory: /home/chymera/src/bidsme/bidsme_examples/example1
bidsme.mapper(204) - INFO Source directory: renamed/
bidsme.mapper(205) - INFO Destination directory: bids/
bidsme.mapper(221) - INFO loading template bidsmap bidsmap_template.yaml
bidsme.bidsmap._bidsmap(103) - WARNING Failed to find type EEG/BrainVision readed from /usr/lib/python3.11/site-packages/bidsme/heuristics/bidsmap_template.yaml
bidsme.mapper(240) - INFO loading working bidsmap /home/chymera/src/bidsme/bidsme_examples/example1/resources/map/bidsmap.yaml
bidsme.bidsmap._bidsmap(64) - INFO bids/code/bidsme/unknown.yaml not found. Bidsmap will be empty
bidsme.plugins.plugins(79) - INFO Loading module bidsify_plugin from resources/plugins/bidsify_plugin.py
bidsme.bidsMeta.BidsTable(88) - ERROR participants.tsv: Unable to find definitions file
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/main.py(164) in main:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/mapper.py(265) in mapper:
bidsme.exceptions.exceptions(49) - ERROR /usr/lib/python3.11/site-packages/bidsme/bidsMeta/BidsTable.py(90) in __init__:
bidsme.exceptions.exceptions(51) - ERROR 1:FileNotFoundError: renamed/participants.json
main(181) - INFO Command: ['map', 'renamed/', 'bids/', '--plugin', 'resources/plugins/bidsify_plugin.py', '-b', 'resources/map/bidsmap.yaml']
main(183) - INFO -------------- FINISHED! -------------------
main(148) - INFO counthangler:{'WARNING': 1, 'ERROR': 5}
main(152) - INFO loghandler:/home/chymera/src/bidsme/bidsme_examples/example1/bids/code/bidsme/map/main.log
main(152) - INFO errorhandler:/home/chymera/src/bidsme/bidsme_examples/example1/bids/code/bidsme/map/main.err
main(185) - INFO Took 1.076100048 seconds
main(186) - INFO --------------------------------------------
[deco]~/src/bidsme/bidsme_examples/example1 ❱ git rev-parse HEAD
8c7f74742fc4b2db92230160cbb98185f5ab2920

[TheChymera]

Also, I would strongly recommend the test suite be captured in a dedicated script or BATS (since you're calling the package CLI via Bash), or in a test module as used by pytest (if you want the Python functions tested directly in Python) so that they can be run automatically without copy-pasting from the GitHub test suite. In the GitHub test suite, you can then call that script/BATS/pytest.

The reason why this is important is that package managers might want to integrate the testing functionality of your package so users can automatically ascertain that what they installed works as it should on their machine. Currently this is not really possible.

[nbeliy]

Hi @TheChymera,

About the failing test: I'm not sure but it looks like the bidsme process is run not after bidsme prepare (maybe you run test in parallel or you remove the artifacts after the prepare). It is an integration test, not unit test (unfortunately).

For the bats, thanks for pointing it, I didn't knew about. But here it looks like an overkill -- just to replace one line of script in workflow.

[TheChymera]

Ok, the tests now pass locally. The environment needed openpyxl to be installed, which pip did not auto-install for some reason.