search

psychoinformatics-de / datalad-hirni

DataLad extension for (semi-)automated, reproducible processing of (medical/neuro)imaging data
http://datalad.org
Other
5 stars 8 forks source link

Fails testing in datalad-extensions #158

Closed yarikoptic closed 3 years ago

yarikoptic commented 4 years ago

See e.g. https://github.com/datalad/datalad-extensions/runs/641512513?check_suite_focus=true . I have failed to reproduce while running locally. PR adding hirni was merged while failure was already in there, so the issue is known but was not resolved for a while, and causes attention fatigue for me while reviewing daily failed builds reports. With constant "red" I am not even sure what is the benefit of having it listed in -extensions CI dashboard.

Given the error message of datalad_hirni.tests.test_dicomconv.test_dicom2bids('functional',) ... ERROR : Image path code/hirni-toolbox/converters/heudiconv/heudiconv.simg doesn't exist: No such file or directory I wonder if additional logging could help to point to the reason for the issue - is it really not downloaded container (why didn't fail while getting it?) or may be incorrect path?

Also I see datalad_hirni.tests.test_demos.test_demo_repro_analysis ... /tmp/datalad_temp_test_demo_repro_analysislqn0ej5i/sourcedata/code/hirni-toolbox/procedures/copy-converter.sh: line 9: $1: unbound variable so might be worth checking on the invocation there as well why missing a (dataset?) argument.

bpoldrack commented 4 years ago

But where to make use of those logs if we can't reproduce it? datalad-extensions tests for released datalad-hirni. So those builds can't be used for debugging. A PR against datalad-extensions using a debug branch of hirni instead? Better idea anyone?

Also FTR: Those builds suggest that the issue is in datalad since the same hirni release passed against released datalad.

yarikoptic commented 4 years ago

copy that workflow into hirni and run it here on master version? may be it was already fixed since release ? (didn't check if there is a diff) In logs you might see what is the underlying issue and address it without reproducing

bpoldrack commented 4 years ago

No diff. But will copy workflow to a debugging PR here, yes.

bpoldrack commented 4 years ago

Ok, so: The issue arose from hirni's tests cloning the toolbox and getting its conatiners over and over again. For some reason this lead to the original data source to not deliver (github workflow only). Therefore solution has several parts:

  1. Need to figure how and why exactly our infrastructure is rejecting those requests. That's not an obvious "too many requests", since it doesn't happen when they are coming from Travis. Furthermore git-annex-get reports "Unauthorized".

However, we should avoid doing that over and over again anyway and hirni comes with a config for that: datalad.hirni.toolbox.url. Therefore:

  1. Setup for tests should clone and get the toolbox once and set this config via env var.
  2. This will affect cfg_hirni procedure only (when it adds the toolbox). But in parts we use a demo dataset (https://github.com/psychoinformatics-de/hirni-demo), that has the toolbox registered already. To prevent those tests from still pulling the toolbox, we need a helper for installing this dataset, that will overwrite the toolbox URL whenever above mentioned config is set.

This is addressed in #160 and https://github.com/datalad/datalad-extensions/pull/12