nomatlab shape_inject.smk missing label

[jordandekraker]

Just encountered an error in one of the HCPaging subjects:

ABORTING PROGRAM DUE TO RUNTIME EXCEPTION -- File 'work/sub-HCA7247574_V1_MR/seg_T2w/sub-HCA7247574_V1_MR_hemi-Lflip_space-corobl_desc-nnunet_dsegsplit/label_06.nii.gz' does not exist

This subject has a bad nnunet segmentation which contains no label 6.

Not sure if we want shape_inject.smk to throw an error, a warning, or just keep going when a label is missing. A missing label is a good clue that nnunet has failed, but if its a small label then it is still possible that the shape_inject.smk could save it (say the HATA), so I think a warning and then perhaps the shape_inject.smk rule should skip labels that do not exist but still proceed.

[akhanf]

ah, yes it was not doing any checking since I've left the label_* files outside of snakemake's control (ie. in a dsegsplit directory that the workflow creates).. What prevented us from adding a check is that the files would have to exist, which wouldn't be the case when just building the DAG, and so would require cumbersome checkpoints..

But luckily version 6.4.0 of snakemake from 5 days ago includes a new feature where input functions for params that depend on input files can be listed as TBD until execution of the rule, so I've used that feature here.. In addition, a warning will be given to stdout when a label is missing but we can consider making that a more visible warning later.. In either case, should now proceed even if not all labels are there -- can you try it out? Note you will need to pip install again to get the latest snakemake..

[akhanf]

oops forgot to push, commit just pushed now

[jordandekraker]

looking good. I have a new nomatlab error but rather than starting a new issue maybe we can just use this issue, like a thread.

Activating singularity image /project/ctb-akhanf/akhanf/singularity/snakemake_containers/86248514372b3330b31a35088bb79331.simg
Traceback (most recent call last):
  File "/scratch/jdekrake/HippUnfold/nomatlab_test/testCases_T2w_nomatlab/.snakemake/scripts/tmp2vyyv4nt.vis_regqc.py", line 5, in <module>
    from nilearn import plotting
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/nilearn/__init__.py", line 67, in <module>
    _check_module_dependencies()
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/nilearn/version.py", line 124, in _check_module_dependencies
    _import_module_with_version_check(
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/nilearn/version.py", line 73, in _import_module_with_version_check
    module = __import__(module_name)
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/sklearn/__init__.py", line 82, in <module>
    from .base import clone
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/sklearn/base.py", line 17, in <module>
    from .utils import _IS_32BIT
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/sklearn/utils/__init__.py", line 22, in <module>
    from .murmurhash import murmurhash3_32
  File "sklearn/utils/murmurhash.pyx", line 1, in init sklearn.utils.murmurhash
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
[Wed May 26 16:52:25 2021]
Error in rule qc_reg_to_template:
    jobid: 115
    output: work/sub-HCA9575298_V1_MR/qc/sub-HCA9575298_V1_MR_from-subject_to-CITI168_regqc.png

RuleException:
CalledProcessError in line 58 of /scratch/jdekrake/HippUnfold/hippunfold/hippunfold/workflow/rules/preproc_t1.smk:
Command 'set -euo pipefail;  /scratch/jdekrake/HippUnfold/venv/bin/python /scratch/jdekrake/HippUnfold/nomatlab_test/testCases_T2w_nomatlab/.snakemake/scripts/tmp46cu89f6.vis_regqc.py' returned non-zero exit status 1.
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2349, in run_wrapper
  File "/scratch/jdekrake/HippUnfold/hippunfold/hippunfold/workflow/rules/preproc_t1.smk", line 58, in __rule_qc_reg_to_template
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 569, in _callback
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 555, in cached_or_run
  File "/scratch/jdekrake/HippUnfold/venv/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2381, in run_wrapper

[akhanf]

That's a weird one, looks to be python library incompatibility -- can you try running on a fresh output folder? Also what is your pip freeze output?

[jordandekraker]

will try a fresh run right now. pip freeze:

amply==0.1.4
appdirs==1.4.4
arff==0.9
argon2-cffi==20.1.0
arrow==1.1.0
async-generator==1.10
attrs==20.2.0
backcall==0.2.0
backports-abc==0.5
backports.shutil-get-terminal-size==1.0.0
batchgenerators==0.21
bcrypt==3.2.0
bids-validator==1.7.2
binaryornot==0.4.4
bitstring==3.1.7
bleach==3.2.1
blist==1.3.6
cached-property==1.5.2
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==8.0.1
ConfigArgParse==1.4.1
connection-pool==0.0.3
cookiecutter==1.7.3
cryptography==3.1.1
cycler==0.10.0
Cython==0.29.21
datrie==0.8.2
deap==1.3.1
decorator==4.4.2
defusedxml==0.6.0
dicom2nifti==2.2.10
dnspython==2.0.0
docopt==0.6.2
docutils==0.17.1
ecdsa==0.16.0
entrypoints==0.3
filelock==3.0.12
funcsigs==1.0.2
future==0.18.2
gitdb==4.0.7
GitPython==3.1.17
h5py==3.1.0
-e git+http://github.com/khanlab/hippunfold@f973340f482cc9a2c33d4e1ea7508f10a50a836f#egg=hippunfold
idna==2.10
imageio==2.9.0
importlib-metadata==2.0.0
ipykernel==5.3.4
ipython==7.18.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.2
Jinja2==2.11.2
jinja2-time==0.2.0
joblib==1.0.1
jsonschema==3.2.0
jupyter-client==6.1.7
jupyter-core==4.6.3
jupyterlab-pygments==0.1.2
kiwisolver==1.2.0
linecache2==1.0.0
lockfile==0.12.2
MarkupSafe==1.1.1
matlabengineforpython===R2020b
matplotlib==3.3.2
MedPy==0.4.0
mistune==0.8.4
mock==4.0.2
mpmath==1.1.0
nbclient==0.5.0
nbconvert==6.0.7
nbformat==5.0.7
nest-asyncio==1.4.1
netaddr==0.8.0
netifaces==0.10.9
networkx==2.5.1
nibabel==3.2.1
nilearn==0.7.1
nnunet==1.6.6
nose==1.3.7
notebook==6.1.4
num2words==0.5.10
numexpr==2.7.1
numpy==1.19.2
packaging==20.4
pandas==1.1.3
pandocfilters==1.4.2
paramiko==2.7.2
parso==0.7.0
path==15.0.0
path.py==12.5.0
pathlib2==2.3.5
patsy==0.5.1
paycheck==1.0.2
pbr==5.5.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
poyo==0.5.0
prometheus-client==0.8.0
prompt-toolkit==3.0.7
psutil==5.8.0
ptyprocess==0.6.0
PuLP==2.4
pybids==0.12.3
pycparser==2.20
pydicom==2.1.2
Pygments==2.7.1
pygraphviz==1.7
PyNaCl==1.4.0
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
python-slugify==5.0.2
pytz==2020.1
PyWavelets==1.1.1
PyYAML==5.3.1
pyzmq==19.0.2
ratelimiter==1.2.0.post0
requests==2.24.0
scikit-image==0.18.1
scikit-learn==0.24.1
scipy==1.5.2
seaborn==0.11.1
Send2Trash==1.5.0
simplegeneric==0.8.1
SimpleITK==2.0.2
singledispatch==3.4.0.3
six==1.15.0
sklearn==0.0
smart-open==5.1.0
smmap==4.0.0
snakebids==0.3.10
snakemake==6.4.0
SQLAlchemy==1.3.20
stopit==1.1.2
sympy==1.6.2
tables==3.6.1
terminado==0.9.1
testpath==0.4.4
text-unidecode==1.3
threadpoolctl==2.1.0
tifffile==2021.4.8
toposort==1.6
torch==1.8.1
tornado==6.0.4
tqdm==4.61.0
traceback2==1.4.0
traitlets==5.0.4
typing-extensions==3.10.0.0
unittest2==1.1.0
urllib3==1.25.10
vtk==8.2.0
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
wrapt==1.12.1
wxPython==4.1.0
zipp==3.3.0

[akhanf]

Try also with a fresh virtualenv.

On Thu, May 27, 2021 at 8:53 AM jordandekraker @.***> wrote:

will try a fresh run right now. pip freeze:

amply==0.1.4 appdirs==1.4.4 arff==0.9 argon2-cffi==20.1.0 arrow==1.1.0 async-generator==1.10 attrs==20.2.0 backcall==0.2.0 backports-abc==0.5 backports.shutil-get-terminal-size==1.0.0 batchgenerators==0.21 bcrypt==3.2.0 bids-validator==1.7.2 binaryornot==0.4.4 bitstring==3.1.7 bleach==3.2.1 blist==1.3.6 cached-property==1.5.2 certifi==2020.6.20 cffi==1.14.3 chardet==3.0.4 click==8.0.1 ConfigArgParse==1.4.1 connection-pool==0.0.3 cookiecutter==1.7.3 cryptography==3.1.1 cycler==0.10.0 Cython==0.29.21 datrie==0.8.2 deap==1.3.1 decorator==4.4.2 defusedxml==0.6.0 dicom2nifti==2.2.10 dnspython==2.0.0 docopt==0.6.2 docutils==0.17.1 ecdsa==0.16.0 entrypoints==0.3 filelock==3.0.12 funcsigs==1.0.2 future==0.18.2 gitdb==4.0.7 GitPython==3.1.17 h5py==3.1.0 -e @.#egg=hippunfold idna==2.10 imageio==2.9.0 importlib-metadata==2.0.0 ipykernel==5.3.4 ipython==7.18.1 ipython-genutils==0.2.0 ipywidgets==7.5.1 jedi==0.17.2 Jinja2==2.11.2 jinja2-time==0.2.0 joblib==1.0.1 jsonschema==3.2.0 jupyter-client==6.1.7 jupyter-core==4.6.3 jupyterlab-pygments==0.1.2 kiwisolver==1.2.0 linecache2==1.0.0 lockfile==0.12.2 MarkupSafe==1.1.1 @.#egg=hippunfoldidna==2.10imageio==2.9.0importlib-metadata==2.0.0ipykernel==5.3.4ipython==7.18.1ipython-genutils==0.2.0ipywidgets==7.5.1jedi==0.17.2Jinja2==2.11.2jinja2-time==0.2.0joblib==1.0.1jsonschema==3.2.0jupyter-client==6.1.7jupyter-core==4.6.3jupyterlab-pygments==0.1.2kiwisolver==1.2.0linecache2==1.0.0lockfile==0.12.2MarkupSafe==1.1.1> matlabengineforpython===R2020b matplotlib==3.3.2 MedPy==0.4.0 mistune==0.8.4 mock==4.0.2 mpmath==1.1.0 nbclient==0.5.0 nbconvert==6.0.7 nbformat==5.0.7 nest-asyncio==1.4.1 netaddr==0.8.0 netifaces==0.10.9 networkx==2.5.1 nibabel==3.2.1 nilearn==0.7.1 nnunet==1.6.6 nose==1.3.7 notebook==6.1.4 num2words==0.5.10 numexpr==2.7.1 numpy==1.19.2 packaging==20.4 pandas==1.1.3 pandocfilters==1.4.2 paramiko==2.7.2 parso==0.7.0 path==15.0.0 path.py==12.5.0 pathlib2==2.3.5 patsy==0.5.1 paycheck==1.0.2 pbr==5.5.0 pexpect==4.8.0 pickleshare==0.7.5 Pillow==7.2.0 poyo==0.5.0 prometheus-client==0.8.0 prompt-toolkit==3.0.7 psutil==5.8.0 ptyprocess==0.6.0 PuLP==2.4 pybids==0.12.3 pycparser==2.20 pydicom==2.1.2 Pygments==2.7.1 pygraphviz==1.7 PyNaCl==1.4.0 pyparsing==2.4.7 pyrsistent==0.17.3 python-dateutil==2.8.1 python-slugify==5.0.2 pytz==2020.1 PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==19.0.2 ratelimiter==1.2.0.post0 requests==2.24.0 scikit-image==0.18.1 scikit-learn==0.24.1 scipy==1.5.2 seaborn==0.11.1 Send2Trash==1.5.0 simplegeneric==0.8.1 SimpleITK==2.0.2 singledispatch==3.4.0.3 six==1.15.0 sklearn==0.0 smart-open==5.1.0 smmap==4.0.0 snakebids==0.3.10 snakemake==6.4.0 SQLAlchemy==1.3.20 stopit==1.1.2 sympy==1.6.2 tables==3.6.1 terminado==0.9.1 testpath==0.4.4 text-unidecode==1.3 threadpoolctl==2.1.0 tifffile==2021.4.8 toposort==1.6 torch==1.8.1 tornado==6.0.4 tqdm==4.61.0 traceback2==1.4.0 traitlets==5.0.4 typing-extensions==3.10.0.0 unittest2==1.1.0 urllib3==1.25.10 vtk==8.2.0 wcwidth==0.2.5 webencodings==0.5.1 widgetsnbextension==3.5.1 wrapt==1.12.1 wxPython==4.1.0 zipp==3.3.0

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/khanlab/hippunfold/issues/66#issuecomment-849608921, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXV2XIETAM2XWFWYHKRE2LTPY6ERANCNFSM45PULJEA .

-- Ali R. Khan, PhD Canada Research Chair in Computational Neuroimaging Assistant Professor, Medical Biophysics Scientist, Robarts Research Institute Schulich School of Medicine & Dentistry, Western University e. @.*** t. 519.931.5777 (x24280) w. www.khanlab.ca

[jordandekraker]

same problem with a fresh venv&install

Note this is with some of the HCPlifespan data. Here is the run command: hippunfold tmp testCases_T2w_nomatlab participant --participant_label HCA6018857_V1_MR HCA7247574_V1_MR HCA9575298_V1_MR HCD0551239_V1_MR HCD1534244_V1_MR HCD2568670_V1_MR --skip_preproc --skip_coreg --modality T2w --path_T1w /project/6050199/ext-data/hcp_lifespan_preproc/structural_preproc/{subject}/T1w/T1w_acpc_dc.nii.gz --path_T2w /project/6050199/ext-data/hcp_lifespan_preproc/structural_preproc/{subject}/T1w/T2w_acpc_dc.nii.gz --profile cc-slurm

[jordandekraker]

update: I finally got this working, it seems something in my environment was interfering. I ended up commenting out most of my .bash_profile and then making a new venv which was much cleaner. I'm still not sure specifically what in my previous environment was interfering though