search

nipreps / mriqc

Automated Quality Control and visual reports for Quality Assessment of structural (T1w, T2w) and functional MRI of the brain
http://mriqc.readthedocs.io
Apache License 2.0
297 stars 132 forks source link

pybids recent changes break compatibility: "No scans were found" #340

Closed parekhpravesh closed 7 years ago

parekhpravesh commented 7 years ago

Hello,

I have been able to run mriqc on a workstation. I then moved on to install a copy on our server and I think I have followed the same steps. When I run mriqc /run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001 /run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001_output/ participant --participant_label sub-01 sub-02 -w /run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001_work/

I get the following error

/home/MBIAL/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20. "This module will be removed in 0.20.", DeprecationWarning) 170124-23:15:25,490 mriqc INFO: Running MRIQC-0.9.0-rc1 (analysis_level=participant, participant_label=['sub-01', 'sub-02']) Settings={u'report_dir': u'/run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001_output/reports', u'work_dir': '/run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001_work', u'n_procs': 5, u'bids_dir': '/run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001', u'hmc_afni': False, u'write_graph': False, u'ants_nthreads': 6, u'verbose_reports': False, u'testing': False, u'output_dir': '/run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001_output'} 170124-23:15:25,509 mriqc WARNING: No anatomical scans were found in /run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001 170124-23:15:25,509 mriqc WARNING: No functional scans were found in /run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001

This is the BIDS example dataset that I am trying to run. I have successfully been able to run the same on the test workstation (and other dataset) but none of them seem to be running on the server (same error message). Any suggestions would be greatly appreciated. I have also tried moving the dataset to the server directly (instead of using a storage device) but the above issue persists.

Interestingly, I can use the workstation to process the same location of data! The test workstation and the server are both running Redhat.

Thanks and Regards Pravesh

oesteban commented 7 years ago

Hi @parekhpravesh, please confirm that your dataset is BIDS compliant by checking the folder /run/media/MBIAL/MBIAL_STORAGE/Parekh/ds001 in the BIDS-validator

parekhpravesh commented 7 years ago

Hello @oesteban, I am confident that the dataset is BIDS compliant because I just got it off Google Drive. Also the fact that I can get mriqc to run on the exact folder on a different workstation. Nonetheless, will run it through the validator and comment again!

oesteban commented 7 years ago

Well, then it looks like data are not found where they are expected... Especially when you run it nicely on the same dataset, with a different workstation. Maybe a problem copying over the dataset?

oesteban commented 7 years ago

Sorry, @chrisfilo just found the source of the problem, sorry about that.

parekhpravesh commented 7 years ago

Hello again,

I just checked the dataset. Apart from the two warning messages about missing slice timing information, and not all subjects have the same specification, it seems alright.

oesteban commented 7 years ago

@parekhpravesh we just applied the patch, please check if it works for you. You will need the latest master branch of this repo.

oesteban commented 7 years ago

Awaiting for #344

parekhpravesh commented 7 years ago

Hello Sorry about the delay. I downloaded the updated repo as a zip file; removed the older version by calling pip uninstall mriqc=0.9.0rc1 and then installed from the zip file by calling pip install <path_to_zip>

Subsequently, on running mriqc /home/MBIAL/Parekh/dataset /home/MBIAL/Parekh/dataset_output/ participant --participant_label sub-001 -w /home/MBIAL/Parekh/dataset_work/

I get the following error

/home/MBIAL/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20. "This module will be removed in 0.20.", DeprecationWarning) 170125-00:54:33,199 mriqc INFO: Running MRIQC-0.9.0-rc1 (analysis_level=participant, participant_label=['sub-001']) Settings={u'report_dir': u'/home/MBIAL/Parekh/dataset_output/reports', u'work_dir': '/home/MBIAL/Parekh/dataset_work', u'n_procs': 5, u'bids_dir': '/home/MBIAL/Parekh/dataset', u'hmc_afni': False, u'write_graph': False, u'ants_nthreads': 6, u'verbose_reports': False, u'testing': False, u'output_dir': '/home/MBIAL/Parekh/dataset_output'} 170125-00:54:33,206 mriqc WARNING: No anatomical scans were found in /home/MBIAL/Parekh/dataset 170125-00:54:33,206 mriqc WARNING: No functional scans were found in /home/MBIAL/Parekh/dataset Traceback (most recent call last): File "/home/MBIAL/Enthought/Canopy_64bit/User/bin/mriqc", line 11, in load_entry_point('mriqc==0.9.0rc1', 'console_scripts', 'mriqc')() File "/home/MBIAL/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mriqc/bin/mriqc_run.py", line 244, in main 'BIDS-compliant.' % settings['bids_dir']) RuntimeError: Error reading BIDS directory (/home/MBIAL/Parekh/dataset), or the dataset is not BIDS-compliant.

I have checked this particular folder using BIDS Validator and it confirms that the directory is valid BIDS dataset. The same error persists for the downloaded example dataset too.

oesteban commented 7 years ago

Yeah, I merged in some changes too fast. Please uninstall and install mriqc this way:

pip uninstall mriqc
pip install git+https://github.com/poldracklab/mriqc.git@chrisfilo-patch-2

That should work for you (and will confirm we fixed the bug).

Thanks very much

parekhpravesh commented 7 years ago

Cloning now! Will update asap. Thank you!

parekhpravesh commented 7 years ago

Its still cloning...is it supposed to take so long?

parekhpravesh commented 7 years ago

Hello again,

I don't think the problem is resolved:

mriqc /home/MBIAL/Parekh/dataset /home/MBIAL/Parekh/dataset_output/ participant --participant_label sub-001 -w /home/MBIAL/Parekh/dataset_work/

/home/MBIAL/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20. "This module will be removed in 0.20.", DeprecationWarning) 170125-01:18:43,141 mriqc INFO: Running MRIQC-0.9.0-rc1 (analysis_level=participant, participant_label=['sub-001']) Settings={u'report_dir': u'/home/MBIAL/Parekh/dataset_output/reports', u'work_dir': '/home/MBIAL/Parekh/dataset_work', u'n_procs': 5, u'bids_dir': '/home/MBIAL/Parekh/dataset', u'hmc_afni': False, u'write_graph': False, u'ants_nthreads': 6, u'verbose_reports': False, u'testing': False, u'output_dir': '/home/MBIAL/Parekh/dataset_output'} 170125-01:18:43,147 mriqc WARNING: No anatomical scans were found in /home/MBIAL/Parekh/dataset 170125-01:18:43,147 mriqc WARNING: No functional scans were found in /home/MBIAL/Parekh/dataset Traceback (most recent call last): File "/home/MBIAL/Enthought/Canopy_64bit/User/bin/mriqc", line 11, in load_entry_point('mriqc==0.9.0rc1', 'console_scripts', 'mriqc')() File "/home/MBIAL/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mriqc/bin/mriqc_run.py", line 244, in main 'BIDS-compliant.' % settings['bids_dir']) RuntimeError: Error reading BIDS directory (/home/MBIAL/Parekh/dataset), or the dataset is not BIDS-compliant.

parekhpravesh commented 7 years ago

My bad...I just closed and opened the terminal and it seems to be running now! Thank you so much! :D

parekhpravesh commented 7 years ago

Hello again,

It seems that the problem is only partially resolved. I can get the example dataset to run fine but the exact error (Error reading BIDS directory, or the dataset is not BIDS-compliant) persists when using some of our dataset. The BIDS validator confirms that the dataset is alright. Over and above this, I can get the workstation installation of mriqc (installed two days prior to the present installation) to process the dataset perfectly well! Any suggestions on what could be going wrong would be most welcome...

Thanks and Regards Pravesh

oesteban commented 7 years ago

Hi @parekhpravesh sorry for the inconvenience.

Can you post here the outputs of: pip freeze and also the structure of the BIDS dataset?, something like find path/to/bids/data.

Thanks very much

parekhpravesh commented 7 years ago

Hello,

Thank you for the reply and for all the time :)

First, the output of the BIDS structure (as its shorter):

/run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS056 /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS056/anat /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS056/anat/sub-HS056_T1w.json /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS056/anat/sub-HS056_T1w.nii.gz /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/derivatives /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/derivatives/Log_Files /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/derivatives/Log_Files/sub-HS032_dcm2niix_LOG.txt /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/derivatives/Log_Files/sub-HS043_dcm2niix_LOG.txt /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/derivatives/Log_Files/sub-HS056_dcm2niix_LOG.txt /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS043 /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS043/anat /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS043/anat/sub-HS043_T1w.json /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS043/anat/sub-HS043_T1w.nii.gz /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS032 /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS032/anat /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS032/anat/sub-HS032_T1w.nii.gz /run/media/MBIAL/MBIAL_STORAGE/Parekh/mriqc_dst_25Jan2017_BIDS_Test/sub-HS032/anat/sub-HS032_T1w.json

Now, the output of pip freeze follows (have tried to highlight all the mriqc dependencies):

appdirs==1.4.0 appinst==2.1.5 apptools==4.4.0 atom==0.3.10 attrs==15.2.0 backports-abc==0.4 beautifulsoup4==4.4.1 boto==2.40.0 Canopy==1.7.4.3348 canopy-dashboard==0.7.19 canopy-packman-plugin==0.1.19 canopy-platform==0.9.12 CanopyDebugger==0.1.2.dev0 casuarius==1.1 catalyst==1.0.2 certifi==2016.2.28 chaco==4.5.0 chardet==2.3.0 click==6.7 codetools==4.2.0 colorama==0.3.7 configobj==5.0.6 configparser==3.5.0 cycler==0.10.0 decorator==4.0.11 > dipy==0.11.0 distribute-remove==1.0.0 docutils==0.12 ecdsa==0.13 enable==4.5.1 enaml==0.9.8 enapp==1.0.0 enclosure==0.4.3 encore==0.6.0 enstaller==4.8.12 entrypoints==0.2.2 enum34==1.1.6 envisage==4.5.1 esky==0.9.8 Examples==7.3 faulthandler==2.4 feedparser==5.2.1 flake8==2.5.1 freetype==2.5.3 funcsigs==1.0.2 functools32==3.2.3.post2 > future==0.16.0 futures==3.0.3 glib==2.36.1 gnureadline==6.3.3 grabbit==0.0.5.1 grits-client==0.1 gst-plugins-base==0.10.36 gstreamer==0.10.36 hatcher==0.10.1 html5lib==0.999999999 humanize==0.5.1 idle==2.7.3 ipykernel==4.3.1 ipython==4.0.0 ipython-genutils==0.1.0 ipython4==4.1.2 ipywidgets==5.1.5 isodate==0.5.4 jigna==0.9.9 > Jinja2==2.9.4 jsonpickle==0.4.0 jsonschema==2.4.0 jupyter==1.0.0 jupyter-client==4.2.2 jupyter-console==4.1.1 jupyter-core==4.1.0 kernmagic==0.2.0 keyring==4.0 kiwisolver==0.1.3 libffi==3.0.13 libgfortran==3.0.0 libjpeg==7.0 libncurses==5.9 libopenjpeg==2.1.0 libpng==1.6.12 libsodium==1.0.3 libtheora==1.1.1 libxml2==2.9.2 libxslt==1.1.28 libYAML==0.1.4 lxml==3.7.2 MarkupSafe==0.23 > matplotlib==2.0.0 mccabe==0.3.1 memory-profiler==0.41 mistune==0.7.1 MKL==11.1.4 mkl-service==1.0 mock==2.0.0 mpmath==0.19 mriqc==0.9.0rc1 nbconvert==4.2.0 nbformat==4.0.1 networkx==1.11 > nibabel==2.1.0 > nilearn==0.2.6 > nipy==0.4.0 > nipype==0.13.0rc1 > nitime==0.7 > niworkflows==0.0.5 nose==1.3.7 notebook==4.2.1 > numpy==1.12.0 okonomiyaki==0.15.0 packaging==16.8 > pandas==0.19.2 path.py==8.1.1 patsy==0.4.1 pbr==1.10.0 pep8==1.7.0 pexpect==3.3 pickleshare==0.5 PIL-remove==1.0.0 pillow==3.2.0 plotly==1.9.10 ply==3.8 prov==1.5.0 psutil==3.3.0 ptyprocess==0.4 py==1.4.32 pyasn1==0.1.9 pyaudio==0.2.4 > pybids==0.0.1 pycrypto==2.6.1 pyface==5.1.0 pyflakes==1.1.0 pyglet==1.1.4 Pygments==2.1.3 pygraphviz==1.3.1 pyparsing==2.1.10 PyPDF2==1.26.0 PySide==1.2.2 pytest==3.0.6 python-dateutil==2.6.0 pytz==2016.10 > PyYAML==3.12 pyzmq==15.2.0 qap==1.0.5 Qt==4.8.7 qtconsole==4.2.1 rdflib==4.2.1 reportlab==3.3.0 requests==2.10.0 rsa==3.1.2 scikit-learn==0.18.1 > scipy==0.18.1 > seaborn==0.7.1 shiboken==1.2.2 simplegeneric==0.8.1 simplejson==3.10.0 singledispatch==3.4.0.3 six==1.10.0 sklearn==0.0 SPARQLWrapper==1.8.0 ssl-match-hostname==3.4.0.2 > statsmodels==0.6.1 subprocess32==3.2.7 supplement==0.5.dev0 > svgutils==0.2.0 sympy==1.0 tabulate==0.7.3 terminado==0.5 tinycss==0.4 tornado==4.3 traitlets==4.2.1 traits==4.6.0 traits-enaml==0.2.1 traitsui==5.1.0 tzlocal==1.2 webencodings==0.5 wxPython==3.0.2.0 xhtml2pdf==0.0.6 xvfbwrapper==0.2.9 zeromq==4.1.3 zipfile2==0.0.12

parekhpravesh commented 7 years ago

Okay, I think I found the problem: When I include a 'func' folder with functional data, the pipeline starts to run! When I remove the functional scans, mriqc can't read the dataset! Passing a 'anat' flag along with -d doesn't help though

parekhpravesh commented 7 years ago

Further update: when putting in the functional scans, it is running only on the functional scans (still fails to detect anatomical scans).

P.S: Sorry about the many messages!

oesteban commented 7 years ago

Don't be sorry, this is all very helpful!

Thanks very much

On Mon, Jan 30, 2017 at 10:25 AM, Pravesh Parekh notifications@github.com wrote:

Further update: when putting in the functional scans, it is running only on the functional scans (still fails to detect anatomical scans).

P.S: Sorry about the many messages!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/poldracklab/mriqc/issues/340#issuecomment-276146614, or mute the thread https://github.com/notifications/unsubscribe-auth/AAkhxveNAftXJpt4fBL5tLklokQQzrHhks5rXisDgaJpZM4LslPJ .

parekhpravesh commented 7 years ago

Same holds true for the example dataset. It can only detect and run functional scans for some reason

mfalkiewicz commented 7 years ago

Hello, same issue here. I've installed the version patched by Chris, it detects the functional scans, but not the anatomical ones. The docker version correctly identifies both, but unfortunately we cannot run docker in our computational environment.

chrisgorgo commented 7 years ago

Hi @mfalkiewicz and @parekhpravesh - I think I found the problem. Could you see if this PR fixes it for you? https://github.com/poldracklab/mriqc/pull/352

parekhpravesh commented 7 years ago

Hello,

Apologies for the delay. I had already left lab for the day...too late to try the pull request now but I did clone from the master branch and the pipeline is running! I will post again once I have the output. Thank you so much @oesteban @chrisfilo :grinning:

mfalkiewicz commented 7 years ago

@chrisfilo : this worked, not the pipeline is picking up both the anatomical and functional scans. Thanks for a quick fix! I've tested the master branch after @oesteban merged your patch. I think this can be closed now.

parekhpravesh commented 7 years ago

I can add that it works alright for some of our structural scans...I have had a crash with the example dataset (sub-01)...I am re-running that part again...will post and close if it works alright. Thanks!

parekhpravesh commented 7 years ago

Hello,

The patch is working fine. Thank you so much @oesteban @chrisfilo

angoldst commented 7 years ago

Which install should I be using? Just tried uninstalling and reinstalling using the standard pip install but it still runs into the same errors.

Thanks again!

The install I just did seems to be running MRIQC-0.8.9-1 in case that helps.

parekhpravesh commented 7 years ago

@angoldst you could try pip install git+https://github.com/poldracklab/mriqc

angoldst commented 7 years ago

@parekhpravesh That did it thank you!

seantma commented 7 years ago

Hi @chrisfilo @oesteban I'd like to re-open up this no scans found issue when running MRIQC 0.9.2 under Docker. I'm on OSX and have 10GB memory assigned to Docker. The dataset I'm testing this on is OpenfMRI's ds114 (v2.0.1; BIDS compliant)

The command I ran is: docker run \ -v ~/Projects/BIDS-app/ds114_new:/data \ -v ~/Projects/BIDS-app/scratch:/scratch \ -w /scratch \ poldracklab/mriqc:0.9.2 \ /data /scratch/out participant --participant_label 001 002 003 \ -w /scratch/work

The error message I got was:

his module will be removed in 0.20. "This module will be removed in 0.20.", DeprecationWarning) 2017-04-21 13:45:19,381 root:INFO Running MRIQC-0.9.2 (analysis_levels=[participant], participant_label=['001', '002', '003']) Settings={'bids_dir': '/data', 'write_graph': False, 'testing': False, 'hmc_afni': True, 'hmc_fsl': False, 'fft_spikes_detector': False, 'n_procs': 3, 'ants_nthreads': 6, 'output_dir': '/scratch/out', 'work_dir': '/scratch/work', 'verbose_reports': False, 'float32': False, 'ica': False, 'deoblique': False, 'despike': False, 'correct_slice_timing': False, 'report_dir': '/scratch/out/reports'} 2017-04-21 13:45:19,651 root:WARNING No T1w scans were found in /data 2017-04-21 13:45:19,651 root:WARNING No bold scans were found in /data 2017-04-21 13:45:19,651 root:WARNING No T2w scans were found in /data participant labels ['^001$', '^002$', '^003$'] Traceback (most recent call last): File "/usr/local/miniconda/bin/mriqc", line 11, in load_entry_point('mriqc==0.9.2', 'console_scripts', 'mriqc')() File "/usr/local/miniconda/lib/python3.6/site-packages/mriqc/bin/mriqc_run.py", line 259, in main 'BIDS-compliant.' % settings['bids_dir']) RuntimeError: Error reading BIDS directory (/data), or the dataset is not BIDS-compliant.

Your help is appreciated!

Regards, Sean

chrisgorgo commented 7 years ago

I think the problem is that you are requesting incorrect participant labels. Try 01 instead of 001.

seantma commented 7 years ago

@chrisfilo Thanks! That worked! The participant label format was following the BIDS-app syntax listed on the BIDS-app Github page. Seems like at least for MRIQC, the BIDS-app page is quite outdated.

chrisgorgo commented 7 years ago

You mean this page https://github.com/BIDS-Apps/mriqc/blob/master/README.md? Could you clarify what about the participant label is incorrect there?

seantma commented 7 years ago

@chrisfilo After reviewing my notes, the 3 digit participant syntax format probably was pulled from the OpenfMRI study digits and got mixed into the BIDS syntax. My bad!

chrisgorgo commented 7 years ago

No problem! Please mind that in BIDS participant labels do not have to be just digits but any strings (with [a-zA-Z0-9]).

On Tue, Apr 25, 2017 at 12:05 PM, Sean Ma notifications@github.com wrote:

@chrisfilo https://github.com/chrisfilo After reviewing my notes, the 3 digit participant syntax format probably was pulled from the OpenfMRI study digits and got mixed into the BIDS syntax. My bad!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/poldracklab/mriqc/issues/340#issuecomment-297133178, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOkp9fw4WDteaTXHx2CZOeJSnSjj8Siks5rzkPrgaJpZM4LslPJ .