search

poldracklab / ds003-post-fMRIPrep-analysis

An exemplary task analysis workflow to run on OpenNeuro's ds000003 data, after preprocessing with fMRIPrep
Apache License 2.0
28 stars 8 forks source link

AttributeError and missing py package in image? #10

Open arovai opened 3 years ago

arovai commented 3 years ago

Hello!

I have build the docker image from the Dockerfile and now I'd like to re-run the data analysis. I have downloaded the post-fMRIprep data from datalad, but running the docker image gives an error.

The command I run is

sudo docker run -v ds003_fmriprep:/home/bidsapp/bids_dir -v output:/home/bidsapp/output ds003-post-fmrirep-analysis /home/bidsapp/bids_dir /home/bidsapp/output group

where ds003-post-fmrirep-analysis is the tag I have used for the build, ds003_fmriprep is the datalad dataset (locator ///labs/poldrack/ds003_fmriprep), and output is some empty folder I have created.

The error I am getting is

Failed to import duecredit due to No module named 'duecredit'
Traceback (most recent call last):
  File "/src/run.py", line 232, in <module>
    sys.exit(main())
  File "/src/run.py", line 118, in main
    bids_dir = opts.bids_dir.resolve()
AttributeError: 'NoneType' object has no attribute 'resolve'

Note that the error Failed to import duecredit due to No module named 'duecredit' also appears when I run the images with whatever option, such as sudo docker run ds003-post-fmrirep-analysis -h(which otherwise seems to work just fine, i.e. I get the run.py help).

Since the resolve attribute seems to have something to do with getting absolute paths, I have tried various combinations for the -v option and the main arguments, but no luck so far.

Any thoughs?

PS: I am running docker v20.10.5 and Ubuntu v20.04

oesteban commented 3 years ago

Hi @arovai, the authoritative voice here would be @rciric, but I'll try to figure this one out by looking at the code.

You are getting:

    bids_dir = opts.bids_dir.resolve()
AttributeError: 'NoneType' object has no attribute 'resolve'

Basically, this is telling you that opts.bids_dir is None (and therefore there's no way to call resolve). So, next is to determine what's that opts object here. It is the parser from the argparse standard library. The parser defines a --bids-dir argument that maps to that bids_dir member of the parser here:

https://github.com/poldracklab/ds003-post-fMRIPrep-analysis/blob/e300357c5714e56157e6aaf63b9b70beab3f8b42/run.py#L58

I cannot see this argument is marked as mandatory (unfortunately) - but it is mandatory.

I think you need to correct the command line you are using, making sure you set --bids-dir:

sudo docker run -v ds003_fmriprep:/home/bidsapp/bids_dir \
                -v output:/home/bidsapp/output \
                ds003-post-fmrirep-analysis \
                <path-to-fmriprep-derivatives-folder> /home/bidsapp/output group \
                --bids-dir /home/bidsapp/bids_dir

Please replace the placeholder <path-to-fmriprep-derivatives-folder> with the appropriate path.

arovai commented 3 years ago

Hi @oesteban, thanks for your input, it helped a bit but also brought another issue and a several questions.

First, I am a bit confused when you write that I must replace the placeholder <path-to-fmriprep-derivatives-folder> (in your answer) because that would be /home/bidsapp/bids_dir according to the volumes mounted in the docker image and, unless I am mistaken, shouldn't be anything else.

Second, adding as you suggest --bids-dir /home/bidsapp/bids_dir produces another, different error. The exact command I am running reads

sudo docker run -v /path/to/ds003_fmriprep:/home/bidsapp/bids_dir -v /path/to/output:/home/bidsapp/output ds003-post-fmrirep-analysis:latest ./ds003_fmriprep /home/bidsapp/output participant --bids-dir /home/bidsapp/bids_dir

(Note that another, easily resolved error I had, came up because I didn't use full paths in the -v options of docker. Also note that I am running at participant level and added the :latest tag - another couple of minor changes from the command mentioned in my OP).

The output reads

Failed to import duecredit due to No module named 'duecredit'
/usr/local/miniconda/lib/python3.7/site-packages/nipype/workflows/__init__.py:28: UserWarning: Nipype 1 workflows have been moved to the niflow-nipype1-workflows package. pip install niflow-nipype1-workflows to continue using them.
  warnings.warn(" ".join(_msg))
Traceback (most recent call last):
  File "/src/run.py", line 232, in <module>
    sys.exit(main())
  File "/src/run.py", line 142, in main
    from workflows import first_level_wf
  File "/src/workflows.py", line 8, in <module>
    from nipype.workflows.fmri.fsl.preprocess import create_susan_smooth
ModuleNotFoundError: No module named 'nipype.workflows.fmri'

So now it complains that a module is missing... but I should not have to install anything myself, as the image itself should contain everything it need to run, right?

arovai commented 3 years ago

Alright, I was going to edit my comment to mention that this is probably a version issue, as I found reading https://github.com/nipy/nipype/blob/master/nipype/workflows/__init__.py :-)

effigies commented 3 years ago

Can you try building from current master, and see how far you can get?

arovai commented 3 years ago

Rebuild successful, new error: says cmp not installed, something from mrtrix?

sudo docker run -v $PWD/ds003_fmriprep:/home/bidsapp/bids_dir -v $PWD/output:/home/bidsapp/output ds003-post-fmrirep-analysis:latest /home/bidsapp/bids_dir /home/bidsapp/output participant --bids-dir /home/bidsapp/bids_dir

Failed to import duecredit due to No module named 'duecredit'
/usr/local/miniconda/lib/python3.7/site-packages/niflow/nipype1/workflows/dmri/mrtrix/group_connectivity.py:16: UserWarning: cmp not installed
  warnings.warn('cmp not installed')
/usr/local/miniconda/lib/python3.7/site-packages/nipype/workflows/__init__.py:28: UserWarning: Nipype 1 workflows have been moved to the niflow-nipype1-workflows package. nipype.workflows.* provides a reference for backwards compatibility. Please use niflow.nipype1.workflows.* to avoid this warning.
  warnings.warn(" ".join(_msg))
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/feature_extraction/image.py:167: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  dtype=np.int):
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/__init__.py:90: FutureWarning: Fetchers from the nilearn.datasets module will be updated in version 0.9 to return python strings instead of bytes and Pandas dataframes instead of Numpy arrays.
  "Numpy arrays.", FutureWarning)
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:35: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  eps=np.finfo(np.float).eps,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:597: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:836: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:862: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  eps=np.finfo(np.float).eps, positive=False):
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1074: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  max_n_alphas=1000, n_jobs=1, eps=np.finfo(np.float).eps,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1306: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  max_n_alphas=1000, n_jobs=1, eps=np.finfo(np.float).eps,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1442: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  eps=np.finfo(np.float).eps, copy_X=True, positive=False):
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:152: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  precompute=False, eps=np.finfo(np.float).eps,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:318: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  eps=np.finfo(np.float).eps, random_state=None,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:575: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  eps=4 * np.finfo(np.float).eps, n_jobs=1,
/usr/local/miniconda/lib/python3.7/site-packages/sklearn/decomposition/online_lda.py:31: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  EPS = np.finfo(np.float).eps
Traceback (most recent call last):
  File "/src/run.py", line 232, in <module>
    sys.exit(main())
  File "/src/run.py", line 161, in main
    space=query['space'],
KeyError: 'space'
arovai commented 3 years ago

The error above comes from the fact that --space must also be specified. So adding e.g. --space MNI152NLin6Asym to the command seems to do the job.

So in summary, (almost) from scratch:

git clone https://github.com/poldracklab/ds003-post-fMRIPrep-analysis
sudo docker build -t ds003-post-fmriprep-analysis-test:latest .
datalad install ///labs/poldrack/ds003_fmriprep
datalad get ds003_fmriprep/
datalad install ///openfmri/ds000003
datalad get ds000003/
mv ds003_fmriprep/ ds000003/derivatives/fmriprep
sudo docker run -v $PWD/ds000003:/home/bidsapp/bids_dir -v $PWD/output:/home/bidsapp/output ds003-post-fmriprep-analysis-test:latest /home/bidsapp/bids_dir/derivatives /home/bidsapp/output participant --bids-dir /home/bidsapp/bids_dir --space MNI152NLin6Asym

The group analysis also run without errors (although there is a bunch of FutureWarnings):

sudo docker run -v $PWD/../ds0003_bidsdir:/home/bidsapp/bids_dir -v $PWD/../ds0003_bidsdir/output:/home/bidsapp/output ds003-post-fmrirep-analysis:latest /home/bidsapp/bids_dir/derivatives /home/bidsapp/output group --bids-dir /home/bidsapp/bids_dir --space MNI152NLin6Asym

Viewing the results using the jupyter notebook has only one problem, namely that the 3D render are black, as if it failed to fetch fsaverage (although no errors are produced):

fsaveblack