search

Mouse-Imaging-Centre / pydpiper

Python code for flexible pipeline control
Other
24 stars 10 forks source link

pydpiper 2.0.12 won't run with defaults (missing configured default protocols) #390

Open gdevenyi opened 5 years ago

gdevenyi commented 5 years ago

Updating my default configs to see how things have changed, My runscript:

MBM.py --files input/*mnc \
  --output-dir testoutputdir \
  --resolution 0.1 \
  --subject-matter mousebrain \
  --queue-type sge \
  --check-input-files \
  --check-outputs \
  --init-model /opt/quarantine/resources/Dorr_2008_Steadman_2013_Ullmann_2013_Richards_2011_Qiu_2016_Egan_2015_40micron/ex-vivo/DSURQE_40micron.mnc \
  --lsq6-large-rotations-parameters mousebrain \
  --no-nuc \
  --maget-atlas-library /opt/quarantine/resources/Dorr_2008_Steadman_2013_Ullmann_2013_Richards_2011_Qiu_2016_Egan_2015_40micron/ex-vivo

Results in the following error:

bash run-MBM.sh 

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Traceback (most recent call last):
  File "/opt/quarantine/pydpiper/2.0.12/install/bin/MBM.py", line 4, in <module>
    __import__('pkg_resources').run_script('pydpiper==2.0.12', 'MBM.py')
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 661, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1441, in run_script
    exec(code, namespace, namespace)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 532, in <module>
    mbm_application()
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/execution/application.py", line 206, in f
    execute(pipeline(options).stages, options)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 49, in mbm_pipeline
    output_dir=output_dir))
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 268, in mbm
    lsq12_options=maget_options.mbm.lsq12)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/pipelines/MAGeT.py", line 319, in fixup_maget_options
    raise ValueError(format(error_string))
ValueError: 

The MAGeT non-linear masking protocol (--maget-masking-nlin-protocol) was not specified. Tried to use the non-linear protocol for the main part of MAGeT (--maget-nlin-protocol), but the registration methods for masking (--maget-masking-method) and the main MAGeT procedure (--maget-registration-method) are different. (minctracc vs ANTS)

Defaults are:
--maget-masking-method minctracc
--maget-registration-method minctracc
--maget-masking-nlin-protocol default_nlin_MAGeT_minctracc_prot.csv
--maget-nlin-protocol default_nlin_MAGeT_minctracc_prot.csv

Which can be found in either of these two locations:
/hpf/largeprojects/MICe/tools/protocols/nonlinear/
/axiom2/projects/software/non-linear-protocol/

So, this error is specially coded somewhere, referring to MiCe Stuff.

Can the pydpiper defaults be configured so this is just the case?

gdevenyi commented 5 years ago

And a followup, after specifying the configuration as per the error message, now it fails with:

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Traceback (most recent call last):
  File "/opt/quarantine/pydpiper/2.0.12/install/bin/MBM.py", line 4, in <module>
    __import__('pkg_resources').run_script('pydpiper==2.0.12', 'MBM.py')
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 661, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1441, in run_script
    exec(code, namespace, namespace)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 532, in <module>
    mbm_application()
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/execution/application.py", line 206, in f
    execute(pipeline(options).stages, options)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 49, in mbm_pipeline
    output_dir=output_dir))
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 306, in mbm
    "%s_atlases" % prefix))))
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/pipelines/MAGeT.py", line 173, in maget_mask
    resolution)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py", line 1371, in get_linear_configuration_from_options
    raise NotImplementedError(error_message)
NotImplementedError: 

Error while retrieving the linear (lsq12) minctracc configuration (flag: --maget-lsq12-protocol). No protocol was specified and no defaults are currently in place.
bcdarwin commented 5 years ago

I think this situation where you have to explicitly specify protocols is also true of the previous version though, no?

Overhaul of this situation is on our list for the next release ...

gdevenyi commented 5 years ago

Yes, this is not a big change from previous versions

This is more of a note that in its current state, MBM.py will not run because MAGeT is turned on by default, and the defaults configurations are not available.

Is the MAGeT lsq12 protocol included in the package right now? That error (unlike the earlier one) doesn't point me at a file.

bcdarwin commented 5 years ago

pydpiper/applications_testing/test_data/default_linear_MAGeT_prot.csv

gdevenyi commented 5 years ago

Another round, after resolving #387 

bash run-MBM.sh 

Warning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 [/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 [/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: msgpack serializer unavailable. requires msgpack 0.5.2+, found (0, 5, 1) [/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/Pyro4/util.py:839]

Warning: Note: the 'simplex' parameter is not used for non-linear minctracc registrations. It's specified in the protocol, but won't have any effect. [/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py:1274]

Warning: Note: the 'w_translations' parameter is not used for non-linear minctracc registrations. It's specified in the protocol, but won't have any effect. [/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py:1274]

Traceback (most recent call last):
  File "/opt/quarantine/pydpiper/2.0.12/install/bin/MBM.py", line 4, in <module>
    __import__('pkg_resources').run_script('pydpiper==2.0.12', 'MBM.py')
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 750, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1527, in run_script
    exec(code, namespace, namespace)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 532, in <module>
    mbm_application()
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/execution/application.py", line 206, in f
    execute(pipeline(options).stages, options)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 49, in mbm_pipeline
    output_dir=output_dir))
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 373, in mbm
    nlin_conf=nlin_conf))  #options.mbm.nlin
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py", line 2016, in lsq12_nlin_build_model
    lsq12_dir=lsq12_dir))
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py", line 1920, in lsq12_pairwise
    file_resolution=resolution)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py", line 1371, in get_linear_configuration_from_options
    raise NotImplementedError(error_message)
NotImplementedError: 

Error while retrieving the linear (lsq12) minctracc configuration (flag: --lsq12-protocol). No protocol was specified and no defaults are currently in place.

Is this file included?

bcdarwin commented 5 years ago

pydpiper/applications_testing/test_data/minctracc_example_linear_protocol.csv

but again this was this case in all 2.x versions.

gdevenyi commented 5 years ago

Yes, this has been the case in all my previous configs.

I'm just actually reporting all the things that don't work on a default install (like for example MINC-VM's install which as of right now, would need all this hackery to get working as well)

gdevenyi commented 5 years ago

And now:

You have specified a different lsq12 protocol for MAGeT as compared to the lsq12 protocol for the main registration: 

--maget-lsq12-protocol /opt/quarantine/pydpiper/2.0.12/src/pydpiper-2.0.12/applications_testing/test_data/default_linear_MAGeT_prot.csv
--lsq12-protocol /opt/quarantine/pydpiper/2.0.12/src/pydpiper-2.0.12/applications_testing/test_data/minctracc_example_linear_protocol.csv

One lsq12 protocol can be used for both parts of the pipeline. Please decide which of the two you want to use, and specify only that one using the --lsq12-protocol flag.