search

Mouse-Imaging-Centre / pydpiper

Python code for flexible pipeline control
Other
25 stars 10 forks source link

Python's builtin typing version 3.6.6 breaks get_model_building_procedure #387

Closed nzxwang closed 5 years ago

nzxwang commented 5 years ago

For the sake of releasing pydpiper/2.0.12, we require typing<=3.6.4 in setup.py as a temporary workaround.

[nzxwang@gra-login3 MBM]$ MBM.py --pipeline-name=MBM_test --subject-matter mousebrain --num-executors 1000 --time 48:00:00 --csv-file test-data_20180925/input.csv --lsq6-large-rotations-tmp-dir=/tmp --init-model test-data_20180925/Pydpiper-40-micron-basket-dec-2014/basket_mouse_brain_40micron.mnc --run-maget --maget-registration-method minctracc --maget-atlas-library test-data_20180925/ex-vivo/ --maget-nlin-protocol test-data_20180925/default_nlin_MAGeT_minctracc_prot.csv --maget-masking-nlin-protocol test-data_20180925/default_nlin_MAGeT_minctracc_prot.csv --lsq12-protocol test-data_20180925/Pydpiper_testing_default_lsq12.csv

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.6.3/lib/python3.6/importlib/_bootstrap.py:219]

Warning: Note: the 'simplex' parameter is not used for non-linear minctracc registrations. It's specified in the protocol, but won't have any effect. [/project/def-jlerch/tools/programs/pydpiper/2.0.12/lib/python/pydpiper-2.0.11-py3.6.egg/pydpiper/minc/registration.py:1274]

Warning: Note: the 'w_translations' parameter is not used for non-linear minctracc registrations. It's specified in the protocol, but won't have any effect. [/project/def-jlerch/tools/programs/pydpiper/2.0.12/lib/python/pydpiper-2.0.11-py3.6.egg/pydpiper/minc/registration.py:1274]

Traceback (most recent call last):
  File "/project/def-jlerch/tools/programs/pydpiper/2.0.12/bin/MBM.py", line 4, in <module>
    __import__('pkg_resources').run_script('pydpiper==2.0.11', 'MBM.py')
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.6.3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 743, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.6.3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1498, in run_script
    exec(code, namespace, namespace)
  File "/project/6001494/tools/programs/pydpiper/2.0.12/lib/python/pydpiper-2.0.11-py3.6.egg/EGG-INFO/scripts/MBM.py", line 532, in <module>
    mbm_application()
  File "/project/def-jlerch/tools/programs/pydpiper/2.0.12/lib/python/pydpiper-2.0.11-py3.6.egg/pydpiper/execution/application.py", line 206, in f
    execute(pipeline(options).stages, options)
  File "/project/6001494/tools/programs/pydpiper/2.0.12/lib/python/pydpiper-2.0.11-py3.6.egg/EGG-INFO/scripts/MBM.py", line 49, in mbm_pipeline
    output_dir=output_dir))
  File "/project/6001494/tools/programs/pydpiper/2.0.12/lib/python/pydpiper-2.0.11-py3.6.egg/EGG-INFO/scripts/MBM.py", line 334, in mbm
    reg_module=nlin_module)
  File "/project/def-jlerch/tools/programs/pydpiper/2.0.12/lib/python/pydpiper-2.0.11-py3.6.egg/pydpiper/minc/registration_strategies.py", line 476, in get_model_building_procedure
    return f(reg_module)
  File "/project/def-jlerch/tools/programs/pydpiper/2.0.12/lib/python/typing-3.6.6-py3.6.egg/typing.py", line 1232, in __new__
    return _generic_new(cls.__next_in_mro__, cls, *args, **kwds)
  File "/project/def-jlerch/tools/programs/pydpiper/2.0.12/lib/python/typing-3.6.6-py3.6.egg/typing.py", line 1189, in _generic_new
    return base_cls.__new__(cls, *args, **kwds)
TypeError: object() takes no parameters
bcdarwin commented 5 years ago

It's probably related to some of the weird ABCMeta stuff in this module, which might make sense to remove in favour of data classes (IIRC the types are no longer correct anyway, so ...)

gdevenyi commented 5 years ago

I get warnings but I have the specified versions


Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]
bcdarwin commented 5 years ago

I also get these warnings but I don't know why. They seem to be harmless.

From: Gabriel A. Devenyi notifications@github.com Sent: November 9, 2018 11:49:39 AM To: Mouse-Imaging-Centre/pydpiper Cc: Ben Darwin; Comment Subject: Re: [Mouse-Imaging-Centre/pydpiper] Python's builtin typing version 3.6.6 breaks get_model_building_procedure (#387)

I get warnings but I have the specified versions

Warning: can't resolve package from spec or package, falling back on name and path [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: can't resolve package from spec or package, falling back on name and path [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Mouse-2DImaging-2DCentre_pydpiper_issues_387-23issuecomment-2D437421041&d=DwMFaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=RVvUWkVE6prpUsksCnMiI9NpYZfxHQz1YEvRqlDM6NU&s=dzXALTc7iW7aLpe-CGMtvuF2AwSwp9ZZiTr8mj0x7cs&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAKBNFIdd7YMKjKs2KM-2DZgZyo8mhaJoIks5utbIjgaJpZM4YVElc&d=DwMFaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=RVvUWkVE6prpUsksCnMiI9NpYZfxHQz1YEvRqlDM6NU&s=njT0F-ZvkzmLazT9h_XRRETtx3cQG47ITkitzX_FEG0&e=.

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.

gdevenyi commented 5 years ago

Agree, seem to be harmless, things seem to work.

gdevenyi commented 5 years ago

So, after saying that.. fixing all the bits in #390, I now trigger this bug:

bash run-MBM.sh 

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ [/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/importlib/_bootstrap.py:219]

Warning: Note: the 'simplex' parameter is not used for non-linear minctracc registrations. It's specified in the protocol, but won't have any effect. [/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py:1274]

Warning: Note: the 'w_translations' parameter is not used for non-linear minctracc registrations. It's specified in the protocol, but won't have any effect. [/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration.py:1274]

Traceback (most recent call last):
  File "/opt/quarantine/pydpiper/2.0.12/install/bin/MBM.py", line 4, in <module>
    __import__('pkg_resources').run_script('pydpiper==2.0.12', 'MBM.py')
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 661, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1441, in run_script
    exec(code, namespace, namespace)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 532, in <module>
    mbm_application()
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/execution/application.py", line 206, in f
    execute(pipeline(options).stages, options)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 49, in mbm_pipeline
    output_dir=output_dir))
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MBM.py", line 334, in mbm
    reg_module=nlin_module)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/minc/registration_strategies.py", line 476, in get_model_building_procedure
    return f(reg_module)
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/typing.py", line 1231, in __new__
    return _generic_new(cls.__next_in_mro__, cls, *args, **kwds)
  File "/opt/quarantine/anaconda/5.2.0-python3/install/lib/python3.6/typing.py", line 1188, in _generic_new
    return base_cls.__new__(cls, *args, **kwds)
TypeError: object() takes no parameters
python -c "import pkg_resources; print(pkg_resources.get_distribution('typing').version)"
3.6.4
bcdarwin commented 5 years ago

This is a library issue of some sort. Chris Percival had this problem trying to install 2.0.11 on his local system, but when I built my fork of Vlad's singularity container recipe with that version it worked fine.

bcdarwin commented 5 years ago

Any luck with this? If Python installs are randomly broken with no clear workaround then I should work on this right away.

gdevenyi commented 5 years ago

Nothing so far, can you point me at your container config so I can compare what I have here?

gdevenyi commented 5 years ago

I believe I've found the problem, see: https://github.com/python/typing/blob/master/setup.py#L26-L35

My python version:

$ python --version
Python 3.6.6 :: Anaconda custom (64-bit)

So, I cannot downgrade to a lower version of typing.

2.0.12 pydpiper can only run on python 3.6.4 or earlier, with typing pypy version<3.6.4

bcdarwin commented 5 years ago

Unfortunate ... I'll look into this.

bcdarwin commented 5 years ago

Nothing so far, can you point me at your container config so I can compare what I have here?

https://github.com/Mouse-Imaging-Centre/minc-toolkit-containers/blob/master/singularity/Singularity.1.9.16

gdevenyi commented 5 years ago

I will quarantine up an anaconda 5.1.0 and reinstall and see if that resolves it.

gdevenyi commented 5 years ago

5.1.0(python 3.6.4) doesn't throw the typing error.

bcdarwin commented 5 years ago

This change might be somewhat invasive so I'd be a bit reluctant to re-release right away. Are you OK with running with this slightly older Python version for a bit?

gdevenyi commented 5 years ago

That's fine with me, I was just building out a new anaconda quarantine for all my python stuff so I will just not go to 5.2.0 and only upgrade to 5.1.0

bcdarwin commented 5 years ago

Actually fixed in 3c53c854a063 .