search

Mouse-Imaging-Centre / pydpiper

Python code for flexible pipeline control
Other
24 stars 10 forks source link

v2.0.13 crashes attempting to merge analysis.csv in twolevel_model_building.py #426

Closed gdevenyi closed 4 years ago

gdevenyi commented 5 years ago

This commit: https://github.com/Mouse-Imaging-Centre/pydpiper/commit/ba79d3633d4bf0f5519ceb5ddedc36334493f91d

Causes this crash:

Traceback (most recent call last):
  File "/opt/quarantine/pydpiper/2.0.13/install/bin/twolevel_model_building.py", line 4, in <module>
    __import__('pkg_resources').run_script('pydpiper==2.0.13', 'twolevel_model_building.py')
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 750, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1527, in run_script
    exec(code, namespace, namespace)
  File "/opt/quarantine/pydpiper/2.0.13/install/lib/python3.6/site-packages/pydpiper-2.0.13-py3.6.egg/EGG-INFO/scripts/twolevel_model_building.py", line 307, in <module>
    main(sys.argv)
  File "/opt/quarantine/pydpiper/2.0.13/install/lib/python3.6/site-packages/pydpiper-2.0.13-py3.6.egg/EGG-INFO/scripts/twolevel_model_building.py", line 304, in main
    execute(two_level_pipeline(options).stages, options)
  File "/opt/quarantine/pydpiper/2.0.13/install/lib/python3.6/site-packages/pydpiper-2.0.13-py3.6.egg/EGG-INFO/scripts/twolevel_model_building.py", line 93, in two_level_pipeline
    (analysis.merge(pd.concat([resampled, overall], axis=1)).to_csv("analysis.csv", index=False))
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pandas/core/frame.py", line 6389, in merge
    copy=copy, indicator=indicator, validate=validate)
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 61, in merge
    validate=validate)
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 546, in __init__
    self._validate_specification()
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 1035, in _validate_specification
    lidx=self.left_index, ridx=self.right_index))
pandas.errors.MergeError: No common columns to perform merge on. Merge options: left_on=None, right_on=None, left_index=False, right_index=False
bcdarwin commented 4 years ago

Can't reproduce ... do you have example command/csv ?

gdevenyi commented 4 years ago

As always I'm the conduit reporting one of my user's problems. I fixed it by changing versions I think. Will try to remember who reported it and see if we still have the failing config around.

gdevenyi commented 4 years ago

My memory of how this was triggered is long gone as is whatever voodoo I did to get around it.

nzxwang commented 4 years ago

I got the same error AFTER Ben's afe83d0e43b415a6ec5366d49a42ede69dcc8ba5 currently working on it

nzxwang commented 4 years ago

I got the same error AFTER Ben's afe83d0 currently working on it

It has nothing to do with afe83d0e43b415a6ec5366d49a42ede69dcc8ba5. The merge is implicitly meant to happen on column native_file which is generated https://github.com/Mouse-Imaging-Centre/pydpiper/blob/2b19c0cf3b922b52d8bc4e43f37f82a9d9438244/pydpiper/pipelines/twolevel_model_building.py#L81 but only exists in the other dataframe if maget is run.

bcdarwin commented 4 years ago

We should probably make the join column explicit.

nzxwang commented 4 years ago

We should probably make the join column explicit.

agreed and done