search

Mouse-Imaging-Centre / pydpiper

Python code for flexible pipeline control
Other
24 stars 10 forks source link

pydpiper/2.0.12 MAGeT.py with outputdir fails to start, can't find segmentations.csv #403

Open gdevenyi opened 5 years ago

gdevenyi commented 5 years ago 
MAGeT.py --files input/*mnc \
  --num-executors 350 \
  --output-dir testoutputdir \
  --resolution 0.1 \
  --subject-matter mousebrain \
  --queue-type sge \
  --check-input-files \
  --check-outputs \
  --lsq12-protocol /opt/quarantine/pydpiper/2.0.12/src/pydpiper-2.0.12/applications_testing/test_data/minctracc_example_linear_protocol.csv \
  --atlas-library /opt/quarantine/resources/Dorr_2008_Steadman_2013_Ullmann_2013_Richards_2011_Qiu_2016_Egan_2015_40micron/ex-vivo \
  --masking-method minctracc \
  --registration-method minctracc \
  --masking-nlin-protocol /opt/quarantine/pydpiper/2.0.12/src/pydpiper-2.0.12/applications_testing/test_data/default_nlin_MAGeT_minctracc_prot.csv \
  --nlin-protocol /opt/quarantine/pydpiper/2.0.12/src/pydpiper-2.0.12/applications_testing/test_data/default_nlin_MAGeT_minctracc_prot.csv

Error:

Traceback (most recent call last):
  File "/opt/quarantine/pydpiper/2.0.12/install/bin/MAGeT.py", line 4, in <module>
    __import__('pkg_resources').run_script('pydpiper==2.0.12', 'MAGeT.py')
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 750, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1527, in run_script
    exec(code, namespace, namespace)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MAGeT.py", line 591, in <module>
    maget_application()
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/pydpiper/execution/application.py", line 206, in f
    execute(pipeline(options).stages, options)
  File "/opt/quarantine/pydpiper/2.0.12/install/lib/python3.6/site-packages/pydpiper-2.0.12-py3.6.egg/EGG-INFO/scripts/MAGeT.py", line 534, in maget_pipeline
    .to_csv(os.path.join(options.application.output_directory, "segmentations.csv"), index=False))
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pandas/core/frame.py", line 1745, in to_csv
    formatter.save()
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pandas/io/formats/csvs.py", line 156, in save
    compression=self.compression)
  File "/opt/quarantine/anaconda/5.1.0-python3/install/lib/python3.6/site-packages/pandas/io/common.py", line 400, in _get_handle
    f = open(path_or_buf, mode, encoding=encoding)
FileNotFoundError: [Errno 2] No such file or directory: 'testoutputdir/segmentations.csv'

Runs properly if --outputdir is removed.

bcdarwin commented 5 years ago

Does this happen if testoutputdir has been created previously?

gdevenyi commented 5 years ago

Things are okay if testoutputdir exists

nzxwang commented 5 years ago

@bcdarwin we still need to decide if we will fix or remove --output-dir. I recall that much of the pydpiper vomit ignores it, and many pydpiper functions ignore it which causes an error along the lines of "trying to create a file outside of output directory".

nzxwang commented 4 years ago

@bcdarwin we still need to decide if we will fix or remove --output-dir. I recall that much of the pydpiper vomit ignores it, and many pydpiper functions ignore it which causes an error along the lines of "trying to create a file outside of output directory".

We plan on temporarily removing the --output-dir feature because of how much broken it is, with the intention of re-implementing in the future when pydpiper does a better job of storing its files internally.

bcdarwin commented 4 years ago

Some pipelines (e.g. MAGeT) are now broken. Also, I don't think os.getcwd is the right answer to this since now all paths are absolute, i.e., you can't relocate the pipeline (and relative input file deps) freely, which is slightly annoying. Probably best to revert this commit and just fix options.output_dir at a single place?