con / opfvta-reexecution

Container-based Replication of https://doi.org/10.1038/s41398-022-01812-5
Apache License 2.0
1 stars 1 forks source link

Interactive container working differently than automatic execution #69

Closed TheChymera closed 8 months ago

TheChymera commented 8 months ago

As a short preface, the interactive container is not really needed for this study, it's just a shell entry point with possible utility for debugging. This issue came to my attention via a user relying on the opfvta-reexecution container as a way to gain containerized access to SAMRI (since there's no SAMRI container yet).

SAMRI uses nipype, and as part of running the preprocessing analysis nipype is imported. That works fine (at the beginning I quit as soon as the preporcessing started, but it would continue going). However, if I enter the container and try to import either nipype or analogous submodules of nipype which SAMRI imports, that fails:

[deco]~/docsrc/opfvta-reexecution ❱ make analysis-oci OCI_BINARY=podman
podman run \
    -it \
    --rm \
    -v /home/chymera/docsrc/opfvta-reexecution:/opt \
    -v /home/chymera/docsrc/opfvta-reexecution/inputs/opfvta_bidsdata:/usr/share/opfvta_bidsdata \
    -v /home/chymera/docsrc/opfvta-reexecution/inputs/mouse-brain-templates/mouse-brain-templates:/usr/share/mouse_brain_atlases \
    -v /home/chymera/docsrc/opfvta-reexecution/scratch:/root/.scratch \
    -v /home/chymera/docsrc/opfvta-reexecution/code/empty:/opfvta/ \
    --workdir /opt/code \
    docker.io/centerforopenneuroscience/opfvta:2.0.0-alpha \
    ./produce-analysis.sh podman
/opt/code/opfvta /opt/code
rubber --clean article.tex
removing tree pythontex-files-article
removing tree pythontex-files-article
rm _minted-article -rf
rubber --clean pitch.tex
removing tree pythontex-files-pitch
removing tree pythontex-files-pitch
rm _minted-pitch -rf
rubber --clean review.tex
removing tree pythontex-files-review
rm _minted-review -rf
pushd prepare && ./run.sh
/opt/code/opfvta/prepare /opt/code/opfvta
!!! Repository 'gentoo' has sync-uri attribute, but is missing sync-type attribute
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:283: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:296: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
/usr/lib/python3.10/site-packages/samri/pipelines/utils.py:335: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  res_df = res_df.append(_df)
                                                   path modality subject session   task acquisition        acq  run  type scan_type
1365  /root/.scratch/opfvta/bids/sub-6257/ses-ofM/fu...      cbv    6257     ofM   CogB         EPI        EPI    0  func
1367  /root/.scratch/opfvta/bids/sub-6257/ses-ofM/an...      T2w    6257     ofM    NaN   TurboRARE  TurboRARE  NaN  anat
225   /root/.scratch/opfvta/bids/sub-6259/ses-ofM/fu...      cbv    6259     ofM   CogB         EPI        EPI    0  func
230   /root/.scratch/opfvta/bids/sub-6259/ses-ofMr1/...      cbv    6259   ofMr1  JPogP         EPI        EPI    0  func
232   /root/.scratch/opfvta/bids/sub-6259/ses-ofMr1/...      cbv    6259   ofMr1   CogB         EPI        EPI    1  func
...                                                 ...      ...     ...     ...    ...         ...        ...  ...   ...       ...
1350  /root/.scratch/opfvta/bids/sub-6654/ses-ofMr1/...      T2w    6654   ofMr1    NaN   TurboRARE  TurboRARE  NaN  anat
265   /root/.scratch/opfvta/bids/sub-6655/ses-ofM/fu...      cbv    6655     ofM   CogB         EPI        EPI    0  func
269   /root/.scratch/opfvta/bids/sub-6655/ses-ofMr1/...      cbv    6655   ofMr1  CogBr         EPI        EPI    0  func
267   /root/.scratch/opfvta/bids/sub-6655/ses-ofM/an...      T2w    6655     ofM    NaN   TurboRARE  TurboRARE  NaN  anat
272   /root/.scratch/opfvta/bids/sub-6655/ses-ofMr1/...      T2w    6655   ofMr1    NaN   TurboRARE  TurboRARE  NaN  anat

[134 rows x 10 columns]
[['6257', 'ofM'], ['6259', 'ofM'], ['6259', 'ofMr1'], ['6260', 'ofMr2'], ['6260', 'ofM'], ['5682', 'ofMr2'], ['5682', 'ofMr1'], ['5682', 'ofMr3'], ['5683', 'ofMr4'], ['5683', 'ofM'], ['5683', 'ofMr1'], ['5683', 'ofMr3'], ['5685', 'ofM'], ['5685', 'ofMr1'], ['6467', 'ofM'], ['6468', 'ofM'], ['6471', 'ofM'], ['6471', 'ofMr1'], ['6560', 'ofM'], ['6560', 'ofMr1'], ['6562', 'ofM'], ['6562', 'ofMr1'], ['6573', 'ofM'], ['6573', 'ofMr1'], ['6574', 'ofM'], ['6574', 'ofMr1'], ['6575', 'ofM'], ['6575', 'ofMr1'], ['6576', 'ofM'], ['6576', 'ofMr1'], ['6578', 'ofM'], ['6578', 'ofMr1'], ['6582', 'ofMr1'], ['6582', 'ofM'], ['6586', 'ofM'], ['6587', 'ofM'], ['6587', 'ofMr1'], ['6588', 'ofM'], ['6588', 'ofMr1'], ['6589', 'ofM'], ['6589', 'ofMr1'], ['6590', 'ofM'], ['6591', 'ofM'], ['6592', 'ofM'], ['6592', 'ofMr1'], ['6593', 'ofM'], ['3839', 'ofM'], ['6639', 'ofM'], ['6643', 'ofM'], ['6642', 'ofM'], ['6654', 'ofM'], ['6654', 'ofMr1'], ['6655', 'ofM'], ['6655', 'ofMr1']]
240227-18:26:48,280 nipype.workflow INFO:
     Generated workflow graph: /root/.scratch/opfvta/preprocess_work/graph.png (graph2use=hierarchical, simple_form=True).
240227-18:26:48,303 nipype.workflow INFO:
     Workflow preprocess_work settings: ['check', 'execution', 'logging', 'monitoring']
^CTraceback (most recent call last):
  File "/opt/code/opfvta/prepare/preprocess.py", line 17, in <module>
    generic(bids_base,
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 637, in generic
    workflow.run(plugin="MultiProc", plugin_args={'n_procs' : n_jobs})
  File "/usr/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 637, in run
    self._write_report_info(self.base_dir, self.name, execgraph)
  File "/usr/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 699, in _write_report_info
    dict(source=nodes.index(u), target=nodes.index(v), value=1)
KeyboardInterrupt
make: *** [Makefile:55: data] Interrupt
make: *** [Makefile:76: analysis-oci] Error 130
[deco]~/docsrc/opfvta-reexecution ❱ make analysis-oci-interactive
docker run \
    -it \
    --rm \
    -v /home/chymera/docsrc/opfvta-reexecution:/opt \
    -v /home/chymera/docsrc/opfvta-reexecution/inputs/opfvta_bidsdata:/usr/share/opfvta_bidsdata \
    -v /home/chymera/docsrc/opfvta-reexecution/inputs/mouse-brain-templates/mouse-brain-templates:/usr/share/mouse_brain_atlases \
    -v /home/chymera/docsrc/opfvta-reexecution/scratch:/root/.scratch \
    -v /home/chymera/docsrc/opfvta-reexecution/code/empty:/opfvta/ \
    --workdir /opt/code \
    docker.io/centerforopenneuroscience/opfvta:2.0.0-alpha \
    /bin/bash
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
make: *** [Makefile:89: analysis-oci-interactive] Error 125
[deco]~/docsrc/opfvta-reexecution ❱ make analysis-oci-interactive OCI_BINARY=podman
podman run \
    -it \
    --rm \
    -v /home/chymera/docsrc/opfvta-reexecution:/opt \
    -v /home/chymera/docsrc/opfvta-reexecution/inputs/opfvta_bidsdata:/usr/share/opfvta_bidsdata \
    -v /home/chymera/docsrc/opfvta-reexecution/inputs/mouse-brain-templates/mouse-brain-templates:/usr/share/mouse_brain_atlases \
    -v /home/chymera/docsrc/opfvta-reexecution/scratch:/root/.scratch \
    -v /home/chymera/docsrc/opfvta-reexecution/code/empty:/opfvta/ \
    --workdir /opt/code \
    docker.io/centerforopenneuroscience/opfvta:2.0.0-alpha \
    /bin/bash
f3444f102fb7 /opt/code # python
Python 3.10.10 (main, Mar 26 2023, 17:29:04) [GCC 12.2.1 20230121] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nipype
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/site-packages/nipype/__init__.py", line 101, in <module>
    BaseInterface._etelemetry_version_data = check_latest_version() or "n/a"
  File "/usr/lib/python3.10/site-packages/nipype/__init__.py", line 85, in check_latest_version
    import etelemetry
ModuleNotFoundError: No module named 'etelemetry'

Any ideas what could be going on @yarikoptic @asmacdo

asmacdo commented 8 months ago

My first guess is this looks like a nipype bug to me, theres some unusual magic happening in nipype with etelemetry

https://github.com/nipy/nipype/blob/master/nipype/__init__.py#L75-L89

Maybe try setting NIPYPE_NO_ET

https://github.com/nipy/nipype/blob/056b48f23c896bed0fce1c571f33bb01922a1713/nipype/__init__.py#L96C47-L96C59

asmacdo commented 8 months ago

Since nipype is doing some unusual load logic, maybe try import nipype.interfaces.io as nio like how samri does it.

https://github.com/IBT-FMI/SAMRI/blob/1f1443973d1bcf123aa81456d80d27863d8b8d71/samri/pipelines/preprocess.py#L12C1-L12C35

yarikoptic commented 8 months ago

sorry -- too much text.. what is the problem question exactly?

meanwhile followed up on https://github.com/IBT-FMI/SAMRI/issues/119#issuecomment-1967917509

TheChymera commented 8 months ago

@asmacdo oh god. Thank you so much for bringing up that variable, I now remember everything. So yes, nothing is broken, I just forgot some finer points regarding how to use it:

f3444f102fb7 /opt/code # source /etc/profile
f3444f102fb7 /opt/code # python
Python 3.10.10 (main, Mar 26 2023, 17:29:04) [GCC 12.2.1 20230121] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nipype.interfaces import fsl
>>>
KeyboardInterrupt
>>>
f3444f102fb7 /opt/code # cat /etc/env.d/98nipype
export NIPYPE_NO_ET=1

We need to run source /etc/profile for NIPYPE_NO_ET to be loaded. Our produce_analysis.sh script has that command.

But now that I hit this snag (again), maybe we could do something about it? On a normal Gentoo system, you only need that if you want to have the environment variables loaded in currently open sessions. Each new session will auto-load the environment variables on startup via source /etc/profile. Any idea why OCI images don't auto-do it when you start your shell?

asmacdo commented 8 months ago

According to ChatGPT

Invoking /bin/bash without -l (or --login) starts a non-login shell. Non-login shells do not automatically source /etc/profile or the user's .profile, .bash_profile, etc. They typically source ~/.bashrc.

So I guess you could add -l and see if that does it

TheChymera commented 8 months ago

@asmacdo thanks, can confirm https://github.com/con/opfvta-reexecution/commit/61d09b98defae91c5669706b49656a6fd3726340 fixes it 💛