search

jstout211 / enigma_MEG

Other
9 stars 1 forks source link

fsaverage directory replaced with sym link #56

Open Lychfindel opened 5 months ago

Lychfindel commented 5 months ago

Hi, while running the code I'm receiving the following error:

ERROR:0001_ses_20180410_task_MEGLabRestingEO_run_01:do_make_aparc_sub :: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/bem'
Traceback (most recent call last):
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/bin/process_meg.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1775, in <module>
    process_subject(args.subject, args)  # process the single specified subject
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1440, in process_subject
    proc.do_proc_allsteps()
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1067, in do_proc_allsteps
    self.do_make_aparc_sub()
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 86, in wrapper
    output = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 800, in do_make_aparc_sub
    write_aparc_sub(subjid=f'sub-{self.subject}', 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1315, in write_aparc_sub
    mne.datasets.fetch_fsaverage(verbose='ERROR') #True requires TQDM
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<decorator-gen-481>", line 10, in fetch_fsaverage
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/mne/datasets/_fsaverage/base.py", line 71, in fetch_fsaverage
    _manifest_check_download(
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/mne/datasets/utils.py", line 800, in _manifest_check_download
    ff.extract(name, path=destination)
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/zipfile.py", line 1676, in extract
    return self._extract_member(member, path, pwd)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/zipfile.py", line 1739, in _extract_member
    os.makedirs(upperdirs)
  File "<frozen os>", line 225, in makedirs
PermissionError: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/bem'

As far as I understand the error is due to the fact that fsaverage directory is a symlink. I investigated a little bit and I noticed these lines in process_meg.py:

    # We have to find out if there is an fsaverage in the freesurfer directory, and if there is, if 
    # it is a sym link. If it's a link, it will break later when we try to get the bem directory

    if os.path.isdir(os.path.join(bids_root,'derivatives/freesurfer/subjects/fsaverage')):
        if os.path.islink(os.path.join(bids_root,'derivatives/freesurfer/subjects/fsaverage')):
            raise ValueError('$SUBJECTS_DIR/fsaverage cannot be a symlink; remove and rerun process_MEG to fetch data')

However in my case fsaverage is downloaded at the beginning of the script, and it is a new real folder as expected. I run the script putting a breakpoint after each single step of do_proc_allsteps and I found out that the folder is replaced with a sym link of fsaverage installed in my computer during the step proc_mri.

nugenta commented 4 months ago

Hi - this is very odd. We are having trouble replicating this. What version of freesurfer are you using?

Lychfindel commented 4 months ago

I'm using FreeSurfer 7.4.1 on Ubuntu22. The full version name is freesurfer-linux-ubuntu22_x86_64-7.4.1-20230614-7eb8460

jstout211 commented 4 months ago

It looks like a permission error on the fsaverage folder. It is odd given that you had permission to download fsaverage (??): PermissionError: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/bem'

Just to check that this isnt' using your system SUBJECTS_DIR variable - can you confirm that this freesurfer directory is the one associated with the Enigma processing (MNE will use the environmental var if none is provided - and that would be a bug in our code)

If it is the one associated with the BIDS dir processing the Enigma code, can you do the following: chmod -R a+rw /mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage And you will also need to chmod the .../subjects/morph-maps folder (this will be created several steps later).

If you run into the issue again - can you add the full commandline call to process_meg.py in the comment, so we can investigate further. You can replace any sensitive information (paths/subjids) with alternative characters. This will help us better troubleshoot the issue.

Thanks for providing the info.

Lychfindel commented 4 months ago

Hi Jeff, thanks for the explanation. The permission error arise only after the step proc_mri: before this step the folder has full read/write permissions, but during this step the directory fsaverage is replaced with a symlink to my local folder /usr/local/freesurfer/7.4.1/subjects/fsaverage which has write permissions only for root user.

The command line I'm using is:

process_meg.py -bids_root /mnt/raid/atonin/bids/Protocollone -subject sub-0001 -session 20180410 -mains 50 -n_jobs 32 -run 01 -rest_tag MEGLabRestingEO -emptyroom_tag SysTest

But the $SUBJECTS_DIR variable points to my local directory:

echo $SUBJECTS_DIR
> /home/alessandro.tonin/subjects
jstout211 commented 4 months ago

Looking into this currently - will try to have a fix for you soon.

Thanks

jstout211 commented 4 months ago

I tried a full reinstall on a new system (freesurfer 7.4 / conda / mne / enigma_meg + megnet) and am still not able to replicate the error. We did make a change to the code - it was trying ot pull a new version of fsaverage for all subjects even if it was present.

Could you pull the updated code (or just run pip install git+https://github.com/jstout211/enigma_MEG.git to install the new changes).

If you unlink the fsaverage from your BIDS/freesurfer/subjects folder (linked to your primary SUBEJCTS_DIR) and re-run I'm hoping that this may fix the issue after the code updates. If you still have issues we can try to troubleshoot this over video call. Let me know the outcome.

Thanks

Lychfindel commented 4 months ago

Unfortunately, also with the updated code (commit 373c73) I get an error and the fsaverage folder is replaced with a sym link. However, the error is now different:

INFO:0001_ses_20180410_task_MEGLabRestingEO_run_01:do_beamformer :: COMPLETED
INFO:0001_ses_20180410_task_MEGLabRestingEO_run_01:do_make_aparc_sub :: START
ERROR:0001_ses_20180410_task_MEGLabRestingEO_run_01:do_make_aparc_sub :: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label/tmpzxoa3e11' | Pooch could not write to data cache folder '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label'. Will not be able to download data files.
Traceback (most recent call last):
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/pooch/utils.py", line 264, in make_local_storage
    with tempfile.NamedTemporaryFile(dir=path):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/tempfile.py", line 580, in NamedTemporaryFile
    file = _io.open(dir, mode, buffering=buffering,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/tempfile.py", line 577, in opener
    fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/tempfile.py", line 257, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label/tmpzxoa3e11'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 88, in wrapper
    output = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 819, in do_make_aparc_sub
    write_aparc_sub(subjid=f'sub-{self.subject}',
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1316, in write_aparc_sub
    mne.datasets.fetch_aparc_sub_parcellation(subjects_dir=subjects_dir,
  File "<decorator-gen-479>", line 10, in fetch_aparc_sub_parcellation
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/mne/datasets/utils.py", line 411, in fetch_aparc_sub_parcellation
    pooch.retrieve(
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/pooch/core.py", line 227, in retrieve
    make_local_storage(path)
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/pooch/utils.py", line 276, in make_local_storage
    raise PermissionError(" ".join(message)) from error
PermissionError: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label/tmpzxoa3e11' | Pooch could not write to data cache folder '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label'. Will not be able to download data files.
Traceback (most recent call last):
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/pooch/utils.py", line 264, in make_local_storage
    with tempfile.NamedTemporaryFile(dir=path):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/tempfile.py", line 580, in NamedTemporaryFile
    file = _io.open(dir, mode, buffering=buffering,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/tempfile.py", line 577, in opener
    fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/tempfile.py", line 257, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label/tmpzxoa3e11'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/bin/process_meg.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1773, in <module>
    process_subject(args.subject, args)  # process the single specified subject
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1438, in process_subject
    proc.do_proc_allsteps()
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1070, in do_proc_allsteps
    self.do_make_aparc_sub()
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 88, in wrapper
    output = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 819, in do_make_aparc_sub
    write_aparc_sub(subjid=f'sub-{self.subject}', 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alessandro.tonin@rc.fhsc.net/PycharmProjects/Enigma_MEG/enigma_MEG/enigmeg/process_meg.py", line 1316, in write_aparc_sub
    mne.datasets.fetch_aparc_sub_parcellation(subjects_dir=subjects_dir,
  File "<decorator-gen-479>", line 10, in fetch_aparc_sub_parcellation
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/mne/datasets/utils.py", line 411, in fetch_aparc_sub_parcellation
    pooch.retrieve(
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/pooch/core.py", line 227, in retrieve
    make_local_storage(path)
  File "/home/alessandro.tonin@rc.fhsc.net/.conda/envs/enigma_meg/lib/python3.11/site-packages/pooch/utils.py", line 276, in make_local_storage
    raise PermissionError(" ".join(message)) from error
PermissionError: [Errno 13] Permission denied: '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label/tmpzxoa3e11' | Pooch could not write to data cache folder '/mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage/label'. Will not be able to download data files.
jstout211 commented 4 months ago

I think the errors are coming from MNE trying to download (using pooch) the template annotation (aparc-sub) that is not included with Freesurfer. The fsaverage folder will have to be writeable. Can you run the following - its a hack, but should remove this linking issue: 

unlink /mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage
cp -R ${SUBJECTS_DIR}/fsaverage  /mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/
chmod -R 770 /mnt/raid/atonin/bids/Protocollone/derivatives/freesurfer/subjects/fsaverage

And then rerun.

Also - sorry for all the back and forth with this. Appreciate your patience with this issue.