search

maxplanck-ie / snakepipes

Customizable workflows based on snakemake and python for the analysis of NGS data
http://snakepipes.readthedocs.io
381 stars 85 forks source link

snakePipes 2.5.4 PermissionError: [Errno 13] Permission denied: '/package' #835

Closed hero-outman closed 2 years ago

hero-outman commented 2 years ago

Dear fellows: I update snakePipes2.5.3 to snakePipes2.5.4 by command mamba update -n snakePipes.2.5.3 -c mpi-ie -c conda-forge -c bioconda --prune snakePipes, and checked by snakePipes version--> version 2.5.4. Then trying to run the commonly used workflow like mRNA-seq or ATAC-seq, the workflow will fail and throw errors(didn't meet this error in version 2.5.3):

Traceback (most recent call last):
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakemake/__init__.py", line 730, in snakemake
    success = workflow.execute(
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakemake/workflow.py", line 683, in execute
    self.persistence = Persistence(
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakemake/persistence.py", line 98, in __init__
    os.makedirs(d, exist_ok=True)
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/package'

Accroding to the error info, seems I have to chmod or chown something: where is the /package located? May I have some hints to solve this issue?

Here is part of the log file:

/myhome/.conda/envs/snakePipes.2.5.3/bin/ATAC-seq -d /my_project -v -j 20 --local --DAG --peakCaller MACS2 --maxFragmentSize 2000 --minFragmentSize 20 --qval 0.001 --sampleSheet ATACsamplesheet_ncDvsshD.tsv --FDR 0.05 --LFC 1 hg38

TMPDIR=/data/extended/ PYTHONNOUSERSITE=True /myhome/.conda/envs/snakePipes.2.5.3/bin/snakemake --use-conda --conda-prefix /package/anaconda3/envs/ --latency-wait 300 --snakefile /myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/workflows/ATAC-seq/Snakefile --jobs 20 --directory /my_project --configfile /my_project/ATAC-seq.config.yaml --keep-going --printshellcmds

---- This analysis has been done using snakePipes version 2.5.4 ----
Sample sheet found and header is ok!
Traceback (most recent call last):
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakemake/__init__.py", line 730, in snakemake
    success = workflow.execute(
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakemake/workflow.py", line 683, in execute
    self.persistence = Persistence(
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakemake/persistence.py", line 98, in __init__
    os.makedirs(d, exist_ok=True)
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/myhome/.conda/envs/snakePipes.2.5.3/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/package'

--- Config ---------------------------------------------------------------------
config file: /my_project/ATAC-seq.config.yaml

... ommit ...

Thanks in advance

katsikora commented 2 years ago

Dear Hero-outman,

after updating snakePipes, you'd need to run snakePipes config again, and provide the path to your scratch location with --tempDir as well as any other setup arguments relevant to your working environment. You can also recycle your config file from the previous installation for this means.

Hope this helps,

Best,

Katarzyna

hero-outman commented 2 years ago

Dear Katarzyna, Many thanks for the reply. I'm not sure about:

your scratch location

you mentioned, I guess it's something like conda-prefix, I re-edit default.yaml and organisms.yaml again and run with command snakePipes config. For default.yaml: 

snakemakeOptions: ' --use-conda --conda-prefix /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/ '
tempDir: ./temp/

Then, I run with ATAC work-flow, and still got errors like:

/myHome/.conda/envs/snakePipes.2.5.3/bin/ATAC-seq -d /Data/Projects/my_project -v -j 20 --local --DAG --peakCaller MACS2 --maxFragmentSize 2000 --minFragmentSize 20 --qval 0.001 --sampleSheet ATACsamplesheet_ncDvsshD.tsv --FDR 0.05 --LFC 1 hg38

TMPDIR=./temp/ PYTHONNOUSERSITE=True /myHome/.conda/envs/snakePipes.2.5.3/bin/snakemake --use-conda --conda-prefix /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/ --latency-wait 300 --snakefile /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/workflows/ATAC-seq/Snakefile --jobs 20 --directory /Data/Projects/my_project --configfile /Data/Projects/my_project/ATAC-seq.config.yaml --keep-going --printshellcmds

---- This analysis has been done using snakePipes version 2.5.4 ----
Sample sheet found and header is ok!
Building DAG of jobs...
Creating conda environment /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/shared.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/shared.yaml:
Command:
mamba env create --quiet --file "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/ce9aeb901059dc9a362a93dd587e56f9.yaml" --prefix "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/ce9aeb901059dc9a362a93dd587e56f9"
Output:

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/conda/exceptions.py", line 1080, in __call__
        return func(*args, **kwargs)
      File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/conda_env/cli/main_create.py", line 141, in execute
        result[installer_type] = installer.install(prefix, pkg_specs, args, env)
      File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/mamba/mamba_env.py", line 45, in mamba_install
        index = load_channels(pool, channel_urls, repos, prepend=False)
      File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/mamba/utils.py", line 122, in load_channels
        index = get_index(
      File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/mamba/utils.py", line 103, in get_index
        is_downloaded = dlist.download(True)
    RuntimeError: Download error (28) Timeout was reached [https://conda.anaconda.org/conda-forge/noarch/repodata.json]
    Failed to connect to conda.anaconda.org port 443 after 7016 ms: Connection timed out

`$ /myHome/.conda/envs/snakePipes.2.5.3/bin/mamba create --quiet --file /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/ce9aeb901059dc9a362a93dd587e56f9.yaml --prefix /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/ce9aeb901059dc9a362a93dd587e56f9`

...ommit...

An unexpected error has occurred. Conda has prepared the above report.

--- Config ---------------------------------------------------------------------
config file: /Data/Projects/my_project/ATAC-seq.config.yaml

...ommit...

snakemakeOptions:  --use-conda --conda-prefix /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/ 
tempDir: ./temp/

-------------------------------------------------------------------------------- 

...ommit...

According to the error log, it's a connection error, but I think I need to create conda environments. So I run the command snakePipes createEnvs --keepCondaDir. Running this command takes quite a long time for me, about 12hrs(?), and I also got en error:


...ommit...

Creating environment from /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/sc_rna_seq.yaml in /myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/
conda-forge/linux-64     Using cache
conda-forge/noarch       Using cache
bioconda/linux-64        Using cache
bioconda/noarch          Using cache
r/linux-64               Using cache
r/noarch                 Using cache
pkgs/r/noarch            Using cache
python: /home/conda/feedstock_root/build_artifacts/libsolv_1625855492383/work/src/rules.c:261: solver_addrule: Assertion `!p2 && d > 0' failed.
anaconda/cloud/msys2/lin  
anaconda/cloud/bioconda/  
anaconda/cloud/bioconda/  
pkgs/r/linux-64           
anaconda/pkgs/free/noarc  
anaconda/cloud/msys2/noa  
anaconda/pkgs/free/linux  
anaconda/cloud/conda-for  
pkgs/main/noarch          
pkgs/main/linux-64        
anaconda/cloud/conda-for  
Traceback (most recent call last):
  File "/myHome/.conda/envs/snakePipes.2.5.3/bin/snakePipes", line 377, in createCondaEnvs
    subprocess.check_call(cmd)
  File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mamba', 'env', 'create', '--force', '--file', '/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/sc_rna_seq.yaml', '--prefix', '/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/f7aeb679151810ac255d61ea4ac452ff']' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myHome/.conda/envs/snakePipes.2.5.3/bin/snakePipes", line 460, in <module>
    main(sys.argv[1:])
  File "/myHome/.conda/envs/snakePipes.2.5.3/bin/snakePipes", line 454, in main
    createCondaEnvs(args)
  File "/myHome/.conda/envs/snakePipes.2.5.3/bin/snakePipes", line 380, in createCondaEnvs
    shutil.rmtree(os.path.join(condaDirUse, h), ignore_errors=False)
  File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/shutil.py", line 708, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/shutil.py", line 706, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/myHome/.conda/envs/snakePipes.2.5.3/lib/python3.10/site-packages/snakePipes/shared/rules/envs/f7aeb679151810ac255d61ea4ac452ff'

I think connection issues give the create_eviroments err, so I had a 2nd try of snakePipes createEnvs. when I get the result I will paste it here.

Best

Chu

adRn-s commented 2 years ago

Possible duplicate #349

katsikora commented 2 years ago

Hi Hero-outman,

great, so you have now successfully configured your condaDir and started downloading packages necessary for creating workflow-specific envs.

Do you have mamba installed in your base environment ? From your commands above, it looks like you have installed mamba under the snakePipes_2.5.3 environment.

With scratch I meant a tempDir, in case you wanted to configure it to something else than /tmp.

HTH,

Katarzyna

hero-outman commented 2 years ago

Hi Katarzyna,

I update snakePipes to 2.5.4 under the 2.5.3 environment and checked by command snakePipes version. After finishing running create env(failed many times on package downloading by curl, e.g.: GenomeInfoDbData_1.2.7), snakepipes workflows could be run successfully, and no more issues occurred. So I think the reason is I didn't realize after updating to the new version, create env command needed to be run again.

Many thanks for your patience and detailed reply.

Best

chu

hero-outman commented 2 years ago

Hi adRn-s,

You are right, I didn't create environments after updating to 2.5.4. After createEnvs the workflow goes fine. Thank you so much

Best chu