search

maxplanck-ie / snakepipes

Customizable workflows based on snakemake and python for the analysis of NGS data
http://snakepipes.readthedocs.io
389 stars 88 forks source link

error while running the DNA-mapping #1045

Open sunta3iouxos opened 2 months ago

sunta3iouxos commented 2 months ago

Hallo again, This something new:

DNA-mapping -i /mnt/Z/Shared\ folder/Shared\ data/DNA/NGS_AP06_apouikli_A006200409/ -o /mnt/Z/Shared\ folder/Shared\ data/DNA/NGS_AP06_apouikli_A006200409/bam/ -c /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/DNA-mapping/defaults.yaml --local -j 8 --fastqc --trim --trimmer fastp --trimmerOptions "--trim_poly_g --trim_poly_x -Q -L --correction" --dedup --plotFormat "pdf" --mapq 3 --cutntag --verbose --insertSizeMax 5000     /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/mm10.yaml
/home/tgeorgom/mambaforge/envs/snakePipes/bin/DNA-mapping:61: SyntaxWarning: invalid escape sequence '\-'
  "the first - (e.g., '\--very-fast'). Default: '%(default)s'.",

TMPDIR=/home/tgeorgom/temp/ PYTHONNOUSERSITE=True /home/tgeorgom/mambaforge/envs/snakePipes/bin/snakemake --use-conda --conda-prefix /home/tgeorgom/mambaforge/envs --latency-wait 300 --snakefile /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/DNA-mapping/Snakefile --jobs 8 --directory /mnt/Z/Shared folder/Shared data/DNA/NGS_AP06_apouikli_A006200409/bam --configfile /mnt/Z/Shared folder/Shared data/DNA/NGS_AP06_apouikli_A006200409/bam/DNA-mapping.config.yaml --keep-going --printshellcmds

PID: 364423

WorkflowError:
Config file must be given as JSON or YAML with keys at top level.
Error: snakemake returned an error code of 1, so processing is incomplete!

my snakePipes version

(snakePipes) tgeorgom@CECAD:~$ snakePipes version
version 2.5.3

freshly created environment

$ snakePipes  config

--- Final Updated Config ---------------------------------------------------------------------
config file: /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/defaults.yaml
clusterConfig: shared/cluster.yaml
configMode: manual
emailSender: None
max_thread: 8
oldConfig: None
onlySSL: False
organismsDir: shared/organisms
smtpPassword: None
smtpPort: 0
smtpServer: None
smtpUsername: None
snakemakeOptions:  --use-conda --conda-prefix /home/tgeorgom/mambaforge/envs
tempDir: /home/tgeorgom/temp/
toolsVersion: True
--------------------------------------------------------------------------------

and the info:

 snakePipes info
Organism YAML files:
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/SchizoSPombe_ASM294v2.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/GRCz10.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/hg38.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/dm3.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/mm10.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/hs37d5.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/dm6.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/mm9.yaml
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/mm10_gencodeM13.yaml

The workflow-generic defaults.yaml file is:
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/defaults.yaml

The default cluster.yaml file. Its defaults are overridden by the per-workflow cluster.yaml files:
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/cluster.yaml

Workflow-specific cluster.yaml and defaults.yaml files are in:
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/preprocessing
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/HiC
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/noncoding-RNA-seq
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/mRNA-seq
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/ATAC-seq
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/WGBS
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/createIndices
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/scRNAseq
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/ChIP-seq
    /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/DNA-mapping

Temp dir under /home/tgeorgom/temp/ exists and will be used.

I am using the /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/shared/organisms/mm10.yaml

and I am calling the pipeline with:

DNA-mapping -i /mnt/Z/Shared\ folder/Shared\ data/DNA/NGS_AP06_apouikli_A006200409/ -o /mnt/Z/Shared\ folder/Shared\ data/DNA/NGS_AP06_apouikli_A006200409/bam/ -c /home/tgeorgom/mambaforge/envs/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/DNA-mapping/defaults.yaml --local -j 8 --fastqc --trim --trimmer fastp --trimmerOptions "--trim_poly_g --trim_poly_x -Q -L --correction" --dedup --plotFormat "pdf" --mapq 3 --cutntag --verbose --insertSizeMax 5000 mm10
sunta3iouxos commented 2 months ago

I think I solved the first one: the pipeline does not like special characters, like spaces, even if it is under quotes. maybe this can be addressed? The second are the values for the bowtie2 this part: SyntaxWarning: invalid escape sequence '-' "the first - (e.g., '--very-fast'). Default: '%(default)s'.", so something is wrong in the code there

katsikora commented 2 months ago

Hi,

It's good practice to remove spaces from paths, as they lead to all kind of issues in shell, such as those you have encountered. I don't think I'd see it on our side :shrug:

Could you be more explicit about the second error? Did you pass any --alignerOpts to DNA-mapping?

Also, why are you explicitly passing the defaults.yaml config from some (other?) snakePipes installation to DNA-mapping?

Best wishes,

Katarzyna

sunta3iouxos commented 2 months ago

I passed defaults.yaml to just trying to figure out where the error comes from. I then by checking a couple of other things found that the error is due to the name. I symling the fastq.gz to another folder and things are working. I know about the spaces, but this is not of my doing 😢

now regarding that part:

/scratch/Theo/envs/snakePipes/bin/DNA-mapping:61: SyntaxWarning: invalid escape sequence '\-'
  "the first - (e.g., '\--very-fast'). Default: '%(default)s'.",

although the pipeline works very well (till now), I am always getting this error. I thought that is comes from bowtie, since I faintly remember the --very-fast string, but I can not tell for sure