`too many open files` in clustering

palfalvi commented 6 years ago

Hi,

Thank you for the softwares and the pipeline, I am really keen to try them.

I tried pychopper and started pinfish pipeline, when I run into the following error, several times:

Error in rule polish_clusters:
    jobid: 9
    output: results/polished_transcripts.fas
    conda-env: /home/peegee/pipeline/.snakemake/conda/900fb89f

RuleException:
CalledProcessError in line 99 of /home/peegee/bin/pipeline-pinfish-analysis/Snakefile:
Command 'source activate  /home/peegee/pipeline/.snakemake/conda/900fb89f; set -euo pipefail;  
    / /home/peegee/pipeline/polish_clusters/polish_clusters -t 10 -a results/cluster_memberships.tsv -c 2 -o results/polished_transcripts.fas alignments/reads_aln_sorted.bam ' returned non-zero exit status 1.
  File "/home/peegee/bin/pipeline-pinfish-analysis/Snakefile", line 99, in __rule_polish_clusters
  File "/home/peegee/anaconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run

The end of the pipeline, I found this error:

Failed running command: minimap2 -ax map-ont -t 10 -k14  /tmp/pinfish_4aa784ff-73cd-4a97-bfec-7932725895c1_054730664/reference.fq /tmp/pinfish_4aa784ff-73cd-4a97-bfec-7932725895c1_054730664/reads.fq | samtools view -h -F 2304 > /tmp/pinfish_4aa784ff-73cd-4a97-bfec-7932725895c1_054730664/alignments.sam - fork/exec /usr/bin/bash: too many open files

I tried to increase open file limit, but got the same again. Is there any idea why I get this and how I can resolve? Thank you for your answer in advance!

bsipos commented 6 years ago

Hi,

Could you give me a bit more details about your analysis:

Which OS are you using?
Are you using by any chance a shared machine?
Is there any chance that you are running out of space in /tmp?
Could you please paste in the parameters from the config file.
How many mRNA clusters do you have in the clustered_transcripts.gff?

Regards, Botond

UPDATE: I believe I found the file descriptor leak and fixed it. Please try again after updating pinfish.

palfalvi commented 6 years ago

Hi,

Thank you for your quick reply and for the update.

I updated pinfish and looks like it passed the point it was killed before. Clustering is running right now, I will update if it could finish or not.

If it is still relevant:

I use my local mini-server with centos7.
Space should be not an issue (I checked storage, /tmp and memory).
I had 1.047.455 lines in 72.555 clusters in cluster_memberships.tsv.
Input was a ~7.5 Gb fastq as an output of pychopper.

palfalvi commented 6 years ago

It finished without any trouble. Thank you very much for your help!

nanoporetech / pipeline-pinfish-analysis

`too many open files` in clustering #1