Open jalalsiddiqui opened 4 years ago
shalomhillelroth
commented
4 years ago Hi, Sorry for the belayed answer. The main problem seems to be the lack of memory needed for the java based indexer.
I'm trying to work on a solution for that - could you kindly let me know what are the system configurations you use (especially RAM and java version)?
Best, Shalom Hillel Roth
jalalsiddiqui
commented
4 years ago No problem!
Thanks for all your help.
The system we use is the Owens cluster:
https://www.osc.edu/resources/technical_support/supercomputers/owens/technical_specifications
I called 2 cores but can increase the memory if need be.
The java version we use if as follows:
-bash-4.2$ java -version openjdk version "1.8.0_222" OpenJDK Runtime Environment (build 1.8.0_222-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
Hope this helps,
Jalal
jalalsiddiqui
commented
4 years ago Hope all is well,
I am wondering if there is a solution to get this program running. I am thinking of trying Docker but Ohio Supercomputing Center has support for Singularity. I am wondering if there is a way around this.
shalomhillelroth
commented
4 years ago Hi,
I apologise for the belayed answer, I have limited computer access with the whole COVID19 event.
There seems to be a simple way to covert or import from Docker into singularity (I saw several post for this issue specifically) however I'm not familiar at all with the Singularity platform so I'm afraid I can't right now recommend the best solution for this.
Did increasing the memory demand help?
Best,
Shalom Hillel Roth
jalalsiddiqui
commented
4 years ago It didn't seem to help. I discuss that in the ValueError issue
This is the commands I entered:
RNAEditingIndex \ -d /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/BAMs \ -f _with_0.1.Aligned.sortedByCoord.out.bam.AluChr1Only.bam \ -l /fs/project/PAS1348/RNAEditingIndex_test/logs_directory \ -o /fs/project/PAS1348/RNAEditingIndex_test/output_directory \ -os /fs/project/PAS1348/RNAEditingIndex_test/summary_directory \ --genome hg38 \ -rb /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/AnnotationAndRegions/ucscHg38RefSeqCurated.OnlyChr1.bed.gz \ --refseq /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/AnnotationAndRegions/ucscHg38RefSeqCurated.OnlyChr1.bed.gz \ --snps /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/AnnotationAndRegions/ucscHg38CommonGenomicSNPs150.OnlyChr1.bed.gz \ --genes_expression ucscHg38GTExGeneExpression.OnlyChr1.bed.gz \ --verbose --stranded --paired
This is the error results
-bash-4.2$ cat test_rei.sh.e9540545 [bam_sort_core] merging from 0 files and 10 in-memory blocks... [bam_sort_core] merging from 0 files and 10 in-memory blocks... [bam_sort_core] merging from 0 files and 10 in-memory blocks... [bam_sort_core] merging from 0 files and 10 in-memory blocks... Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962217/SRR5962217_sampled/SRR5962217_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962217/SRR5962217_sampled/SRR5962217_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam.trimmed_5.bam
Bases to trim from each side : 5
Number of records read = 76230 Number of records written = 76230 Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962209/SRR5962209_sampled/SRR5962209_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962209/SRR5962209_sampled/SRR5962209_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam.trimmed_5.bam
Bases to trim from each side : 5
Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962217/SRR5962217_sampled/SRR5962217_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962217/SRR5962217_sampled/SRR5962217_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam.trimmed_5.bam
Bases to trim from each side : 5
Number of records read = 62293 Number of records written = 62293 Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962209/SRR5962209_sampled/SRR5962209_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962209/SRR5962209_sampled/SRR5962209_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam.trimmed_5.bam
Bases to trim from each side : 5
Number of records read = 76338 Number of records written = 76338
Number of records read = 63750 Number of records written = 63750 [mpileup] 1 samples in 1 input files Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962219/SRR5962219_sampled/SRR5962219_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962219/SRR5962219_sampled/SRR5962219_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam.trimmed_5.bam
Bases to trim from each side : 5
[mpileup] 1 samples in 1 input files Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962201/SRR5962201_sampled/SRR5962201_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962201/SRR5962201_sampled/SRR5962201_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_1.bam.trimmed_5.bam
Bases to trim from each side : 5
Number of records read = 106797 Number of records written = 106797 Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962219/SRR5962219_sampled/SRR5962219_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962219/SRR5962219_sampled/SRR5962219_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam.trimmed_5.bam
Bases to trim from each side : 5
Number of records read = 123921 Number of records written = 123921 Arguments in effect: Input file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962201/SRR5962201_sampled/SRR5962201_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam Output file : /fs/project/PAS1348/RNAEditingIndex_test/output_directory/BAMs/SRR5962201/SRR5962201_sampled/SRR5962201_sampled_region_ucscHg38RefSeqCurated.OnlyChr1.bed.gz_alignments_strand_2.bam.trimmed_5.bam
Bases to trim from each side : 5
Number of records read = 103573 Number of records written = 103573 [mpileup] 1 samples in 1 input files
Number of records read = 130933 Number of records written = 130933 [mpileup] 1 samples in 1 input files [mpileup] 1 samples in 1 input files [mpileup] 1 samples in 1 input files [mpileup] 1 samples in 1 input files [mpileup] 1 samples in 1 input files Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuffer.append(StringBuffer.java:367) at java.io.BufferedReader.readLine(BufferedReader.java:370) at java.io.BufferedReader.readLine(BufferedReader.java:389) at EditingIndexJavaUtils.BEDGenomeIndexer.GenerateIndex(BEDGenomeIndexer.java:41) at EditingIndexJavaUtils.BEDGenomeIndexer.main(BEDGenomeIndexer.java:117) at EditingIndexJavaUtils.EditingIndexBEDUtils.main(EditingIndexBEDUtils.java:30) [2020-03-04 15:45:18,212] EIPipelineManger ERROR Process: Editing_Index_Pipline_SRR5962209sampled; Going To Error Step: Step-1 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuffer.append(StringBuffer.java:367) at java.io.BufferedReader.readLine(BufferedReader.java:370) at java.io.BufferedReader.readLine(BufferedReader.java:389) at EditingIndexJavaUtils.BEDGenomeIndexer.GenerateIndex(BEDGenomeIndexer.java:41) at EditingIndexJavaUtils.BEDGenomeIndexer.main(BEDGenomeIndexer.java:117) at EditingIndexJavaUtils.EditingIndexBEDUtils.main(EditingIndexBEDUtils.java:30) [2020-03-04 15:46:49,691] EIPipelineManger ERROR Process: Editing_Index_Pipline_SRR5962217sampled; Going To Error Step: Step-1 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuffer.append(StringBuffer.java:367) at java.io.BufferedReader.readLine(BufferedReader.java:370) at java.io.BufferedReader.readLine(BufferedReader.java:389) at EditingIndexJavaUtils.BEDGenomeIndexer.GenerateIndex(BEDGenomeIndexer.java:41) at EditingIndexJavaUtils.BEDGenomeIndexer.main(BEDGenomeIndexer.java:117) at EditingIndexJavaUtils.EditingIndexBEDUtils.main(EditingIndexBEDUtils.java:30) [2020-03-04 15:48:39,587] EIPipelineManger ERROR Process: Editing_Index_Pipline_SRR5962219sampled; Going To Error Step: Step-1 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuffer.append(StringBuffer.java:367) at java.io.BufferedReader.readLine(BufferedReader.java:370) at java.io.BufferedReader.readLine(BufferedReader.java:389) at EditingIndexJavaUtils.BEDGenomeIndexer.GenerateIndex(BEDGenomeIndexer.java:41) at EditingIndexJavaUtils.BEDGenomeIndexer.main(BEDGenomeIndexer.java:117) at EditingIndexJavaUtils.EditingIndexBEDUtils.main(EditingIndexBEDUtils.java:30) [2020-03-04 15:49:20,818] EIPipelineManger ERROR Process: Editing_Index_Pipline_SRR5962201sampled; Going To Error Step: Step-1 [2020-03-04 15:49:20,846] general_functions WARNING GGPSResources.general_functions.remove_files Failed To Remove 04-03-2020-20.cnf [2020-03-04 15:49:20,847] general_functions WARNING GGPSResources.general_functions.remove_files Failed To Remove 04-03-2020-20.cnf Traceback (most recent call last): File "Tools/EditingIndex/A2IEditingIndex.py", line 1341, in
File "Tools/EditingIndex/A2IEditingIndex.py", line 1166, in main
File "Tools/EditingIndex/DataConverters/RefSeqGenesCoverter.py", line 101, in load_refseq_bed
ValueError: too many values to unpack
[55967] Failed to execute script A2IEditingIndex
This is the screen output
-bash-4.2$ cat test_rei.sh.o9540545 GenerateIndex - Starting! Running: bedtools getfasta -name -bedOut -fi '/fs/project/PAS1348/Software/RNAEditingIndexer/Resources/Genomes/HomoSapiens/ucscHg38Genome.fa' -bed '/fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/AnnotationAndRegions/ucscHg38RefSeqCurated.OnlyChr1.bed.gz' GenerateIndex - Indexing FASTA Records! GenerateIndex - encountered unexpected line from bedtools, skipping line. Line:index file /fs/project/PAS1348/Software/RNAEditingIndexer/Resources/Genomes/HomoSapiens/ucscHg38Genome.fa.fai not found, generating... Failed To Run On /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/BAMs/BAMs/SRR5962209/SRR5962209_sampled_with_0.1.Aligned.sortedByCoord.out.bam.AluChr1Only.bam! GenerateIndex - Starting! Running: bedtools getfasta -name -bedOut -fi '/fs/project/PAS1348/Software/RNAEditingIndexer/Resources/Genomes/HomoSapiens/ucscHg38Genome.fa' -bed '/fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/AnnotationAndRegions/ucscHg38RefSeqCurated.OnlyChr1.bed.gz' GenerateIndex - Indexing FASTA Records! Failed To Run On /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/BAMs/BAMs/SRR5962217/SRR5962217_sampled_with_0.1.Aligned.sortedByCoord.out.bam.AluChr1Only.bam! GenerateIndex - Starting! Running: bedtools getfasta -name -bedOut -fi '/fs/project/PAS1348/Software/RNAEditingIndexer/Resources/Genomes/HomoSapiens/ucscHg38Genome.fa' -bed '/fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/AnnotationAndRegions/ucscHg38RefSeqCurated.OnlyChr1.bed.gz' GenerateIndex - Indexing FASTA Records! Failed To Run On /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/BAMs/BAMs/SRR5962219/SRR5962219_sampled_with_0.1.Aligned.sortedByCoord.out.bam.AluChr1Only.bam! GenerateIndex - Starting! Running: bedtools getfasta -name -bedOut -fi '/fs/project/PAS1348/Software/RNAEditingIndexer/Resources/Genomes/HomoSapiens/ucscHg38Genome.fa' -bed '/fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/AnnotationAndRegions/ucscHg38RefSeqCurated.OnlyChr1.bed.gz' GenerateIndex - Indexing FASTA Records! Failed To Run On /fs/project/PAS1348/Software/RNAEditingIndexer/TestResources/BAMs/BAMs/SRR5962201/SRR5962201_sampled_with_0.1.Aligned.sortedByCoord.out.bam.AluChr1Only.bam!
I will appreciate any help that I can get