Closed ahmedmoustafa closed 5 years ago
Hi ahmedmoustafa,
It seems that the index has not been successfully loaded. I have tested BitMapperBS by grach38 (Homo_sapiens.GRCh38.dna.primary_assembly.fa). Could you please send me the link that can download your reference genome? I will build index for your genome to find the problem, and solve it as soon as possible. Thank you very much.
In addition, I note that the option "--search" is defined by "/data/reference/hg38". Why "/data/reference/hg38" does not have suffix ".fa" or ".fasta"?
Thank you for the interest in BitMapperBS.
Here is a link the reference genome from NCBI ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.38_GRCh38.p12/GCF_000001405.38_GRCh38.p12_genomic.fna.gz
Thank you so much!
Is the index for Homo_sapiens.GRCh38.dna.primary_assembly.fa
available for download?
Is the index for
Homo_sapiens.GRCh38.dna.primary_assembly.fa
available for download?
Hi ahmedmoustafa,
I have tried BitMapperBS with your genome (GCF_000001405.38_GRCh38.p12_genomic.fna.gz), and did not find any problem. Could you please give me the detail information of your index by the command "ls -al /data/reference/hg38"? In my computer, " ls -al hg38*" reported:
-rw-rw-r-- 1 nhpcc502 nhpcc502 3298102737 Jan 23 01:29 hg38 -rw-rw-r-- 1 nhpcc502 nhpcc502 16709 Jan 23 02:13 hg38.index -rw-rw-r-- 1 nhpcc502 nhpcc502 68 Jan 23 02:13 hg38.index.bs.index -rw-rw-r-- 1 nhpcc502 nhpcc502 2251058346 Jan 23 02:13 hg38.index.bs.index.bwt -rw-rw-r-- 1 nhpcc502 nhpcc502 1590504 Jan 23 02:13 hg38.index.bs.index.occ -rw-rw-r-- 1 nhpcc502 nhpcc502 4275231924 Jan 23 02:13 hg38.index.bs.index.sa -rw-rw-r-- 1 nhpcc502 nhpcc502 814329893 Jan 23 01:34 hg38.index.bs.pac -rw-rw-r-- 1 nhpcc502 nhpcc502 3257336246 Jan 23 01:34 hg38.index.methy
In addition, cloud please give me the following information?
As for indexes, cloud you please give me your email address? I can send the indexes for Homo_sapiens.GRCh38.dna.primary_assembly.fa and GCF_000001405.38_GRCh38.p12_genomic.fna.gz to you by email. Thank you very much.
Here is my index information (the sizes are exactly the same as what you indicated above):
root@genomics:/data/reference# ls -al hg38.bitmapper.index*
-rw-r--r-- 1 root root 16709 Jan 21 19:54 hg38.bitmapper.index
-rw-r--r-- 1 root root 68 Jan 21 19:54 hg38.bitmapper.index.bs.index
-rw-r--r-- 1 root root 2251058346 Jan 21 19:54 hg38.bitmapper.index.bs.index.bwt
-rw-r--r-- 1 root root 1590504 Jan 21 19:54 hg38.bitmapper.index.bs.index.occ
-rw-r--r-- 1 root root 4275231924 Jan 21 19:54 hg38.bitmapper.index.bs.index.sa
-rw-r--r-- 1 root root 814329893 Jan 21 18:44 hg38.bitmapper.index.bs.pac
-rw-r--r-- 1 root root 99247 Jan 21 19:54 hg38.bitmapper.index.log
-rw-r--r-- 1 root root 3257336246 Jan 21 18:43 hg38.bitmapper.index.methy
A link to a sample of the fastq file s1.fq: https://drive.google.com/open?id=12u2SWC3DpDUBviDSdeQQsDUkzda6Mttf
Here is my index information (the sizes are exactly the same as what you indicated above):
root@genomics:/data/reference# ls -al hg38.bitmapper.index* -rw-r--r-- 1 root root 16709 Jan 21 19:54 hg38.bitmapper.index -rw-r--r-- 1 root root 68 Jan 21 19:54 hg38.bitmapper.index.bs.index -rw-r--r-- 1 root root 2251058346 Jan 21 19:54 hg38.bitmapper.index.bs.index.bwt -rw-r--r-- 1 root root 1590504 Jan 21 19:54 hg38.bitmapper.index.bs.index.occ -rw-r--r-- 1 root root 4275231924 Jan 21 19:54 hg38.bitmapper.index.bs.index.sa -rw-r--r-- 1 root root 814329893 Jan 21 18:44 hg38.bitmapper.index.bs.pac -rw-r--r-- 1 root root 99247 Jan 21 19:54 hg38.bitmapper.index.log -rw-r--r-- 1 root root 3257336246 Jan 21 18:43 hg38.bitmapper.index.methy
A link to a sample of the fastq file s1.fq: https://drive.google.com/open?id=12u2SWC3DpDUBviDSdeQQsDUkzda6Mttf
Hi ahmedmoustafa,
Thanks for your file. I have tried BitMapperBS with your fastq file (s1.fq) sucessfully. I found the prefix of your index files is hg38.bitmapper. So you should use "--search /data/reference/hg38.bitmapper" instead of "--search /data/reference/hg38". In addition, it seems that s1.fq is generated by pbat protocol. Thus, you should also use the option "--pbat". As a result, could you please try "./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat"?
It is still failing with Segmentation fault (core dumped)
.
It is still failing with
Segmentation fault (core dumped)
.
Well, cloud you please give me the version of BitMapperBS you used (type: "./bitmapperBS -v"), the memory size of your computer (type: "cat /proc/meminfo | grep MemTotal"), and the CPU (type: "cat /proc/cpuinfo | grep 'model name' |uniq")? In addition, could you please run bitmapperBS with "/usr/bin/time -v" (type: "/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38 --seq ./s1.fq -t 6" or "/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat"), and send me the report? Thank you very much.
Please find below the results:
./bitmapperBS -v
BitMapper 1.0.0.8
cat /proc/meminfo | grep MemTotal
MemTotal: 61837476 kB
cat /proc/cpuinfo | grep 'model name' |uniq
model name : Intel(R) Xeon(R) CPU @ 2.20GHz
/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38 --seq ./s1.fq -t 6" or "/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat
BitMapper 1.0.0.8
Command exited with non-zero status 1
Command being timed: "./bitmapperBS --search /data/reference/hg38 --seq ./s1.fq -t 6 or /usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4200
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 171
Voluntary context switches: 1
Involuntary context switches: 1
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 1
Please find below the results:
./bitmapperBS -v BitMapper 1.0.0.8
cat /proc/meminfo | grep MemTotal MemTotal: 61837476 kB
cat /proc/cpuinfo | grep 'model name' |uniq model name : Intel(R) Xeon(R) CPU @ 2.20GHz
/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38 --seq ./s1.fq -t 6" or "/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat BitMapper 1.0.0.8 Command exited with non-zero status 1 Command being timed: "./bitmapperBS --search /data/reference/hg38 --seq ./s1.fq -t 6 or /usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat" User time (seconds): 0.00 System time (seconds): 0.00 Percent of CPU this job got: 100% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 4200 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 171 Voluntary context switches: 1 Involuntary context switches: 1 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 1
Many thanks for your report. As for the command "/usr/bin/time -v", I mean could you please type :
/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38 --seq ./s1.fq -t 6
or:
/usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat
Sure, here is the output:
bam_output:0
Read files are in FASTQ format...
Open /data/reference/hg38.bitmapper.index sucessfully!
Start load hash table!
10000
refGenLength = 3257319537
Open /data/reference/hg38.bitmapper.index.bs.pac sucessfully!
shapline=1012601865
BWT has been loaded!
hash table has been loaded!
SA_length=2219671779
sparse_suffix_array_length=814329885
SA_flag_iterater=127239046
high_occ_table_length=198812
Command terminated by signal 11
Command being timed: "./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat"
User time (seconds): 0.00
System time (seconds): 6.15
Percent of CPU this job got: 73%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.42
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 7178884
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 3
Minor (reclaiming a frame) page faults: 1793924
Voluntary context switches: 5
Involuntary context switches: 71
Swaps: 0
File system inputs: 360
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Sure, here is the output:
bam_output:0 Read files are in FASTQ format... Open /data/reference/hg38.bitmapper.index sucessfully! Start load hash table! 10000 refGenLength = 3257319537 Open /data/reference/hg38.bitmapper.index.bs.pac sucessfully! shapline=1012601865 BWT has been loaded! hash table has been loaded! SA_length=2219671779 sparse_suffix_array_length=814329885 SA_flag_iterater=127239046 high_occ_table_length=198812 Command terminated by signal 11 Command being timed: "./bitmapperBS --search /data/reference/hg38.bitmapper --seq ./s1.fq -t 6 --pbat" User time (seconds): 0.00 System time (seconds): 6.15 Percent of CPU this job got: 73% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.42 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 7178884 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 3 Minor (reclaiming a frame) page faults: 1793924 Voluntary context switches: 5 Involuntary context switches: 71 Swaps: 0 File system inputs: 360 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
Thank you very much! I want to confirm that, if the "s1.fq" your shared with me, (including 1000 single-end reads), also reported "Segmentation fault (core dumped)"?
Yes, it is the same input file.
Yes, it is the same input file.
I am very confused about this problem... since s1.fq could be aligned sucessfully on my computer. Could you please try to align the following .fastq file by BitMapperBS? It consists 1000 single-end reads from SRR6804872.
https://drive.google.com/file/d/1fAB1U1nye3QorvqU7v2ovc8cLRcx5Yhd/view?usp=sharing
Thanks very much for your kind help!
I tried with SRR6804872_1k.fq
and it gave the same error:
root@genomics:/data/reference# ./bitmapperBS --search /data/reference/hg38.bitmapper --seq SRR6804872_1k.fq -t 6 --pbat
bam_output:0
Read files are in FASTQ format...
Open /data/reference/hg38.bitmapper.index sucessfully!
Start load hash table!
10000
refGenLength = 3257319537
Open /data/reference/hg38.bitmapper.index.bs.pac sucessfully!
shapline=1012601865
BWT has been loaded!
hash table has been loaded!
SA_length=2219671779
sparse_suffix_array_length=814329885
SA_flag_iterater=127239046
high_occ_table_length=198812
Segmentation fault (core dumped)
root@genomics:/data/reference#
also:
root@genomics:/data/reference# /usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq SRR6804872_1k.fq -t 6 --pbat
bam_output:0
Read files are in FASTQ format...
Open /data/reference/hg38.bitmapper.index sucessfully!
Start load hash table!
10000
refGenLength = 3257319537
Open /data/reference/hg38.bitmapper.index.bs.pac sucessfully!
shapline=1012601865
BWT has been loaded!
hash table has been loaded!
SA_length=2219671779
sparse_suffix_array_length=814329885
SA_flag_iterater=127239046
high_occ_table_length=198812
Command terminated by signal 11
Command being timed: "./bitmapperBS --search /data/reference/hg38.bitmapper --seq SRR6804872_1k.fq -t 6 --pbat"
User time (seconds): 0.00
System time (seconds): 6.59
Percent of CPU this job got: 33%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:19.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 7179328
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 1793923
Voluntary context switches: 3756
Involuntary context switches: 45
Swaps: 0
File system inputs: 3629472
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
root@genomics:/data/reference#
I tried with
SRR6804872_1k.fq
and it gave the same error:root@genomics:/data/reference# ./bitmapperBS --search /data/reference/hg38.bitmapper --seq SRR6804872_1k.fq -t 6 --pbat bam_output:0 Read files are in FASTQ format... Open /data/reference/hg38.bitmapper.index sucessfully! Start load hash table! 10000 refGenLength = 3257319537 Open /data/reference/hg38.bitmapper.index.bs.pac sucessfully! shapline=1012601865 BWT has been loaded! hash table has been loaded! SA_length=2219671779 sparse_suffix_array_length=814329885 SA_flag_iterater=127239046 high_occ_table_length=198812 Segmentation fault (core dumped) root@genomics:/data/reference#
also:
root@genomics:/data/reference# /usr/bin/time -v ./bitmapperBS --search /data/reference/hg38.bitmapper --seq SRR6804872_1k.fq -t 6 --pbat bam_output:0 Read files are in FASTQ format... Open /data/reference/hg38.bitmapper.index sucessfully! Start load hash table! 10000 refGenLength = 3257319537 Open /data/reference/hg38.bitmapper.index.bs.pac sucessfully! shapline=1012601865 BWT has been loaded! hash table has been loaded! SA_length=2219671779 sparse_suffix_array_length=814329885 SA_flag_iterater=127239046 high_occ_table_length=198812 Command terminated by signal 11 Command being timed: "./bitmapperBS --search /data/reference/hg38.bitmapper --seq SRR6804872_1k.fq -t 6 --pbat" User time (seconds): 0.00 System time (seconds): 6.59 Percent of CPU this job got: 33% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:19.77 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 7179328 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 4 Minor (reclaiming a frame) page faults: 1793923 Voluntary context switches: 3756 Involuntary context switches: 45 Swaps: 0 File system inputs: 3629472 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 root@genomics:/data/reference#
Hi ahmedmoustafa,
I apologize for the delay - I have a deadline yesterday so I have not noticed your comment.
It seems that there is no problem in your computer. Would you mind me to log to your server remotely? I want to find the problem of BitMapperBS.
Thanks very much for your great help.
encountered the same issue, where index was built successfully, segfault without additional hints. My system Linux, 64G ram, 8cores. simulated_1.fastq.gz was simulated with Sherman (as shown in the bioarchive paper)
./simulator/Sherman \
--length 150 \
--number_of_seqs 1000000 \
--genome_folder human_ref/ \
--paired_end \
--minfrag 70 \
--maxfrag 500 \
--CG_conversion 98.5 \
--CH_conversion 0 \
--error_rate 0.1
running the command
time ./bitmapperBS --search /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa --seq1 /mnt/data/bitmapperbs/data/er10_cr985/simulated_1.fastq.gz --seq2 /mnt/data/bitmapperbs/data/er10_cr985/simulated_2.fastq.gz --sensitive --pe -t 8 --mapstats --bam -o /mnt/data/bitmapperbs/er10_cr985/alignment.bam
fileName[0]: /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa
fileName[1]: /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index
bam_output:1
Read files are in compressed FASTQ format...
Open /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index sucessfully!
Start load hash table!
refGenLength = 3266641347
Open /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index.bs.pac sucessfully!
shapline=2481273140
BWT has been loaded!
hash table has been loaded!
SA_length=2238315399
sparse_suffix_array_length=816660337
SA_flag_iterater=127603180
high_occ_table_length=199380
Start alignment in sensitive mode.
Segmentation fault (core dumped)
real 0m37.611s
user 0m0.020s
sys 0m5.360s
This is BitMapper 1.0.1.3 commit a6b04277ff1d7f5963b8d3409f77448fea1313eb
In the new version of BitMapperBS, the problem reported by ahmedmoustafa seems to have been solved. Could you please share your read file with me via google cloud, or email? I will try to fix it as soon as possible.
encountered the same issue, where index was built successfully, segfault without additional hints. My system Linux, 64G ram, 8cores. simulated_1.fastq.gz was simulated with Sherman (as shown in the bioarchive paper)
./simulator/Sherman \ --length 150 \ --number_of_seqs 1000000 \ --genome_folder human_ref/ \ --paired_end \ --minfrag 70 \ --maxfrag 500 \ --CG_conversion 98.5 \ --CH_conversion 0 \ --error_rate 0.1
running the command
time ./bitmapperBS --search /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa --seq1 /mnt/data/bitmapperbs/data/er10_cr985/simulated_1.fastq.gz --seq2 /mnt/data/bitmapperbs/data/er10_cr985/simulated_2.fastq.gz --sensitive --pe -t 8 --mapstats --bam -o /mnt/data/bitmapperbs/er10_cr985/alignment.bam fileName[0]: /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa fileName[1]: /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index bam_output:1 Read files are in compressed FASTQ format... Open /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index sucessfully! Start load hash table! refGenLength = 3266641347 Open /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index.bs.pac sucessfully! shapline=2481273140 BWT has been loaded! hash table has been loaded! SA_length=2238315399 sparse_suffix_array_length=816660337 SA_flag_iterater=127603180 high_occ_table_length=199380 Start alignment in sensitive mode. Segmentation fault (core dumped) real 0m37.611s user 0m0.020s sys 0m5.360s
This is BitMapper 1.0.1.3 commit a6b0427
The problem has been solved. I will close this issue.
encountered the same issue, where index was built successfully, segfault without additional hints. My system Linux, 64G ram, 8cores. simulated_1.fastq.gz was simulated with Sherman (as shown in the bioarchive paper)
./simulator/Sherman \ --length 150 \ --number_of_seqs 1000000 \ --genome_folder human_ref/ \ --paired_end \ --minfrag 70 \ --maxfrag 500 \ --CG_conversion 98.5 \ --CH_conversion 0 \ --error_rate 0.1
running the command
time ./bitmapperBS --search /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa --seq1 /mnt/data/bitmapperbs/data/er10_cr985/simulated_1.fastq.gz --seq2 /mnt/data/bitmapperbs/data/er10_cr985/simulated_2.fastq.gz --sensitive --pe -t 8 --mapstats --bam -o /mnt/data/bitmapperbs/er10_cr985/alignment.bam fileName[0]: /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa fileName[1]: /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index bam_output:1 Read files are in compressed FASTQ format... Open /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index sucessfully! Start load hash table! refGenLength = 3266641347 Open /mnt/data/bitmapperbs/human_ref/hg19_lambda_pUC19_viral_phix.fa.index.bs.pac sucessfully! shapline=2481273140 BWT has been loaded! hash table has been loaded! SA_length=2238315399 sparse_suffix_array_length=816660337 SA_flag_iterater=127603180 high_occ_table_length=199380 Start alignment in sensitive mode. Segmentation fault (core dumped) real 0m37.611s user 0m0.020s sys 0m5.360s
This is BitMapper 1.0.1.3 commit a6b0427
Still same issue. BitMapper 1.0.2.3 whatever local compiled binary file or bioconda version, the alignment process still output "Segmentation fault (core dumped)". But interestingly, the docker version runs smoothly.
The installation and building the index of hg38 worked just fine. But the mapping step fails with "Segmentation fault (core dumped)":