pachterlab / kb_python

A wrapper for the kallisto | bustools workflow for single-cell RNA-seq pre-processing
https://www.kallistobus.tools/
BSD 2-Clause "Simplified" License
154 stars 23 forks source link

kb count error #110

Closed XYZuo closed 3 years ago

XYZuo commented 3 years ago

Hi, I ran kb count on my data which is 10Xv3.I had a similar error with issue 19(https://github.com/pachterlab/kallistobustools/issues/19) I'm sure my data should be 10XV3.Even if I changed the parameter '-x' to 10XV2 or 10XV1,the error still exists. My code: `kb count -i index.idx -g t2g.txt -x 10xv3 -o P01_SCC \

-c1 cdna_t2c.txt -c2 intron_t2c.txt --lamanno \ /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_1.fq.gz \ /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_2.fq.gz [2021-03-06 17:01:42,001] WARNING The --lamanno and --nucleus flags are deprecated. These options will be removed in a future release. Please use --workflow lamanno or --workflow nucleus instead. [2021-03-06 17:01:42,002] INFO Using index index.idx to generate BUS file to P01_SCC from [2021-03-06 17:01:42,002] INFO /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_1.fq.gz [2021-03-06 17:01:42,002] INFO /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_2.fq.gz [2021-03-06 17:05:46,266] ERROR [index] k-mer length: 31 [index] number of targets: 1,277,033 [index] number of k-mers: 1,560,197,793 [index] number of equivalence classes: 4 [2021-03-06 17:05:46,266] ERROR An exception occurred Traceback (most recent call last): File "/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/main.py", line 837, in main COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir) File "/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/main.py", line 197, in parse_count count_velocity( File "/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/count.py", line 1509, in count_velocity bus_result = kallisto_bus( File "/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/validate.py", line 112, in inner results = func(*args, *kwargs) File "/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/count.py", line 149, in kallisto_bus run_executable(command) File "/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/dry/init.py", line 24, in inner return func(args, **kwargs) File "/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/utils.py", line 233, in run_executable raise sp.CalledProcessError(p.returncode, ' '.join(command)) subprocess.CalledProcessError: Command '/home/zuoxinyi/miniconda3/envs/zxy-kb/lib/python3.8/site-packages/kb_python/bins/linux/kallisto/kallisto bus -i index.idx -o P01_SCC -x 10xv3 -t 8 /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_1.fq.gz /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_2.fq.gz' died with <Signals.SIGSEGV: 11>. `

My example file:

`(zxy-kb) [zuoxinyi@es_node1 kb]$ zcat /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_1.fq.gz | head @A00199:369:HGHWVDSXX:4:1101:1154:1016 1:N:0:ACTTCATA+TCTTTCCC NGGATAATCAACGTGTCCACGCGTGTACTAATCTATGATCTCACAGAGAGACAGTTTTGTTATAACTCTAAATTCTTGATGTTACACTATAACTCTACAGCCCGGGAACAGTGGCTCATGCCTGTAAACCCAGCAGTTTTGTGAGGCCAT +

FFFFFFFFFFFFFFFFFFFFFFFFFFFF::,F:F:,F,:::,,,,,,,,,,,,F:,:::F,,,,,:,:,,FF,FF:FF:FFF,:::,,,FFF,FF:FFFF:,:FF::FFF:::F::,FFF,FFFFF,FFF::FFF:FFF,F:,,:FFF,

@A00199:369:HGHWVDSXX:4:1101:1913:1016 1:N:0:ACTTCATA+TCTTTCCC NACAGGTTCGGAGTAGAGATGCTTTTCGTAAATATTGAGATCAGACTCTCTCTCTTGAAAAAAAAAAATTTTATTAAAATCAGTAGATTAGTCTCAGGAAGAACGTACAAATAATTTACAAAATTTAAACAACACAACAATCAGCTCACC +

FFFFFFFFFFFFFFFFFFFFFFFFFFF,F,,:FF,::,,:,,,,,F,:,:,:,FF,F,FF:,FFF:,F,:,::FF:FF,:F,F,:,,,,:,,,F:FF,,:,F::,,FF,F,FF,,:,,:,FF,,,FF:FF,,,:F::FF:FF,FF,:,F

@A00199:369:HGHWVDSXX:4:1101:2474:1016 1:N:0:ACTTCATA+TCTTTCCC NATGCTAGTAATCAAGTATTCAGAAGGCGGCGCCGGCACTTAAAACACACACATTTTTTTCACAGGGGCTAGGCGGAGGTGATTTGGGGCACACCCCGAGGAGGGCCCCGCGCCAAGAATACGGTGTTGGGGGGCGTGGTGGTGGAGCGT (zxy-kb) [zuoxinyi@es_node1 kb]$ zcat /lab_file_oridata/ITP/P01_SCC/Rawdata/P01_SCC_2.fq.gz | head @A00199:369:HGHWVDSXX:4:1101:1154:1016 2:N:0:ACTTCATA+TCTTTCCC CGAAATAAAGTGTGGAGGCCAGGCGTGGTGGCTCACGGTTGTGATCCCAGGTAGCTGGGATTACAGGCGTGCACCACTACCACCCGGCTAATTTTTGTATTTTTAGTAGAGACGGGGTTTCACCATGTTGGCCAGGCTGGTCTCGAACTC + FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF @A00199:369:HGHWVDSXX:4:1101:1913:1016 2:N:0:ACTTCATA+TCTTTCCC TATAGCTGTTTGTGATGATGCCAGTATTTGGCGCTGGGATCGACTTCGATAAAATACTTTTGCCTAATCAAAATTAGAGTGTGTTTGTTGTCTGTGTAAAATAGAATTAATGTATCTTGCTAGTAAGGGCACGTAGAGCATTTAGAGTTG + FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFF,:FFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF @A00199:369:HGHWVDSXX:4:1101:2474:1016 2:N:0:ACTTCATA+TCTTTCCC TGGGTGCCCGGCACCGCGCCCGAGCCCACTCCATTCAGATCATGAAGGTGGAGGAGATCGCGGCCAGCAAGTGCCGCCGGCCGGCTGTCAAGCAGTTCCACGACTCCAAGATCAAGTTCCCGCTGCCCCACCGGGTCCTGCGCCGTCAGC `

Could someone please help with this? Thanks!

Lioscro commented 3 years ago

That error usually occurs when the process is out of usable memory. How much memory do you have on your machine?

rameshbalan commented 3 years ago

I get the same error too! I tried with the default, 64G and 120G memory.

Command:

kb count -i index.idx -g t2g.txt -x 10xv3 --workflow lamanno --overwrite --loom -c1 cdna_t2c.txt -c2 intron_t2c.txt reads/sample_S1_L001_R1_001.fastq.gz reads/sample_S1_L001_R2_001.fastq.gz -t 24 -m 64G

[2021-03-13 20:56:06,381]    INFO Using index index.idx to generate BUS file to . from
[2021-03-13 20:56:06,381]    INFO         reads/sample_S1_L001_R1_001.fastq.gz
[2021-03-13 20:56:06,381]    INFO         reads/sample_S1_L001_R2_001.fastq.gz
[2021-03-13 21:16:10,398]   ERROR
[index] k-mer length: 31
[index] number of targets: 0
[index] number of k-mers: 0
[index] number of equivalence classes: 0
[quant] will process sample 1: reads/sample_S1_L001_R1_001.fastq.gz
reads/sample_S1_L001_R2_001.fastq.gz
[quant] finding pseudoalignments for the reads ... done
[quant] processed 419,679,501 reads, 0 reads pseudoaligned[~warn] no reads pseudoaligned.

[2021-03-13 21:16:10,399]   ERROR An exception occurred
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/main.py", line 837, in main
    COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)
  File "/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/main.py", line 218, in parse_count
    temp_dir=temp_dir
  File "/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/count.py", line 1510, in count_velocity
    fastqs, index_paths[0], technology, out_dir, threads=threads
  File "/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/validate.py", line 112, in inner
    results = func(*args, **kwargs)
  File "/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/count.py", line 149, in kallisto_bus
    run_executable(command)
  File "/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/dry/__init__.py", line 24, in inner
    return func(*args, **kwargs)
  File "/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/utils.py", line 233, in run_executable
    raise sp.CalledProcessError(p.returncode, ' '.join(command))
subprocess.CalledProcessError: Command '/home/user/miniconda3/envs/scRNA/lib/python3.6/site-packages/kb_python/bins/linux/kallisto/kallisto bus -i index.idx -o . -x 10xv3 -t 24 reads/sample_S1_L001_R1_001.fastq.gz reads/sample_S1_L001_R2_001.fastq.gz' returned non-zero exit status 1.

Solutions Tried (Unsuccessfully):

I think it maybe an issue with bustools and not the way the wrappers invoke the program

Lioscro commented 3 years ago

Hi, @rameshbalan, This is a different error than the one posted by @ZxyChopcat. Your error is due to kallisto failing to pseudoalign any reads. I notice that there are zero targets, kmers, and equivalence classes. This is most likely due to your index file. Please open a new issue if you are still having this problem. We can help you out there.

XYZuo commented 3 years ago

Thanks a lot for your reply! I solved this issue by building the index again. kb ref -i index.idx -g t2g.txt -f1 cdna.fa -f2 intron.fa -c1 cdna_t2c.txt -c2 intron_t2c.txt --workflow lamanno \ Homo_sapiens.GRCh38.dna.toplevel.fa.gz \ Homo_sapiens.GRCh38.97.gtf.gz Maybe there were some problem when I build the index.