search

natir / yacrd

Yet Another Chimeric Read Detector
MIT License
72 stars 8 forks source link

thread 'main' panicked at 'slice index starts at 8092 but ends at 7507' #37

Closed dominik-handler closed 4 years ago

dominik-handler commented 4 years ago

Hi,

I was running yacrd with the following commands:

${SINGULARITYdir}minimap2.simg minimap2 -x ava-ont -t $SLURM_CPUS_PER_TASK -g 500 ${TMPdir}filtered_reads.fq.gz ${TMPdir}filtered_reads.fq.gz >${TMPdir}overlap.paf

yacrd -i ${TMPdir}overlap.paf -o ${TMPdir}report.yacrd -c 4 -n 0.4 scrubb -i ${TMPdir}filtered_reads.fq -o ${TMPdir}reads.scrubb.fasta

and got the following error:

\thread 'main' panicked at 'slice index starts at 8092 but ends at 7507', src/libcore/slice/mod.rs:2670:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
SIGABRT: abort
PC=0x47cdab m=0 sigcode=0

goroutine 1 [running, locked to thread]:
syscall.RawSyscall(0x3e, 0x10421, 0x6, 0x0, 0x0, 0xc000110180, 0xc000110180)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:78 +0x2b fp=0xc00023be70 sp=0xc00023be68 pc=0x47cdab
syscall.Kill(0x10421, 0x6, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:597 +0x4b fp=0xc00023beb8 sp=0xc00023be70 pc=0x479bcb
github.com/sylabs/singularity/internal/app/starter.Master.func2()
    internal/app/starter/master_linux.go:152 +0x61 fp=0xc00023bf00 sp=0xc00023beb8 pc=0x7928f1
github.com/sylabs/singularity/internal/pkg/util/mainthread.Execute.func1()
    internal/pkg/util/mainthread/mainthread.go:21 +0x2f fp=0xc00023bf28 sp=0xc00023bf00 pc=0x790f4f
main.main()
    cmd/starter/main_linux.go:102 +0x5f fp=0xc00023bf60 sp=0xc00023bf28 pc=0x972bbf
runtime.main()
    /usr/lib/golang/src/runtime/proc.go:203 +0x21e fp=0xc00023bfe0 sp=0xc00023bf60 pc=0x433b4e
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00023bfe8 sp=0xc00023bfe0 pc=0x45f7c1

goroutine 19 [syscall]:
os/signal.signal_recv(0xb9da80)
    /usr/lib/golang/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
    /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /usr/lib/golang/src/os/signal/signal_unix.go:29 +0x41

goroutine 5 [chan receive]:
github.com/sylabs/singularity/internal/pkg/util/mainthread.Execute(0xc0003cc400)
    internal/pkg/util/mainthread/mainthread.go:24 +0xb4
github.com/sylabs/singularity/internal/app/starter.Master(0x7, 0x4, 0x10436, 0xc00000e140)
    internal/app/starter/master_linux.go:151 +0x44c
main.startup()
    cmd/starter/main_linux.go:75 +0x53e
created by main.main
    cmd/starter/main_linux.go:98 +0x35

rax    0x0
rbx    0x0
rcx    0xffffffffffffffff
rdx    0x0
rdi    0x10421
rsi    0x6
rbp    0xc00023bea8
rsp    0xc00023be68
r8     0x0
r9     0x0
r10    0x0
r11    0x202
r12    0xff
r13    0x0
r14    0xb83b64
r15    0x0
rip    0x47cdab
rflags 0x202
cs     0x33
fs     0x0
gs     0x0

Do you have any idea what could have triggered this error? I am rerunning it now with backtrace enabled.

All the best and thank you, Dominik

dominik-handler commented 4 years ago

Aja, I use the head of the GIT repository.

dominik-handler commented 4 years ago

now with backtrace enabled: 

thread 'main' panicked at 'slice index starts at 8092 but ends at 7507', src/libcore/slice/mod.rs:2670:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1028
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:188
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:205
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:464
  11: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:373
  12: rust_begin_unwind
             at src/libstd/panicking.rs:302
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:139
  14: core::slice::slice_index_order_fail
             at src/libcore/slice/mod.rs:2670
  15: <core::ops::range::Range<usize> as core::slice::SliceIndex<[T]>>::index
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/slice/mod.rs:2845
  16: core::slice::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/slice/mod.rs:2647
  17: yacrd::editor::scrubbing::fastq
             at src/editor/scrubbing.rs:188
  18: yacrd::editor::scrubbing::scrubbing
             at src/editor/scrubbing.rs:45
  19: yacrd::main
             at src/main.rs:93
  20: std::rt::lang_start::{{closure}}
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/rt.rs:61
  21: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:48
  22: std::panicking::try::do_call
             at src/libstd/panicking.rs:287
  23: __rust_maybe_catch_panic
             at src/libpanic_abort/lib.rs:28
  24: std::panicking::try
             at src/libstd/panicking.rs:265
  25: std::panic::catch_unwind
             at src/libstd/panic.rs:396
  26: std::rt::lang_start_internal
             at src/libstd/rt.rs:47
  27: main
  28: __libc_start_main
  29: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
SIGABRT: abort
PC=0x47cdab m=0 sigcode=0

goroutine 1 [running, locked to thread]:
syscall.RawSyscall(0x3e, 0x1112a, 0x6, 0x0, 0x0, 0xc000062000, 0xc000062000)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:78 +0x2b fp=0xc000171e70 sp=0xc000171e68 pc=0x47cdab
syscall.Kill(0x1112a, 0x6, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:597 +0x4b fp=0xc000171eb8 sp=0xc000171e70 pc=0x479bcb
github.com/sylabs/singularity/internal/app/starter.Master.func2()
    internal/app/starter/master_linux.go:152 +0x61 fp=0xc000171f00 sp=0xc000171eb8 pc=0x7928f1
github.com/sylabs/singularity/internal/pkg/util/mainthread.Execute.func1()
    internal/pkg/util/mainthread/mainthread.go:21 +0x2f fp=0xc000171f28 sp=0xc000171f00 pc=0x790f4f
main.main()
    cmd/starter/main_linux.go:102 +0x5f fp=0xc000171f60 sp=0xc000171f28 pc=0x972bbf
runtime.main()
    /usr/lib/golang/src/runtime/proc.go:203 +0x21e fp=0xc000171fe0 sp=0xc000171f60 pc=0x433b4e
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000171fe8 sp=0xc000171fe0 pc=0x45f7c1

goroutine 6 [syscall]:
os/signal.signal_recv(0xb9da80)
    /usr/lib/golang/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
    /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /usr/lib/golang/src/os/signal/signal_unix.go:29 +0x41

goroutine 8 [chan receive]:
github.com/sylabs/singularity/internal/pkg/util/mainthread.Execute(0xc00031e3b0)
    internal/pkg/util/mainthread/mainthread.go:24 +0xb4
github.com/sylabs/singularity/internal/app/starter.Master(0x7, 0x4, 0x11139, 0xc00000f240)
    internal/app/starter/master_linux.go:151 +0x44c
main.startup()
    cmd/starter/main_linux.go:75 +0x53e
created by main.main
    cmd/starter/main_linux.go:98 +0x35

rax    0x0
rbx    0x0
rcx    0xffffffffffffffff
rdx    0x0
rdi    0x1112a
rsi    0x6
rbp    0xc000171ea8
rsp    0xc000171e68
r8     0x0
r9     0x0
r10    0x0
r11    0x202
r12    0xf3
r13    0x0
r14    0xb83e88
r15    0x0
rip    0x47cdab
rflags 0x202
cs     0x33
fs     0x0
gs     0x0
natir commented 4 years ago

Hi @dominik-handler,

Thanks for finding this bug. Last master commit (3a87920) "solve" this bug, if yacrd try to scrubb read shorter than scrubb position yacrd now generate a warning and skip next scrubbing position for this read.

I see you didn't use the same file between overlap search and scrubbing (filtered_reads.fq.gz and filtered_reads.fq), I think this change generate this bug, maybe a read with same id have different length between this two file, or two reads have the same id.

yacrd accept compressed file as input .

dominik-handler commented 4 years ago

Hi,

Thank you for the quick fix. These 2 files are completely the same. Once compressed and once not. I did not try to use the compressed version with yacrd. Good to know that this also works.

One additional question, what is the exact difference between split and scrub? I guess split splits the read into 2 or more pieces if there is a not-covered region in the read. Is scrub doing the same and in addition remove uncovered ends?

Also, is it normal that my reads have multiple regions that overlap or do I read it wrong: Chimeric b582ec56-ba2a-47bd-b170-e1591b3aebf5::sampleid=unknown::read=31::length=44705::QUAL=11.9715::ch=297::runid=cb3f3806869ec0ebc650f8bfb7858f0247e5f647_2-44688 44687 119,0,119;978,40911,41889;1156,40911,42067;1156,40911,42067;1156,40911,42067;1156,40911,42067;1,44686,44687

Dominik

dominik-handler commented 4 years ago

image

Also there is a bug in the documentation. Once the middle number indicates the start and once the first number. Both would make sense in this example so it would be nice to know the correct definition. Which one is it? Start,Extent,Stop Extent,Start,Stop

Thank you, Dominik

dominik-handler commented 4 years ago

While trying to build yacrd I got the following error:

17:59:02     Compiling yacrd v0.6.0 (/yacrd)
17:59:02  error: unknown character escape: `
17:59:02    --> src/cli.rs:83:203
17:59:02     |
17:59:02  83 |         help = "if it set yacrd create tempory file, with value of this parameter as prefix, to reduce memory usage but increase the runtime, warning if prefix contain path separator (`/` for unix or `\` for windows) directory is delete"
17:59:02     |                                                                                                                                                                                                           ^ unknown character escape
17:59:02  
17:59:02  error: unknown character escape: `
17:59:02   --> src/main.rs:5:47
17:59:02    |
17:59:02  5 | of this software and associated documentation files (the "Software"), to deal
17:59:02    |                                               ^ unknown character escape
17:59:02  
17:59:02  error: proc-macro derive panicked
17:59:02    --> src/cli.rs:23:10
17:59:02     |
17:59:02  23 | #[derive(StructOpt, Debug)]
17:59:02     |          ^^^^^^^^^
17:59:02     |
17:59:02     = help: message: unexpected byte 96 after \ character in byte literal
17:59:02  
17:59:02  error: aborting due to 3 previous errors
17:59:02  
17:59:02  error: could not compile `yacrd`.
17:59:02
natir commented 4 years ago

I probably fix this bug to quickly, last commit solve this compilation error sorry.

Difference between scrubb and split task:

It's not an important technical difference, but it's more usage difference, split is design to remove chimeric position, scrubb is design to remove positions all that can't be trusted (even if it's not chimeric position).

Your result for read b582ec56-ba2a-47bd-b170-e1591b3aebf5 is very strange can you send to me all overlaps in which this read contributes.

I fix this error in readme sorry, it's: Extent,Start,Stop

natir commented 4 years ago

I found how to reproduce bug with read b582ec56-ba2a-47bd-b170-e1591b3aebf5, you didn't need to send to my is overlap.

dominik-handler commented 4 years ago

I am currently building the new version. I will test it later.

I also looked for more weird reads and I have plenty of them. I used minimap2 v2.17-r954-dirty to align the reads. The commands are in my first post. Below you find the yacrd report for another 2 reads and the overlaps for all three reads. overlaps-selected.txt

NotCovered 3b9c9197-5909-437d-8c76-667357bb20a0::sampleid=unknown::read=2084::length=67867::QUAL=18.3666::ch=2::runid=8d8451d3ceb17d2cf1ed956ce674b9c163fc1ff0 67867 2944,0,2944;1468,3182,4650;1468,3182,4650;1769,3182,4951;1769,3182,4951;4378,3182,7560;6512,3182,9694;7244,3182,10426;7864,3182,11046;7864,3182,11046;8354,3182,11536;10691,3182,13873;10691,3182,13873;10693,3182,13875;10842,3182,14024;13744,3182,16926;13891,3182,17073;14031,3182,17213;14050,3182,17232;14050,3182,17232;3351,18375,21726;3789,18375,22164;3789,18375,22164;5189,18375,23564;5290,18375,23665;5290,18375,23665;5290,18375,23665;5290,18375,23665;3157,25257,28414;3462,25257,28719;3495,25257,28752;3557,25257,28814;3621,25257,28878;1753,30906,32659;1753,30906,32659;1753,30906,32659;1753,30906,32659;1989,30906,32895;523,34562,35085;700,34562,35262;777,34562,35339;777,34562,35339;1367,34562,35929;1804,36685,38489;1804,36685,38489;2022,36685,38707;2426,36685,39111;2426,36685,39111;1225,40324,41549;1965,40324,42289;2088,40324,42412;2088,40324,42412;2088,40324,42412;219,43688,43907;239,43688,43927;98,44044,44142;1567,45420,46987;1590,45420,47010;1785,45420,47205;1846,45420,47266;1846,45420,47266;392,47976,48368;392,47976,48368;392,47976,48368;30,48387,48417;2234,52239,54473;2234,52239,54473;2234,52239,54473;2259,52239,54498;2352,52239,54591;7646,56450,64096;7807,56450,64257;7917,56450,64367;7917,56450,64367;7917,56450,64367;1,67866,67867

Chimeric 1aa0e6eb-62a0-49a0-947d-32d57fc81025::sampleid=R94_LSK108_wtOSC_CelegansIsolation_BP9-15h::read=6867::length=46609::QUAL=11.7753::ch=425::runid=89d2a090238a2b2e41650c0c1dff2615f3b613cb_10-46578 46569 28,0,28;438,17417,17855;17,46552,46569

Dominik

natir commented 4 years ago

Thanks for all this stuff.

The commit 08271fd8c5e07 solve my test, but I can't repoduce your result with overlap you give to me. Can you say if result look better.

Thanks again for your help.

natir commented 4 years ago

I take a look on some of my real datasets I observe this trouble and the last commit seems to solve it.

If you can confirm please close this issue.

dominik-handler commented 4 years ago

It seems the problem is indeed solved, thank you.