Panic on D3400 decode

[gennyble]

Hello! I tried the sample usage on in the readme with this%20(3:2).NEF) file from raw.pixls.us and got a panic on decode. Here's the backtrace from rawloader which looks suspiciously like an underflow:

thread 'main' panicked at 'index out of bounds: the len is 4096 but the index is 65535', /home/genbyte/src/rawloader/src/decoders/basics.rs:127:30
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:486
  11: rust_begin_unwind
             at src/libstd/panicking.rs:388
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:101
  13: core::panicking::panic_bounds_check
             at src/libcore/panicking.rs:73
  14: <usize as core::slice::SliceIndex<[T]>>::index
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/slice/mod.rs:2872
  15: core::slice::<impl core::ops::index::Index<I> for [T]>::index
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/slice/mod.rs:2732
  16: <alloc::vec::Vec<T> as core::ops::index::Index<I>>::index
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/vec.rs:1942
  17: rawloader::decoders::basics::LookupTable::dither
             at /home/genbyte/src/rawloader/src/decoders/basics.rs:127
  18: rawloader::decoders::nef::NefDecoder::do_decode
             at /home/genbyte/src/rawloader/src/decoders/nef.rs:311
  19: rawloader::decoders::nef::NefDecoder::decode_compressed
             at /home/genbyte/src/rawloader/src/decoders/nef.rs:237
  20: <rawloader::decoders::nef::NefDecoder as rawloader::decoders::Decoder>::image
             at /home/genbyte/src/rawloader/src/decoders/nef.rs:140
  21: rawloader::decoders::RawLoader::decode_unsafe
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:522
  22: rawloader::decoders::RawLoader::decode::{{closure}}
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:530
  23: std::panicking::try::do_call
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297
  24: __rust_try
  25: std::panicking::try
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:274
  26: std::panic::catch_unwind
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394
  27: rawloader::decoders::RawLoader::decode
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:529
  28: rawloader::decoders::RawLoader::decode_file
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:544
  29: rawloader::decode_file
             at /home/genbyte/src/rawloader/src/lib.rs:104
  30: easyraw::main
             at src/main.rs:14
  31: std::rt::lang_start::{{closure}}
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  32: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  33: std::panicking::try::do_call
             at src/libstd/panicking.rs:297
  34: std::panicking::try
             at src/libstd/panicking.rs:274
  35: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  36: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  37: std::rt::lang_start
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  38: main
  39: __libc_start_main
  40: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[gennyble]

Ah, well, it seems that the other D3400%20(3:2).NEF) image works with rawlaoder. I now see that https://raw.pixls.us/#repo with the search query being "d3400" has both a split and non-split example. While I don't exactly know what that means, it seems that the split file make rawloader angry.

[pedrocr]

How are you compiling rawloader? Panics are caught and turned into errors. If you are doing panic=abort that's not a supported configuration. The file should probably be made to work though, unless there's something actually broken with it.

[gennyble]

I did cargo run and passed in the file name. The driving program I used was just the Usage example from the README.md. When I use the library for real (great library by the way! Thank you.), I'll better handle the decoding errors. I just thought I'd report the decode fail in case it was something you were trying to support.

I did some initial investigation yesterday and the error that produced the trace above is because clampbits (might've been called clamp_bits, I don't know) in decode/basics.rs was returning -1, but then gets cast to a u16. When I "fix" that- by making it return 0 if negative- more errors pop-up further into the decode. Don't know if this information is helpful, but I hope it's semi-useful at least.

[pedrocr]

If there are split and non-split files it's probably a different decode method that needs to be detected. I'll have a look.

If you're doing raw decoding you may also be interested in imagepipe and chimper for a raw processing pipeline and a GUI for it:

https://github.com/pedrocr/imagepipe https://github.com/pedrocr/chimper

[pedrocr]

Master should now work for you hopefully. I still need to do some regression testing to make sure I didn't break anything and then some performance work to get back what was almost certainly lost with this.