Index out-of-bounds error when parsing format 5.3/7.3

LafeWessel commented 1 year ago

Simple Description on the Bug

When parsing GRIB data in the 5.3/7.3 format, sometimes there is an out-of-bounds error when calling .dispatch(). This may be related to #30, but I am unsure.

Steps to Reproduce

Using the same data as #29 (NOAA GFS data), I iterated through the submessages and attempted to parse and print a few values of each, along with the definition information about the messages.

// read grib file
let fname = "./test_files/test_grib_file";
let f = std::fs::File::open(fname).unwrap();
let rd = std::io::BufReader::new(f);
let gr = grib::from_reader(rd).unwrap();

// set up decoder
for lyr in gr.iter() {
    let (_, fm) = lyr;
    println!(
        "{}\n{:?}\n{:?}\n{:?}\n{:?}",
        fm.describe(),
        fm.indicator(),
        fm.grid_def(),
        fm.prod_def(),
        fm.grid_def()
    );
    let dc = Grib2SubmessageDecoder::from(fm).unwrap();
    match dc.dispatch() { // WILL PANIC ON SOME MESSAGES
        Ok(dv) => {
            println!("size hint: {:?}", dv.size_hint());

            dv.take(10).for_each(|v| {
                println!("{:?}", v);
            });
        }
        Err(e) => {
            eprintln!("\n\n{:?}\n\n", e);
        }
    };
}

Expected Behavior

Parse GRIB file normally.

Actual Behavior

Here is the backtrace along with the console output about the GRIB layer definitions. The source of the error is at src/decoders/complex.rs:36:14.

Grid:                                   Latitude/longitude
  Number of points:                     1038240
Product:                                Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
  Parameter Category:                   Moisture
  Parameter:                            Relative humidity
  Generating Proceess:                  Forecast
  Forecast Time:                        12
  Forecast Time Unit:                   Hour
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     10
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     0
  2nd Scaled Value:                     0
Data Representation:                    Grid point data - complex packing and spatial differencing
  Number of represented values:         1038240

Indicator { discipline: 0, total_length: 983303 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 1, 1, 2, 0, 96, 0, 0, 0, 1, 0, 0, 0, 12, 100, 0, 0, 0, 0, 10, 255, 0, 0, 0, 0, 0] }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
thread 'read_grib::read_grib_tests::test_read_grib_file' panicked at 'range end index 4 out of range for slice of length 3', /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/complex.rs:36:14
stack backtrace:
   0: rust_begin_unwind
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
   2: core::slice::index::slice_end_index_len_fail_rt
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:76:5
   3: core::slice::index::slice_end_index_len_fail
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:69:9
   4: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:408:13
   5: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:18:9
   6: grib::decoders::complex::decode
             at /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/complex.rs:36:14
   7: grib::decoders::common::Grib2SubmessageDecoder::dispatch
             at /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/common.rs:121:67
   8: crate::read_grib::read_grib_tests::test_read_grib_file::test_impl
             at ./src/read_grib.rs:300:19
   9: crate::read_grib::read_grib_tests::test_read_grib_file
             at ./src/read_grib.rs:272:5
  10: crate::read_grib::read_grib_tests::test_read_grib_file::{{closure}}
             at ./src/read_grib.rs:272:5
  11: core::ops::function::FnOnce::call_once
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
  12: core::ops::function::FnOnce::call_once
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5

Additional Context

Some of the other layers were unable to be parsed and returned a DecodeError(ComplexPackingDecodeError(NotSupported)), but I'm guessing that those errors are related to #30.

Here is a link to my Google Drive folder containing the GRIB file that I am currently testing with. link

noritada commented 1 year ago

@LafeWessel Thank you very much for reporting!

I have reproduced the issue using your code and understand its cause. This is due to the same cause as https://github.com/noritada/grib-rs/issues/30. However, since checking the number of bytes and returning the https://github.com/noritada/grib-rs/issues/30 error is done a little later than accessing the bytes, we are getting a different error in the case where the access fails as in this case.

The current code only supports 2 bytes as a number of extra parameters in Section 7. https://github.com/noritada/grib-rs/issues/30 should occur when it is greater than 2 bytes, and as far as I can tell from checking the data, it occurs when the submessage contains 3-bytes parameters. This https://github.com/noritada/grib-rs/pull/32 occurs when it is less than 2 bytes (the data which I reproduced the issue with contains 1-byte parameters).

Fixing https://github.com/noritada/grib-rs/issues/30 should resolve this issue together, so please be patient for a little longer.

LafeWessel commented 1 year ago

No worries, thanks for all the help!

noritada commented 1 year ago

After merging #33, the program in the report will not crash. However, the number of decoded values of 46th submessage (indexed from zero) will be 2, which is much smaller than its size hint:

size hint: (1038240, Some(1038240))
0.0
0.0

So, this issue still remains. In this submessage, 0 is set as the value of some parameter indicating the number of bits, and I am wondering if some special processing is needed in that case.

noritada / grib-rs