Deserialization fails when using `bits = X` and `read_all` alongside `endian = "little"`

[bhagen55]

Hello! I ran into a strange issue when trying to use read_all. When the struct also contains a specifically sized field using ex #[deku(bits = 40)], deserialization always fails with Incomplete(NeedSize { bits: 8 }), but only when the endian-ness of the struct is explicitly specified. It seems to fail if set to big or little.

use deku::prelude::*;

#[derive(Clone, Debug, DekuWrite, DekuRead)]
#[deku(endian = "little")]
pub struct Foo {
    #[deku(bits = 40)]
    pub weird_sized_thing: u64,
    pub normal_sized_thing: u8,
    pub other_thing: u16,
    #[deku(read_all)]
    pub bulk_things: Vec<u8>,
}

fn main() {
    #[rustfmt::skip]
    const INPUT_DATA_MESSAGE: [u8; 11] = [
        0x01, 0x01, 0x01, 0x01, 0x01,
        0x0,
        0x02, 0x02,
        0x00, 0x00, 0x00,
    ];

    Foo::try_from(INPUT_DATA_MESSAGE.as_ref()).unwrap();
}

If I remove the endian specification, this program runs fine.

#[derive(Clone, Debug, DekuWrite, DekuRead)]
pub struct Foo {
    #[deku(bits = 40)]
    pub weird_sized_thing: u64,
    pub normal_sized_thing: u8,
    pub other_thing: u16,
    #[deku(read_all)]
    pub bulk_things: Vec<u8>,
}

It also runs fine if I specify endian-ness but remove the weird_sized_thing struct member.

#[derive(Clone, Debug, DekuWrite, DekuRead)]
#[deku(endian = "little")]
pub struct Foo {
    pub normal_sized_thing: u8,
    pub other_thing: u16,
    #[deku(read_all)]
    pub bulk_things: Vec<u8>,
}

I am using Deku 0.18.1 with Rust 1.72.0.

This issue does not appear to be present on Deku 0.17.

[wcampbell0x2a]

Good news, this seems to be fixed in https://github.com/sharksforarms/deku/pull/483.

examples/485.rs:

use deku::prelude::*;

#[derive(Clone, Debug, DekuWrite, DekuRead)]
pub struct Foo {
    #[deku(bits = 40)]
    pub weird_sized_thing: u64,
    pub normal_sized_thing: u8,
    pub other_thing: u16,
    #[deku(read_all)]
    pub bulk_things: Vec<u8>,
}

fn main() {
    env_logger::init();
    #[rustfmt::skip]
    const INPUT_DATA_MESSAGE: [u8; 11] = [
        0x01, 0x01, 0x01, 0x01, 0x01,
        0x0,
        0x02, 0x02,
        0x00, 0x00, 0x00,
    ];

    let foo = Foo::try_from(INPUT_DATA_MESSAGE.as_ref()).unwrap();
    println!("{:02x?}", foo);
}

deku> RUST_LOG=trace cargo run --example 485 --features logging
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.13s
     Running `target/debug/examples/485`
[2024-10-01T03:08:12Z TRACE 485] Reading: Foo.weird_sized_thing
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes: requesting 5 bytes
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes: returning [01, 01, 01, 01, 01]
[2024-10-01T03:08:12Z TRACE 485] Reading: Foo.normal_sized_thing
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const: requesting 1 bytes
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const: returning [00]
[2024-10-01T03:08:12Z TRACE 485] Reading: Foo.other_thing
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const: requesting 2 bytes
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const: returning [02, 02]
[2024-10-01T03:08:12Z TRACE 485] Reading: Foo.bulk_things
[2024-10-01T03:08:12Z TRACE deku::reader] not end: read [00]
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const: requesting 1 bytes
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const_leftover: using previous read 00
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const_leftover: returning [00]
[2024-10-01T03:08:12Z TRACE deku::reader] not end: read [00]
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const: requesting 1 bytes
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const_leftover: using previous read 00
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const_leftover: returning [00]
[2024-10-01T03:08:12Z TRACE deku::reader] not end: read [00]
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const: requesting 1 bytes
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const_leftover: using previous read 00
[2024-10-01T03:08:12Z TRACE deku::reader] read_bytes_const_leftover: returning [00]
[2024-10-01T03:08:12Z TRACE deku::reader] end
Foo { weird_sized_thing: 101010101, normal_sized_thing: 00, other_thing: 202, bulk_things: [00, 00, 00] }

I'll make sure this gets a unit test so it doesn't break again :)