sharksforarms / deku

Declarative binary reading and writing: bit-level, symmetric, serialization/deserialization
Apache License 2.0
1.14k stars 55 forks source link

With no rest and input anymore: how to peek into the Reader? Or get total of whole input? #460

Closed peterdk closed 1 month ago

peterdk commented 4 months ago

So I was a bit surprised by the sudden removal of deku:rest and deku:input.

Some usecases I can switch over to the Reader part, and that I like. But... for some cases I need to know the full length of the input file, be able to peek into it, and generally read the input more then I return in with rest.

For example, I use Deku for reading heavily obfuscated/protected data structures. This means I have to deal with for example a

struct{
count: u8,
#[deku(reader = "Foo::read(count, deku::input_bits, deku::rest)")]
data:Vec<u8>
}

That is not a problem normally, but some data I get is deliberately working around the old original C reader implementation loop holes, and I need to handle those.

For example normally I would get as input:

[4,1,1,1,1,1]
resulting in 
count: 4, data: [1,1,1,1] and leftover 1

But the obfuscated one is:

[0,1,1,1,1,0]
resulting in
count: 0, data:[], leftover [1,1,1,1,0]

Since it's taking advantage of that the original C implementation does not check count field, but (simplified) skips the count field and looks for 0 as terminator, I need to parse it as:

count:0, data[1,1,1,1], leftover: [0]

For that I had implemented

fn read<'a>(
        count: u8,
        mut input_bits: &'a BitSlice<u8, Msb0>,
        mut rest: &'a DekuRest,
    ) -> Result<(&'a DekuRest, Foo), DekuError>
{

\\\Read stuff and peek into input_bits, and then when I am sure how much I want to read, 
do a rest.read() with that count, so the other objects that come after this Foo struct, 
will have proper data position.
}

I am not sure how I could work around this with the new streaming setup. I would be helped if I at least could peek() into the stream without changing it's position, but can read the data.

Any ideas?

wcampbell0x2a commented 4 months ago

I don't have time to give a full answer/test ideas. I however have a branch that adds Seek support, which should give you the ability to "peek" around while reading.

https://github.com/sharksforarms/deku/pull/360

wcampbell0x2a commented 1 month ago

@peterdk Seek has been added on master, lmk if you have more questions or use cases in which we didn't resolve