search

kaitai-io / kaitai_struct_javascript_runtime

Kaitai Struct: runtime for JavaScript
Apache License 2.0
37 stars 22 forks source link

Open large binary file in kaitai struct for javascript? #8

Open Meigyoku-Thmn opened 6 years ago

Meigyoku-Thmn commented 6 years ago

The current javascript implementation requires reading a whole file into memory (ArrayBuffer). This is not good if I want to read a large file, for example a very big zip file. Can you add an option to use file descriptor (returned by fs.openSync) instead of ArrayBuffer?

GreyCat commented 6 years ago

That's actually a viable idea, which would hopefully require only adjustments to runtime. Any JavaScript volunteers?

cherue commented 4 years ago

This is even possible in browsers using <input type="file">:

const file = input.files[0]
const slice = file.slice(offset, offset + size)

slice.arrayBuffer().then(
    arrayBuffer => {
        const data = new Uint8Array(arrayBuffer)
    }
)
GreyCat commented 4 years ago

Wow, that's even better :)

I wonder what the overhead of using slice() + arrayBuffer() + new Uint8Array for every single read will be significant vs one ArrayBuffer + Uint8Array for whole file?

cherue commented 4 years ago

Yeah that'll probably be an issue, especially for small files.

Maybe something like Python's from_bytes and from_file could work, if it all goes through the KaitaiStream constructor it wouldn't even need compiler changes.