search

cta-observatory / eventio_kaitai

Prototyping a Katai struct implementation of eventio data structures
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Add extended length support #6

Open maxnoe opened 10 months ago

maxnoe commented 10 months ago

Add support for extended length field.

Unfortunately, it's kind of hard to test as it is only applicable for payloads that are larger than 1 GiB.

That's also why for a long time there was a bug in the c reference writer implementation preventing these objects (fixed somewhen in 2018 I think).

I have one file that exceeds this and is using the extended bit, but it's 11 GiB large.

maxnoe commented 10 months ago

Tested this locally with the large file and it works:

type = 1200          version = 0, extended=True, length =          1096
type = 1212          version = 0, extended=True, length =           696
type = 1201          version = 0, extended=True, length =            20
type = 1202          version = 0, extended=True, length =          1096
type = 1203          version = 0, extended=True, length =            16
type = 1205          version = 0, extended=True, length =         34156
type = 1204          version = 0, extended=True, length =   12576458588
  type = 1205        version = 0, extended=True, length =   12576458572
type = 1209          version = 0, extended=True, length =          1096
type = 1210          version = 0, extended=True, length =            16
zonca commented 10 months ago

what about artificially creating one of those files as a test fixture? if it is time consuming, we could cache it: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows, we have 10 GB storage for the cache.

otherwise we could host one of those files publicly, download it as a test fixture, and then cache it.