A Blocks header size cannot be larger than 127 bytes so for each Block a header (Array) get initialised of size 127.
val header = Slice.of[Byte](Byte.MaxValue)
Issue
Even though this byte size (127 bytes) in terms of max file-size (2.GB) is small the in-memory cost of creating Array[Byte] of size 127 is still expensive.
Solutions
Find the actual maximum header size required by randomly running tests with various configurations. My guess is that maximum wouldn't be any larger than 60 bytes with minimum being 2 bytes.
Or Each data-block should provide it's expected header size so header array sizes are exact.
Overview
A
Blocks header size cannot be larger than 127 bytes so for eachBlocka header (Array) get initialised of size 127.Issue
Even though this byte size (127 bytes) in terms of max file-size (2.GB) is small the in-memory cost of creating
Array[Byte]of size 127 is still expensive.Solutions