Too much memory used in simple read-write scenarii

[jglrxavpok]

See https://github.com/Minestom/Minestom/issues/422

Minestom uses ChunkColumn only for reading and writing to storage. Following my discussion with @TheMode on Discord, an API to stream chunk contents instead of caching them could be introduced to reduce memory allocations.

Below is a memory flame graph following chunk loading with the AnvilLoader of Minestom

[jglrxavpok]

List of elements to implement:

Both Minestom and the internals of Hephaistos should use the streaming API. (Will be updated when new things are found)

Streaming needs to provide both a way to read and a way to write (Minestom will need it). Which features you want to use need to be explicit (to avoid reading stuff we don't care about, and simplify code). Depending on the implementation, this can be an abstract class with empty default implementations, or a bit mask.

Read

Per-chunk data

Features used by Minestom:

• [x] Generation status
• [x] Chunk coordinates
• [x] Min Y
• [x] Max Y

Features represented with the current Hephaistos ChunkColumn/ChunkSection API

• [x] Data version
• [x] Last Update
• [x] InhabitedTime
• [x] Heightmaps
• [x] Entity NBT data (for old chunks)
• [x] Tile ticks
  • [x] "To be ticked" list (< 1.18)
• [x] Liquid ticks
  • [x] Liquids to be ticked (< 1.18)
• [x] Structure data
• [x] Lights
• [x] Post processing data
• [x] Air carving mask
• [x] Liquid carving mask
• [x] LightOn
• [x] Tile entities
• [x] Old biomes (< 1.18)

New features:

• [ ] entities/ folder format

Per-section data

Features used by Minestom:

• [x] Biomes
• [x] Blocks
  • [x] Palette
  • [x] Block indices inside Palette
• [x] Sky light
• [x] Block light

Write

Per-chunk data

Features used by Minestom:

• [ ] Generation status
• [ ] Chunk coordinates
• [ ] Min Y
• [ ] Max Y

Features represented with the current Hephaistos ChunkColumn/ChunkSection API

• [ ] Data version
• [ ] Last Update
• [ ] InhabitedTime
• [ ] Heightmaps
• [ ] Entity NBT data (for old chunks)
• [ ] Tile ticks
  • [ ] "To be ticked" list (< 1.18)
• [ ] Liquid ticks
  • [ ] Liquids to be ticked (< 1.18)
• [ ] Structure data
• [ ] Lights
• [ ] Post processing data
• [ ] Air carving mask
• [ ] Liquid carving mask
• [ ] LightOn
• [ ] Tile entities
• [ ] Old biomes (< 1.18)

New features:

• [ ] entities/ folder format

Per-section data

Features used by Minestom:

• [ ] Biomes
• [ ] Blocks
  • [ ] Palette
  • [ ] Block indices inside Palette
• [ ] Sky light
• [ ] Block light

[jglrxavpok]

Outside of the """new""" entities/ region files, all readers are ready.

[jglrxavpok]

Was done in commit 6b386efa4f610f8abb7917326cd88f1b22ab76d4