How to cache chunk data?

[Jarred-Sumner]

Currently I have multiplayer setup to stream the map chunks over websockets as noa requests world data. Its sent using protobuf and each chunk is about 4 KB of data. The protobuf is parsed in a web worker and then the data is transferred over to the main thread

However, building the array of block IDs on the server is slow (~5ms-20ms/chunk) since these are large arrays and it asks for ~thousand of them on load. Individual chunks don't change often, so in theory, it should be fairly cache-able.

But I don't think the IDs for the worldData chunks are the same on every client or even between page loads (regardless of world origin offset)?

How would you suggest caching the data so that I could stick a blob in redis or a database and just send that to the client when they provide the ID?

[fenomas]

So, the caveat here is that my own noa game isn't multiplayer, so I'm not super confident that the engine correctly covers your use case, and probably need your feedback.

With that said: the IDs attached to worldDataNeeded events are static across clients and reloads, so it should be safe to use them as a key for caching -- provided only that every client uses the same chunkSize.

Specifically, the ID attached to world data events looks like:

[
   (x / chunkSize) | 0,
   (y / chunkSize) | 0,
   (z / chunkSize) | 0,
   worldName,
].join('|')

where x,y,z are the world coords of the lower-left voxel in that chunk (or any voxel in the chunk, really) and worldName is the current value of noa.worldName. (The world name is there for disambiguating sets of world data - like overworld / nether in minecraft.)

So, in general it should be safe to use those IDs as the key for caching world data, but when a single voxel ID changes you'll need to go and poke that value into the cached data for that chunk (on the server, if that's the single source of truth for voxel data).

Note: a rudimentary version of this is implemented in the test world, here. When a chunk is unloaded it caches the voxel data by the request ID, and when a chunk is requested it only generates new world data if there's no cache. However this is just client side, so it doesn't need to update cache when a single voxel changes, only when a whole chunk gets unloaded (because the player moved out of range).

[Jarred-Sumner]

oh great.

I'll just use this ID. I thought it was not globally unique but my caching code was just wrong 😄

[fenomas]

Heh gotcha. As a side note, there's actually currently no API that exposes the mapping between voxel locations and chunk ID strings, so the handling above is kind of coupled to engine internals. The engine should probably expose a proper API related to this, but I haven't really thought out what the right metaphor is for it. But anyway it's safe for now, and since you have the engine cloned locally for tweaking etc.