The problem: sqlite file big, chunk in sql file, uncompressed The goal: chunk data in gzipped files Side quest: Also address #48 and stream line chunk creation Side quest: It's becoming clear to me that u32 is not big enough to store all the data I need per block and since will require a data migration I might as well try out u64s next. It seems silly to think that sqlite would be the place for the extra data I need or separate per block data, it should all be in the chunk. Out of Scope: fixing slow complex chunk rendering during edits, terrain gen, transparencies or any sort of meshing/drawing changes.
Approach
A data migration, I think getting file store working side by side with sqlite on save without loading from it, then attempt to load from it.
Then remove the old data from sqlite. I think storing full Y columns makes sense, and also a chunk version data, so two u64s plus some extra header that has chunk version, and maybe an unused reserved space. So let's go with 3 u64s. Storing more data per file seems suboptimal? Opening many files would be slow, there's no concept currently of anything more than a chunk, so even going to Y columns is already fraught but there's some precedence for it in how ambient lighting code works.
Description
The problem: sqlite file big, chunk in sql file, uncompressed
The goal: chunk data in gzipped files
Side quest: Also address #48 and stream line chunk creation
Side quest: It's becoming clear to me that u32 is not big enough to store all the data I need per block and since will require a data migration I might as well try out u64s next. It seems silly to think that sqlite would be the place for the extra data I need or separate per block data, it should all be in the chunk.
Out of Scope: fixing slow complex chunk rendering during edits, terrain gen, transparencies or any sort of meshing/drawing changes.
Approach
A data migration, I think getting file store working side by side with sqlite on save without loading from it, then attempt to load from it. Then remove the old data from sqlite. I think storing full Y columns makes sense, and also a chunk version data, so two u64s plus some extra header that has chunk version, and maybe an unused reserved space. So let's go with 3 u64s. Storing more data per file seems suboptimal? Opening many files would be slow, there's no concept currently of anything more than a chunk, so even going to Y columns is already fraught but there's some precedence for it in how ambient lighting code works.