search

fireproof-storage / fireproof

Realtime database, runs anywhere. Install Fireproof in your front-end app or edge function, and sync data via any backend.
https://fireproof.storage
Other
303 stars 16 forks source link

compact before upload #281

Closed jchris closed 1 week ago

jchris commented 2 weeks ago

when the user hasn't modified in a while they can end up with local car files that they dont need to send 100 of at a time. so we can compact before send as described here: https://github.com/fireproof-storage/fireproof/pull/280#issuecomment-2397824553

at beginning of _doProcess coalece all operations and noLoaderOps into a single car file (or files if fragment or threshold are active). file uploads should not be modified. upload the new car file, with the latest db meta pointing to it. the header of the new car file should be from the most recent car file in the coalced group. coalce can be block level (default compaction) not database compaction

jchris commented 2 weeks ago

maybe useful code https://github.com/fireproof-storage/fireproof/pull/282/files

jchris commented 2 weeks ago

we have to compact, not coalesce

jchris commented 2 weeks ago

I tried to do it here but not quite https://github.com/fireproof-storage/fireproof/pull/283

jchris commented 2 weeks ago

solved by https://github.com/fireproof-storage/fireproof/pull/286