Closed RickeyWard closed 8 months ago
The limit is not really arbitrary. KV on Deploy is backed by FoundationDB, not SQLite, and it is this backing DB which imposes the constraints for optimum performance. In order to allow the local CLI environment to match the Deploy environment the same constraints are applied, despite SQLite being capable of larger limits. While an option could be added for local development to have larger constraints, this sounds like it could lead to applications being developed against the larger limits and then running into issues when deployed to Deploy in the future.
FoundationDB has a 100kb limit while DenoKV has a 64kib limit which is an interesting choice not to choose the same limit.
Users of FoundationDB say you can split chunks across keys and then range grab them and reassemble them so maybe I'll try that for KV with the SQLite driver.
I just don't like the idea of using a 3rd party SQLite driver when one is built in though I certainly understand the confusion it could cause.
It's possible Deploy is being conservative, or it finds it more efficient. In either case, have a look at https://github.com/kitsonk/kv-toolbox?tab=readme-ov-file#blob which splits large blobs into multiple KV entires and reassembles them with single get/set operations.
Thanks for the heads up. I'll close this as I'm guessing there's not a lot can be done.
It's possible Deploy is being conservative, or it finds it more efficient. In either case, have a look at https://github.com/kitsonk/kv-toolbox?tab=readme-ov-file#blob which splits large blobs into multiple KV entires and reassembles them with single get/set operations.
that looks promising but it's kind of odd for such an important use case to be trusted to a third party
KV imposes an arbitrary limit of 65kb for any value. When using KV locally its using SQLite which imposes practically no limits and works very well without them. it would be great if the limits could be overwritten with environment variables or some other solution.