search

denoland / deno

A modern runtime for JavaScript and TypeScript.
https://deno.com
MIT License
97k stars 5.36k forks source link

KV should allow larger values #22246

Closed RickeyWard closed 8 months ago

RickeyWard commented 8 months ago

KV imposes an arbitrary limit of 65kb for any value. When using KV locally its using SQLite which imposes practically no limits and works very well without them. it would be great if the limits could be overwritten with environment variables or some other solution.

cknight commented 8 months ago

The limit is not really arbitrary. KV on Deploy is backed by FoundationDB, not SQLite, and it is this backing DB which imposes the constraints for optimum performance. In order to allow the local CLI environment to match the Deploy environment the same constraints are applied, despite SQLite being capable of larger limits. While an option could be added for local development to have larger constraints, this sounds like it could lead to applications being developed against the larger limits and then running into issues when deployed to Deploy in the future.

RickeyWard commented 8 months ago

FoundationDB has a 100kb limit while DenoKV has a 64kib limit which is an interesting choice not to choose the same limit.

Users of FoundationDB say you can split chunks across keys and then range grab them and reassemble them so maybe I'll try that for KV with the SQLite driver.

I just don't like the idea of using a 3rd party SQLite driver when one is built in though I certainly understand the confusion it could cause.

cknight commented 8 months ago

It's possible Deploy is being conservative, or it finds it more efficient. In either case, have a look at https://github.com/kitsonk/kv-toolbox?tab=readme-ov-file#blob which splits large blobs into multiple KV entires and reassembles them with single get/set operations.

RickeyWard commented 8 months ago

Thanks for the heads up. I'll close this as I'm guessing there's not a lot can be done.

janzheng commented 1 month ago

It's possible Deploy is being conservative, or it finds it more efficient. In either case, have a look at https://github.com/kitsonk/kv-toolbox?tab=readme-ov-file#blob which splits large blobs into multiple KV entires and reassembles them with single get/set operations.

that looks promising but it's kind of odd for such an important use case to be trusted to a third party