search

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
539 stars 29 forks source link

Investigate metadata size limits in Dotmesh, and decide how to increase them #593

Open alaric-dotmesh opened 6 years ago

alaric-dotmesh commented 6 years ago

Dotscience makes commits with lots of metadata, and even more so with per-file provenance tracking. We'll hit limits soon.

Hypothesis: It's better to keep DS storing things in metadata because it's a good model, and improve the storage/processing in DM, than to hack something in DS because of the limits in DM.

Find out what the limits to number/size of metadata keys are.

Work out how to raise them.

ACs:

rusenask commented 6 years ago

Needs more details. How much actually is "lots of metadata" and what is a reasonable amount?

I think we will always be limited here by the etcd value size.