search

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
539 stars 29 forks source link

preludes: inefficient AND unnecessary #700

Open lukemarsden opened 5 years ago

lukemarsden commented 5 years ago

the way ConsumePrelude works is that it reads one byte at a time, and every time it reads a byte, it checks whether the end marker is in the entire string. This is O(N^2) at best... and destroys performance with non-trivial sizes of metadata.

we don't need preludes anymore, since zfs user props are no longer in use.

lukemarsden commented 5 years ago

https://github.com/dotmesh-io/dotmesh/pull/702 did a basic fix by emptying out the preludes we send but leaving this issue open for ripping out all the prelude infrastructure