search

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
539 stars 29 forks source link

693 zfs diff is too slow #695

Closed lukemarsden closed 5 years ago

lukemarsden commented 5 years ago

Ready for review

lukemarsden commented 5 years ago

Pipeline: https://gitlab.dotmesh.com/dotmesh/dotmesh/pipelines/19631

lukemarsden commented 5 years ago

Results from interactive testing:

yay this caching implementation is working really well now when i start downloading a big file, it stops using the result cache because dirtyData != 0 each time it moves the snapshot forwards when i stop the download, it notices that the fs stops changing and starts using the cache for every result so it’s not having to iterate over all the files (or do any snaps, mounts etc) when the filesystem is quiet and in all cases, it’s caching the file list from the latest dotmesh commit, so it’s only doing 50% of the work it was doing before

lukemarsden commented 5 years ago

Should see slightly improved unzip performance as well, as there'll be less file I/O now hopefully.