search

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
539 stars 29 forks source link

support uploading multiple files in a tarball to S3 API #754

Closed lukemarsden closed 4 years ago

lukemarsden commented 4 years ago

to avoid locking complexity in dotscience-anywhere mode python lib, support uploading multiple files into S3 API in a single go by POSTing a tarball, which gets untarred on the fly, and a single commit made therein.

will make uploads of multiple files to S3 API much faster as well, ds run --upload-path can be updated to use this.

required to implement this design in https://github.com/dotmesh-io/dotscience-python/issues/28: https://github.com/dotmesh-io/dotscience-python/pull/29/files#diff-2f43d6c5b812eaf70dccaf083fad9967R354-R365

lukemarsden commented 4 years ago

should unpack the tarball "into the root" of the POSTed-to location, so that multiple files in different places can be put in place using this mechanism.

rusenask commented 4 years ago

added in https://github.com/dotmesh-io/dotmesh/pull/771 and https://github.com/dotmesh-io/dotmesh/pull/772