Limit data movement during build

VIDA-NYU / reproserver

A web application reproducing ReproZip packages in the cloud.

https://server.reprozip.org/

BSD 3-Clause "New" or "Revised" License

10 stars 8 forks source link

Limit data movement during build #42

Closed remram44 closed 3 years ago

remram44 commented 3 years ago

Originally opened 2019-09-11 16:59 EDT by @remram44

The server is made slower by the need to move data around.

RPZ is received from upload / downloaded from repo
uploaded to S3
downloaded from S3 to builder container
uploaded to Docker-in-Docker in builder pod
then image is uploaded to registry

remram44 commented 3 years ago

Originally posted by @remram44

GITLAB-41 would help with this eliminating one round-trip if we build and run in the same pod (with the same docker daemon).

Originally posted by @remram44

Also relevant: https://github.com/VIDA-NYU/reprozip/issues/274

remram44 commented 3 years ago

I thought REPROZIP-375 would help (using RUN --mount untar... instead of COPY data.tgz then RUN untar data.tgz) but it doesn't seem compatible with userns-remap.

remram44 commented 3 years ago

9e004ac1 makes it so we don't wait for the push to the registry before starting to run.

remram44 commented 3 years ago

Probably acceptable now. Though more improvements would be welcome, closing.