search

dchaley / deepcell-imaging

Tools & guidance to scale DeepCell imaging on Google Cloud Batch
7 stars 2 forks source link

Do quick pass on container optimization #209

Open dchaley opened 3 months ago

dchaley commented 3 months ago

After #192 we shrank the container from ~8gb to ~3gb.

Do a quick pass on the resulting container using du --max-depth=1 -h at root file system, to see if there's more stuff we can get rid of. Probably some temporary files at least.

Davidnet commented 3 months ago

I can take a look at this, but something I do to minimize the amount of storage (specially with nvidia drivers) is to use a multistage build https://docs.docker.com/build/building/multi-stage/.

We will need only to move the site-package of your pip install and the binaries of the apt

Davidnet commented 3 months ago

I could take a look if you want and can I ask how large is the current container that you have?

dchaley commented 3 months ago

Ooh, that's very nice. Yeah, there's a lot of build tools we can remove once the pip dependencies are installed.

Here's our Dockerfile, here's the pip requirements.

dchaley commented 1 month ago

Example optimization: for the google sdk we had to install: can we follow this approach? https://github.com/tonymet/gcloud-lite/blob/master/Dockerfile