search

ocean-data-factory-sweden / kso

Notebooks to upload/download marine footage, connect to a citizen science project, train machine learning models and publish marine biological observations.
GNU General Public License v3.0
4 stars 12 forks source link

Dockerfile nvidia starting image #198

Closed Diewertje11 closed 1 year ago

Diewertje11 commented 1 year ago

The dockerfile (the one that is for both repositories combined (data-management) and (object-detection)) currently first loads the nvidia cuda devel docker image to start with, in order to build the ffmpeg from scratch.

Then it starts over from a new image, copies the final installation of the ffmpeg and builds up the rest of the environment. In theory this should be possible to do with the runtime image, which has the advantage that it is smaller. However, when trying that, the dockerfile could not get through the builder test on github since it ran out of disk. This is the error that occurred:

Image

This error is resolved by using the devel image for a second time, instead of loading in this new image. However, now we end up with a larger image at the end (which is not a problem), but it is not the neatest solution. So this is something we can take a look at again in the future.

jannesgg commented 1 year ago

@Diewertje11 Have a look at the Github workflow changes. I have added a step which clears up some memory when the runner starts up, which has fixed some of these out of memory errors for me. Feel free to test this with the new workflow.

Diewertje11 commented 1 year ago

This indeed fixed the issue. I have changed the Dockerfile so that it can use the runtime image now the second time. See PR #212