Improve docker build scripts

vfdev-5 commented 3 years ago

🚀 Feature

Currently, we can build and publish dockers using Circle CI and it works more or less well, however, PR's modifying Dockerfile can not build docker images to ensure PR correctness (related to the way how the PR is checkout on Circle CI).

The idea is

refactor all build_all.sh scripts (docker/main/build_all.sh, docker/hvd/build_all.sh and docker/msdp/build_all.sh) into a single shell script that could build a single docker image.
use github actions to build only on PR a single docker image and introduce a single yml file for that (to see what can be done).
- publishing will use existing scripts
add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2 ; maybe something else.

To Do list:

[x] refactor build_all.sh into build.sh
[x] run docker image build for PR on GA
[x] run tests on built image. Tests
- For all images
- can import torch and its version == required one
- can import ignite and its version == required one
- for all -vision images
- can import opencv without driver issue
- for all horovod images
- can import horovod and its version == required one
- for all msdp images
- can import deepspeed and its version == required one

cc @fco-dv

sparkingdark commented 3 years ago

I am not sure about the GitHub actions, that build the docker image from the pr, but I think the first point is an idea to merge all shell file to one main shell script.

what is the difference between the folders main , hvd, and msdp

vfdev-5 commented 3 years ago

what is the difference between the folders main , hvd, and msdp

@sparkingdark checkout https://github.com/pytorch/ignite/tree/master/docker and installation section. It is about having different frameworks in addition to pytorch and ignite.

sparkingdark commented 3 years ago

Okay I will check that.

vfdev-5 commented 3 years ago

@trsvchn would you like to tackle this issue ?

trsvchn commented 3 years ago

@vfdev-5 sure!

vfdev-5 commented 3 years ago

I'll start on this point

refactor all build_all.sh scripts (docker/main/build_all.sh, docker/hvd/build_all.sh and docker/msdp/build_all.sh) into a single shell script that could build a single docker image.

If you could help with:

create gtihub actions
add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2 ; maybe something else.

would be great !

trsvchn commented 3 years ago

@vfdev-5 Yes, I am going to start from this:

create gtihub actions
add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2 ; maybe something else.

vfdev-5 commented 3 years ago

One another thing I was also thinking about is about building docker images on "release" event. Currently, everything happens in parallel:

binary distributions to pypi and conda
docker image build

As "binary distributions to pypi and conda" faster than "docker image build", layer responsible for fetching the lastest ignite can get the released version. Maybe, we should force to wait "docker image build"...

trsvchn commented 3 years ago

@vfdev-5 I think it possible to trigger workflow manually, so we can trigger "docker image build" after "binary distributions to pypi and conda"

vfdev-5 commented 3 years ago

I'm not that fan of manual trigger, I was thinking about naive "check last version and wait 1 minute"...

trsvchn commented 3 years ago

@vfdev-5 Oh, no "manually" means using a kind workaround with manual events

https://docs.github.com/en/actions/reference/events-that-trigger-workflows#manual-events

I've found this solution

https://github.community/t/trigger-an-action-upon-completion-of-another-action/17642/2

vfdev-5 commented 3 years ago

That's could be great !

Asrst commented 1 year ago

@vfdev-5,

Hi, Can I work on task 3 to close this issue (add a test inside docker folder as a python script to check docker image versions for : torch, ignite ; can import cv2) ?

vfdev-5 commented 1 year ago

@Asrst thanks for asking. Actually, we already did that here: https://github.com/pytorch/ignite/blob/master/docker/test_image.py I'll close this issue as done.

if you would like to contribute, please pick another "help wanted" issue: https://github.com/pytorch/ignite/labels/help wanted

pytorch / ignite

Improve docker build scripts #1644

🚀 Feature