OME Files Docker files: rationalization

[sbesson]

State of the art

Docker images for the OME Files C++ library have been originally developed in decoupled repositories (Ubuntu 16.04, CentOS 7). As we increase the usage of these Docker images for our application development and use it for testing the downstream components (ome-files-performance, ome-qtwidgets, ome-files-py), we might reconsider this strategy.

Proposal

Unlike the OMERO server and OMERO Web Docker images, the OME Files Dockerfile do not require to consume built artifacts but rather build directly the library from the sources. It should be possible to re-use the strategy used byome-files-py and described as follows:

• the Dockerfiles for all supported platforms should be maintained directly within the source code repository in a top-level file
• each Dockerfile should be suffixed with a string identifying the platform i.e. Dockerfile.{c7,u1604}
• each Dockerfile should use COPY .
• individual Docker Hub repositories should be configured per platform and consume the relevant Dockerfile

The table below summarizes the current and proposed locations for the Dockerfiles:

Platform	Component	Travis	Docker Hub	Current location	Proposed location
Ubuntu 16.04	OME Files C++	N	ome-files-cpp-u1604	Dockerfile	Dockerfile.u1604
CentOS7	OME Files C++	N	ome-files-cpp-c7	Dockerfile	Dockerfile.c7
Ubuntu 16.04	OME Files Python	Y	ome-files-py-u1604	Dockerfile.u1604	unchanged
CentOS7	OME Files Python	Y	ome-files-py-c7	Dockerfile.c7	unchanged
Ubuntu 16.04	OME Files Qt Widgets	Y	none	Dockerfile	Dockerfile.u1604
CentOS7	OME Files Qt Widgets	Y	none	Dockerfile	Dockerfile.c7

Implementation

The principal technical caveat is that Docker Hub does not allow to switch the source GitHub repository for an existing repositories. Preserving the existing tags of the existing ome-files-cpp-u1604 images might be achievable using the following steps:

1. locally retrieve the Docker tags from https://hub.docker.com/r/openmicroscopy/ome-files-cpp-u1604
2. open a PR against ome-files-cpp migrating Dockerfile{.c7,u1604} from the decoupled Docker repositories
3. delete the Hub https://hub.docker.com/r/snoopycrimecop/ome-files-cpp-c7/ and https://hub.docker.com/r/snoopycrimecop/ome-files-cpp-u1604/ repositories
4. recreate the Hub https://hub.docker.com/r/snoopycrimecop/ome-files-cpp-c7/ and https://hub.docker.com/r/snoopycrimecop/ome-files-cpp-u1604/ repositories and use https://github.com/snoopycrimecop/ome-files-cpp as the source GitHub repository
5. optionally push the ome-files-cpp-u1604 tags retrieved in step 1 to https://hub.docker.com/r/snoopycrimecop/ome-files-cpp-u1604
6. test the daily merge workflow
7. if successful, delete the https://hub.docker.com/r/openmicroscopy/ome-files-cpp-c7 and https://hub.docker.com/r/openmicroscopy/ome-files-cpp-u1604 Hub repositories and recreate them using https://github.com/ome/ome-files-cpp as the source GitHub repository
8. push the ome-files-cpp-u1604 Docker tags from step 1 to https://hub.docker.com/r/openmicroscopy/ome-files-cpp-u1604
9. merge the Docker PR opened in 2 and let the upstream repositories create the latest images
10. either mark the GitHub openmicroscopy/ome-files-cpp-docker-{u1604,c7} repositories as deprecated or delete them and set up GitHub to redirect them to ome/ome-files-cpp
11. review all existing forks on GitHub and Docker Hub and notify consumers

Future steps

Immediately, the Docker-based build files would be maintained in sync with the code (which allows to track changes). In the mid-term, we could even have the base OME Files C++ images built and tested in Travis. Doing so probably requires to think the requirements to splitting the ome-model component down into a base image per platform.

[sbesson]

Another issue with this set of Docker files was highlighted by the thread in https://github.com/ome/ome-common-cpp/pull/57#issuecomment-398674719.

In addition to the problem of finding these repositories and maintaining them, in the case of coupled PRs across the various C++ components, getting the Docker files building and Travis sucessful becomes complex for little added value.

With the birth of https://github.com/ome/ome-files-build, an alternative proposal would be:

• migrate the Dockerfile.{c7,u1604} to the top-level of the ome-files-build super-repository and let it build the component from the submodules directly to create an image with all the OME Files components (common, model, C++, Qt)
• deprecating the individual Docker images for each component in DockerHub (or redirecting them)
• archiving these repositories
• the only unknown might be the Python binding i.e. https://github.com/ome/ome-files-py since the building and testing logic is not completely integrated into the Cmake logic and would require some special handling

[joshmoore]

One addition: https://gitlab.com/openmicroscopy/incubator/omero-all (renamed to omero-server-build?) and https://github.com/ome/bio-formats-build/pull/18 should get the same style of Dockerfile(s) (cc @rgozim) including a few design thoughts:

• one per supported host (where that's useful)
• perhaps a flag to git submodule update --init within the dockerfile if it wasn't already done
• unified default of either "master" or some tag
• layered strategy to minimize size
• clear instructions on how to scc merge where branch names differ
• a plan for omero-web-build and and omero-ms-build and similar
• copy .git only when necessary (triggers unnecessary rebuilds)
• save cache layers in travis or use docker build --cache-from (experimental)