integration with HPCCM (HPC Container Maker)

[boegel]

We should look into leveraging the functionality provided by https://github.com/NVIDIA/hpc-container-maker to generate Docker/Singularity recipes, rather than implementing that functionality ourselves.

The HPCCM developers have re-introduced compatibility with Python 2.7 (cfr. https://github.com/NVIDIA/hpc-container-maker/pull/28) basically by our request (https://github.com/NVIDIA/hpc-container-maker/pull/28), and are also looking into integrating with EasyBuild (cfr. https://github.com/NVIDIA/hpc-container-maker/pull/28 and https://github.com/NVIDIA/hpc-container-maker/pull/28).

We may need to wait a bit until their API stabilizes though...

cc @shahzebsiddiqui, @FooBarQuaxx

[samcmill]

FYI: PyPi package at https://pypi.org/project/hpccm/

[boegel]

HPCCM is now also available on PyPI https://pypi.org/project/hpccm/, that's certainly going to be helpful.

@samcmill How stable is the hpccm API currently in terms of using it as a library? Should we expect things to move around in the near future? It seems to me that the hpccm package could maybe be a bit more organized (there are no subpackages, it's one big flat mix of things), any plans of refactoring that?

[samcmill]

There are no plans right now to significantly change the API. The user facing "API", meaning the recipe definition, has not changed since inception.

There are no plans right now to re-organize the package structure, but that could be useful for the long term. If you think it's important, I can look into it soon.

I suspect you want to either call hpccm.recipe() with a recipe file that that you have generated for a "loose" integration (essentially a variant of https://github.com/NVIDIA/hpc-container-maker/blob/master/recipes/easybuild.py), or call the primitives (hpccm.shell, hpccm.environment, etc.) directly for a "tight" integration. The building blocks (hpccm.openmpi, etc.) and templates (hpccm.ConfigureMake, etc.) overlap with existing EB functionality and probably are not interesting.

One HPCCM enhancement that has been suggested would be to have the option to output a "bash script" in addition to Dockerfiles and Singularity recipe files. With that capability, EB may be able to leverage HPCCM for a single backend to support both bare metal and containers (Docker & Singularity)?

[boegel]

@samcmill Thanks for clarifying the status of the HPCCM API.

I think it makes sense to re-organize the hpccm package a bit, for example by having a subpackage for Python modules for building blocks (e.g. hpccm/blocks) and another for templates (hpccm/templates), just to make things a bit clearer.

But you're right that this won't affect EasyBuild too much, as long as the "library" functionality like recipe, and maybe shell/environment stay in the top-level hpccm package (and even if that's moved, it should be a matter of changing the import statements).

W.r.t. the bash script functionality: I'm not sure how that would be interesting for EasyBuild...

The idea with leveraging HPCCM is mainly to avoid that we have to implement the generation of Docker/Singularity definition files ourselves. In those recipes, eb itself would be used again to do the actual installation. So, a bash script generated by HPCCM would include an eb command again, which would pretty much be the same as the eb command used to generate the bash script... Unless I'm missing something?

[samcmill]

@boegel I am not familiar with the EasyBuild internals, but presumably eb ultimately invokes system, environ, file.write, etc. (or equivalents) to actually do the low level installation. The idea behind the bash script suggestion would be to build a bash script to do those low level operations and then invoke the bash script from eb. (This wouldn't have to be monolithic, it could be per component.)

This would allow a single backend regardless of the target (bare metal, Docker, Singularity). This way Docker and Singularity would not be "second-class citizens" relative to bare metal. For instance, natively supporting Docker layering capabilities so that intermediate builds can be cached and if eb does not complete successfully when building a container, all the progress up to that point is not lost. (This happens to me periodically due to timeouts.) You can manually layer intermediate builds to mitigate this, but that is cumbersome.

I am not suggesting this for the initial integration, that should just be calling 'eb' from within the container image build spec.

I'll look into re-organizing the HPCCM package, but it probably won't be high priority.