Spectrum MPI license in container

[qhaas]

I'm trying to use a powerai derived container on an IBM AC922 (power9, RHEL7) that includes mpi4py in its conda environment for use with Spectrum MPI. Specifically, I'm using this container: docker run --name spectrum-mpi4py -it --env LICENSE=yes ibmcom/powerai:1.6.2-pytorch-ubuntu18.04-py36

However, I'm experiencing license errors when trying to use mpi related functionality inside the container, even with basic C based MPI programs. The commands below generated the error in c_mpi_hello_error.txt. The license script referenced by the error, accept_spectrum_mpi_license.sh, is not in the container.

# apt-get update && apt-get install -y build-essential cmake
...
$ find / -type f -name 'accept_spectrum_mpi_license.sh' 2> /dev/null | wc -l
0
$ export IBM_SPECTRUM_MPI_LICENSE_ACCEPT=yes
$ wget https://raw.githubusercontent.com/huyle333/mpi-hello-world/master/mpi_hello_world.c
$ mpicc mpi_hello_world.c
$ mpirun -n 2 a.out

Separately, the container's conda environment (wmlce) pulls a build targeting mpich/openmpi, despite the default environment using Spectrum MPI:

$ conda info mpi4py 2>/dev/null | grep -i mpich | wc -l
4
$ conda info mpi4py 2>/dev/null | grep -i openmpi | wc -l
4
$ conda info mpi4py 2>/dev/null | grep -i sp | wc -l
0
...
$ conda list | grep spectrum
spectrum-mpi              10.03              622.gfc88b70    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

Building mpi4py from source using the Spectrum MPI headers works, but I get that same license error when trying to run it.

Finally, merely attempting to get additional info regarding this MPI implementation using ompi_info generated the errors in ompi_info.txt.

[qhaas]

While we have a Spectrum MPI license, it looks like this is intentional behavior for the powerai container, which is isolated from the host's license information: https://github.com/IBM/powerai/issues/29#issuecomment-512005188

[hartb]

Yeah, I'm afraid this is true. The Spectrum MPI included in the WML CE containers is unaware of any host licensing, and is licensed only for use with the WML CE components themselves.

If you have your own Spectrum MPI license, though, it might be possible to expose the host SMPI into the container and then use that with some of the WML CE stuff. I think DDL will respect MPI_ROOT, and some of the other components may tolerate it as well.

[qhaas]

Looks like one might be able to build / test containers using the SMPI Community Edition and then when one needs to scale, mount the license info from the host?