Installation on CentOS (AWS Sagemaker)

[mspachecoagoro]

Hi, I am trying to build it on a Sagemaker notebook instance and I am facing an error when running cmake install

I am getting this error:

`...
[ 29%] Building C object ncgen/CMakeFiles/ncgen.dir/dump.c.o
[ 29%] Building C object ncgen/CMakeFiles/ncgen.dir/escapes.c.o
[ 29%] Building C object ncgen3/CMakeFiles/ncgen3.dir/ncgeny.c.o
[ 29%] Building C object ncdump/CMakeFiles/nccopy.dir/dimmap.c.o
[ 30%] Building C object ncdump/CMakeFiles/nccopy.dir/list.c.o
[ 30%] Building C object ncgen/CMakeFiles/ncgen.dir/f77data.c.o
[ 30%] Linking C executable nccopy
[ 30%] Building C object ncgen/CMakeFiles/ncgen.dir/genbin.c.o
../liblib/libnetcdf.so.19}: undefined reference to `H5free_memory'
collect2: error: ld returned 1 exit status
gmake[2]: *** [ncdump/nccopy] Error 1
gmake[1]: *** [ncdump/CMakeFiles/nccopy.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 31%] Building C object ncdump/CMakeFiles/ncdump.dir/indent.c.o
[ 31%] Building C object ncdump/CMakeFiles/ncdump.dir/nctime0.c.o
[ 31%] Building C object ncdump/CMakeFiles/ncdump.dir/utils.c.o
[ 31%] Building C object ncgen/CMakeFiles/ncgen.dir/genc.c.o
[ 31%] Building C object ncdump/CMakeFiles/ncdump.dir/nciter.c.o
[ 31%] Building C object ncgen/CMakeFiles/ncgen.dir/genchar.c.o
[ 31%] Linking C executable ncdump
[ 31%] Building C object ncgen/CMakeFiles/ncgen.dir/generate.c.o
[ 31%] Linking C executable ncgen3
../liblib/libnetcdf.so.19}: undefined reference to `H5free_memory'
collect2: error: ld returned 1 exit status
gmake[2]: *** [ncdump/ncdump] Error 1
gmake[1]: *** [ncdump/CMakeFiles/ncdump.dir/all] Error 2
[ 32%] Building C object ncgen/CMakeFiles/ncgen.dir/generr.c.o
../liblib/libnetcdf.so.19}: undefined reference to `H5free_memory'
collect2: error: ld returned 1 exit status
gmake[2]: *** [ncgen3/ncgen3] Error 1
gmake[1]: *** [ncgen3/CMakeFiles/ncgen3.dir/all] Error 2
[ 32%] Building C object ncgen/CMakeFiles/ncgen.dir/genf77.c.o
[ 32%] Building C object ncgen/CMakeFiles/ncgen.dir/genj.c.o
[ 32%] Building C object ncgen/CMakeFiles/ncgen.dir/genlib.c.o
[ 32%] Building C object ncgen/CMakeFiles/ncgen.dir/getfill.c.o
[ 32%] Building C object ncgen/CMakeFiles/ncgen.dir/jdata.c.o
[ 32%] Building C object ncgen/CMakeFiles/ncgen.dir/list.c.o
[ 33%] Building C object ncgen/CMakeFiles/ncgen.dir/main.c.o
[ 33%] Building C object ncgen/CMakeFiles/ncgen.dir/ncgeny.c.o
[ 33%] Building C object ncgen/CMakeFiles/ncgen.dir/semantics.c.o
[ 33%] Building C object ncgen/CMakeFiles/ncgen.dir/util.c.o
[ 33%] Linking C executable ncgen
../liblib/libnetcdf.so.19}: undefined reference to `H5free_memory'
collect2: error: ld returned 1 exit status
gmake[2]: *** [ncgen/ncgen] Error 1
gmake[1]: *** [ncgen/CMakeFiles/ncgen.dir/all] Error 2
gmake: *** [all] Error 2
Command failed: ['/usr/local/bin/cmake', '--build', '/home/ec2-user/SageMaker/cm-vsm/vsm/build/cget/cget/build/tmp-a11e7b5e29904e95bcce3cd6b67c64b2/build', '--config', 'Release', '--', '-j', '4']
Failed to build package Unidata/netcdf-c

**More details about the linux distribution:**
sh-4.2$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"`

Do you have any idea how to solve this issue?

[indrek-sunter]

I've yet to try building it on Amazon Linux. On SageMaker we have mostly used custom Ubuntu-based images with the necessary tools pre-built. However, the registration of Conda environments as Jupyter kernels for us does not work properly from the custom images. At this point we don't have a dedicated image with just cm-vsm and km_predict installed.

You could try creating a custom image for SageMaker. Some resources:

1. km_predict Dockerfile (currently not adapted for SageMaker)
2. AWS official examples for SageMaker Studio custom images
3. AWS official guide for preparing a custom SageMaker image

In the meantime I will try the installation on Amazon Linux.

Edit: I think the custom image would be needed anyway because all system changes in the Image Terminal only affect the overlay and next time the SageMaker instances would be created, everything would have to be reinstalled and rebuilt. First tests with a standard TensorFlow SageMaker image seemed to confirm this.

[indrek-sunter]

Are you using SageMaker Studio notebook or just SageMaker notebook? Which SageMaker image are you using?

Edit: Oh, the System Terminal is running on Amazon Linux. Are you using the System Terminal for building cm-vsm?

[mspachecoagoro]

Thanks @indrek-sunter for your comments and help.

I am using a notebook instance in Sagemaker. In this instance, I am using the terminal to build cm-vsm.

I successfully install all the other libraries inside a persistent conda environment (held inside SageMaker folder) required for km_predict, the only problematic part for me now is cm-vsm. I have configured a life-cycle script to load the environment as a kernel every time the instance is turned on (this part is not an issue for me now, I can share the lifecycle configuration if you wish). My application is not using Docker at this moment.

But if you could share an image containing cm-vsm and km_predict I would use Docker and it also would save my problem.

In the meanwhile, if you have a Google Colabs example working, it would also help me to explore KappaMask while I don't solve the issue with SageMaker. To be honest, I managed to install cm-vsm on Google Colabs, but unfortunately this tutorial did not work for me...

[indrek-sunter]

Interesting. I have been trying to do the same with the lifecycle configuration but it usually takes a long time before the kernel shows up, and once it does, there would be duplicate entries in the list. We would be interested in the configuration.

Thank you for letting us know about the issue with the tutorial! I informed my colleague but she has been a bit occupied recently.

I updated the Docker configuration and pushed the image to DockerHub: https://hub.docker.com/r/kappazeta/kappamask

By default it still fetches the S2 product from S3 and stores the output on S3. I haven't implemented the support for processing a product mounted via volumes yet. Additionally, I think that for the Docker image to make use of GPU, some more work is needed.

Regardless, I hope that the Docker image may be helpful.

[indrek-sunter]

@mspachecoagoro Sorry, I forgot to mention / ping.