Conda dependency baseos for ctsm-fates in NorESM Galaxy project

[glemieux]

The Nordic Earth System Model (NorESM) is integrating FATES and working on utilizing its model within the Galaxy project to enable web-enabled, cloud-hosted research: https://github.com/NordicESMhub/galaxy-tools/issues/39. Since the Galaxy project makes use of containers to promote reproducible science and cloud computing, we are collaborating with NorESM to bring containerized hlm-fates to Galaxy. The hope is that this will help further experience with and adoption of the FATES model.

Galaxy utilizes conda's package management infrastructure to distribute and maintain containers, specifically bioconda containers. As such, it is ideal that the dockerfile recipes utilize conda-based libraries for the containerized application dependencies.

[glemieux]

Attempted to adopt the galaxy workflow of automating the container build. Currently unable to get the local tests to run. Following this documentation: https://docs.galaxyproject.org/en/latest/admin/special_topics/mulled_containers.html#building-docker-containers-for-local-conda-packages. Here is the error:

14:29 $ mulled-build build-and-test 'ctsm-fates=14.0.0' -c file://home/gregorylemieux/local/miniconda3/envs/galaxy-py3.8/conda-bld --test 'create_newcase --help'
Could not execute: '['docker', 'run', 'continuumio/miniconda3:latest', 'conda', 'search', '--full-name', '--json', '--platform', 'linux-64', '--override-channels', '--channel', 'conda-forge', '--channel', 'bioconda', 'ctsm-fates']'
Executing: ./involucro -v=2 -f /home/gregorylemieux/local/miniconda3/envs/galaxy-py3.8/lib/python3.8/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=file://home/gregorylemieux/local/miniconda3/envs/galaxy-py3.8/conda-bld -set TARGETS=ctsm-fates=14.0.0 -set REPO=quay.io/biocontainers/ctsm-fates:14.0.0 -set BINDS=build/dist:/usr/local/,/home/gregorylemieux/local/miniconda3/envs/galaxy-py3.8/conda-bld:/home/gregorylemieux/local/miniconda3/envs/galaxy-py3.8/conda-bld -set DEST_BASE_IMAGE=bgruening/busybox-bash:0.1 -set 'TEST=create_newcase --help' build-and-test
.[Oct 20 14:30:01] DEBU Run file [/home/gregorylemieux/local/miniconda3/envs/galaxy-py3.8/lib/python3.8/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
[Oct 20 14:30:01] INFO Invoke Task [build]
[Oct 20 14:30:01] STEP Run image [continuumio/miniconda3:latest] with command [[rm -rf /data/dist]]
[Oct 20 14:30:01] DEBU Creating container [step-d2f64e6636]
[Oct 20 14:30:01] DEBU Created container [26dbca89dc60 step-d2f64e6636], starting it
[Oct 20 14:30:02] DEBU Container [26dbca89dc60 step-d2f64e6636] started, waiting for completion
[Oct 20 14:30:02] DEBU Container [26dbca89dc60 step-d2f64e6636] completed with exit code [0] as expected
[Oct 20 14:30:02] DEBU Container [26dbca89dc60 step-d2f64e6636] removed
[Oct 20 14:30:02] STEP Run image [continuumio/miniconda3:latest] with command [[/bin/sh -c conda install  -c 'file://home/gregorylemieux/local/miniconda3/envs/galaxy-py3.8/conda-bld'  'ctsm-fates=14.0.0' -p /usr/local --copy --yes --quiet]]
[Oct 20 14:30:02] DEBU Creating container [step-19eb16c8f8]
[Oct 20 14:30:02] DEBU Created container [62829da39120 step-19eb16c8f8], starting it
[Oct 20 14:30:03] DEBU Container [62829da39120 step-19eb16c8f8] started, waiting for completion
[Oct 20 14:30:05] SOUT Collecting package metadata (current_repodata.json): ...working... done
[Oct 20 14:30:05] SOUT Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
[Oct 20 14:30:05] SOUT Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
[Oct 20 14:30:09] SOUT Collecting package metadata (repodata.json): ...working... done
[Oct 20 14:30:09] SOUT Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
[Oct 20 14:30:10] SOUT Solving environment: ...working...
[Oct 20 14:30:10] SOUT Found conflicts! Looking for incompatible packages.
[Oct 20 14:30:10] SOUT This can take several minutes.  Press CTRL-C to abort.
[Oct 20 14:30:10] SOUT failed
[Oct 20 14:30:10] SERR 
[Oct 20 14:30:10] SERR ResolvePackageNotFound:
[Oct 20 14:30:10] SERR - libgcc-ng[version='>=9.3.0']
[Oct 20 14:30:10] SERR 
[Oct 20 14:30:11] ERRO Task processing failed: Unexpected exit code [1] of container [62829da39120 step-19eb16c8f8], container preserved

Got help from @bgruening about error on gitter, but I had to drop this to pivot to other work before I was able to resolve this.

Test repo for this is here: https://github.com/glemieux/staged-recipes/tree/ctsm-fates. Based on recommendations from @annefou, attempted to submit a PR to conda-forge, but has failed checks for linux. Created an issue for that here: https://github.com/glemieux/staged-recipes/issues/1

[glemieux]

Looking at alternative method of providing existing dockerfile recipe to fit within the Galaxy workflow if possible.

[annefou]

Yes I have seen this kind of error; right now I can't remember exactly why and when but i will look at it.

[bgruening]

@glemieux your conda package failed because it does not work under python3 can this be?

[annefou]

The python errors in statistical_ensemble_test should not be a problem (I mean not for the conda build). The license is missing: File "/opt/conda/lib/python3.8/site-packages/conda_build/build.py", line 783, in copy_license raise ValueError("License file given in about/license_file ({}) does not exist in " ValueError: License file given in about/license_file (/home/conda/staged-recipes-copy/recipes/ctsm-fates/LICENSE.txt) does not exist in source root dir or in recipe root dir (with meta.yaml)

[annefou]

in meta.yaml:

license_file: LICENSE

While now you have

license_file: LICENSE.txt

And this is my fault because I made this error and corrected it later for fates-emerald. The conda-forge recipe for fates-emerald is at https://github.com/conda-forge/fates-emerald-feedstock

[bgruening]

But this is also hinting at a python2 problem: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=230158&view=logs&j=10bda42a-d420-5c28-a43f-bc4e66405e5c&t=bef775d0-a1a2-55a3-9721-17147029baf0&l=885

[annefou]

Yes, I also have it in my conda recipe... It passes even with these errors (this part is not used; additional tools still in python 2; I mean for print statements... Could easily be migrated to python 3; i am not sure who would be using these).

[glemieux]

Sorry for the late replies. Thanks for all the feedback. I'm guessing that this is low-hanging fruit that the CIME team just hasn't prioritized due to more pressing efforts. My guess is that they have to support python 2 by default on their supported machines anyway so their isn't a huge impetus to fix this. I'll double check with them though.

Thanks for the pointers to the errors btw; I simply missed seeing that I could click through to the azure job report.

[glemieux]

After talking with one of the CTSM engineers at NCAR, he submitted an issue to address the python2 print statements: https://github.com/ESMCI/cime/issues/3776