$SCIF_* environment variables not available in %appinstall

[bilke]

I am building several apps into one container

...
%post
  ...
  scif install /foo.sci
  scif install /bar.scif

The app bar depends on foo.

foo.scif:

...
%appenv foo
  CC=$SCIF_APPS/foo/bin/gcc
  CXX=$SCIF_APPS/foo/bin/g++

bar.scif:

...
%appinstall bar
  # does not work: source $SCIF_APPENV_foo
  source /scif/apps/foo/scif/environment.sh
  echo $CC # prints: /foo/bin/gcc

Note that the conent of $SCIF_APPS is missing and also $SCIF_APPENV_foo seems to be not defined.

[vsoch]

I will check this out for you! It should be the case that the application being installed has variables exposed for itself and global exposed (so missing $SCIF_APPS is a bug) , but not for other applications (because if you were to install one without the other it would break). That said, I'm well aware that there are many times when a user has this kind of dependency, and placing the different applications in the same recipe file implies that they are used together (and remove Foo from bar at your own risk!). so we might want to make all those variables available after all.

A follow up question for my thinking - given that you are installing to a host that already has applications, should it be the case that those applications are exposed as well? My instinct is saying yes but I wanted to get your feedback.

[vsoch]

Actually of course we need to support the environment variables from other apps installed, otherwise the double recipe doesn't work! bonk. Can you give me your entire recipe (singularity and scif) files so I can work with them to test this out?

[vsoch]

I think something important to talk about is the statement someone is making with having foo and bar in separate files. It suggests that they don't depend on one another. If it is the case that foo and bar do depend on one another, they should be installed from the same recipe, and you can control the ordering by the order shown in the recipe. I use an ordered dict for the lookup so the ordering should be maintained based on the order found in the file. So that said, I would propose the following changes:

1. apps that depend on one another should be included in the same recipe file.
2. when installed from the same recipe, we should expose the other application environment variables
3. based on 1, if a user installs a new recipe to an existing system that references one of the variables, given different files, we won't source the old variables. I think I could probably go either way on this one, but given that containers are the biggest use case and tend to be "one build sort of deal" I think this is reasonable. Is there a scenario I'm not thinking of here?

okay! So I have a branch for you to test, I'll open a PR for it, here you go! https://github.com/vsoch/scif/pull/41/files

Here is me walking through testing it. First we have our one foobar.scif recipe, and we do this because we have dependencies between foo and bar. Foo is first so we install it first.

%appenv foo
  CC=$SCIF_APPS/foo/bin/gcc
  CXX=$SCIF_APPS/foo/bin/g++
%appinstall bar
  echo "Command  env | grep SCIF"
  env | grep SCIF
  echo "Sourcing foo environment"
  . $SCIF_APPENV_foo
  echo " Printing variable CC"
  echo $CC # prints: /foo/bin/gcc

Here we have our Singularity recipe to install foobar.scif

Bootstrap: docker
From: continuumio/miniconda3

# sudo singularity build foobar Singularity

%files
   foobar.scif

%post
    apt-get update
    git clone -b fix/environment https://www.github.com/vsoch/scif
    cd scif
    /opt/conda/bin/python setup.py install
    /opt/conda/bin/scif install /foobar.scif

Note that it's cloning the scif client from the branch for the pull request. Now we run the build! I'll skip over the non scif stuffs.

$ sudo singularity build foobar Singularity
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
...
+ cd scif
+ /opt/conda/bin/python setup.py install
...
Finished processing dependencies for scif==0.0.72
+ /opt/conda/bin/scif install /foobar.scif
Installing base at /scif
+ apprun     foo
+ appenv     foo
+ appinstall bar
Command  env | grep SCIF         # Here is where we are seeing ALL SCIF variables in the environment!
SCIF_APPDATA=/scif/data/bar
SCIF_APPLIB_foo=/scif/apps/foo/lib
SCIF_APPRUN=/scif/apps/bar/scif/runscript
SCIF_APPRECIPE=/scif/apps/bar/scif/bar.scif
SCIF_APPROOT=/scif/apps/bar
SCIF_APPMETA_foo=/scif/apps/foo/scif
SCIF_APPBIN_foo=/scif/apps/foo/bin
SCIF_APPENV_bar=/scif/apps/bar/scif/environment.sh
SCIF_APPHELP_foo=/scif/apps/foo/scif/runscript.help
SCIF_APPNAME=bar
SCIF_DATA=/scif/data
SCIF_APPLIB=/scif/apps/bar/lib
SCIF_APPLABELS_bar=/scif/apps/bar/scif/labels.json
SCIF_APPMETA=/scif/apps/bar/scif
SCIF_APPBIN=/scif/apps/bar/bin
SCIF_APPTEST_bar=/scif/apps/bar/scif/test.sh
SCIF_APPDATA_bar=/scif/data/bar
SCIF_APPHELP=/scif/apps/bar/scif/runscript.help
SCIF_APPRUN_bar=/scif/apps/bar/scif/runscript
SCIF_APPRECIPE_bar=/scif/apps/bar/scif/bar.scif
SCIF_APPROOT_bar=/scif/apps/bar
SCIF_APPENV_foo=/scif/apps/foo/scif/environment.sh
SCIF_APPNAME_bar=bar
SCIF_APPLIB_bar=/scif/apps/bar/lib
SCIF_APPLABELS_foo=/scif/apps/foo/scif/labels.json
SCIF_APPMETA_bar=/scif/apps/bar/scif
SCIF_APPBIN_bar=/scif/apps/bar/bin
SCIF_APPTEST_foo=/scif/apps/foo/scif/test.sh
SCIF_APPENV=/scif/apps/bar/scif/environment.sh
SCIF_APPDATA_foo=/scif/data/foo
SCIF_APPHELP_bar=/scif/apps/bar/scif/runscript.help
SCIF_APPRUN_foo=/scif/apps/foo/scif/runscript
SCIF_APPRECIPE_foo=/scif/apps/foo/scif/foo.scif
SCIF_APPROOT_foo=/scif/apps/foo
SCIF_MESSAGELEVEL=INFO
SCIF_APPS=/scif/apps
SCIF_APPLABELS=/scif/apps/bar/scif/labels.json
SCIF_APPNAME_foo=foo
SCIF_APPTEST=/scif/apps/bar/scif/test.sh
Sourcing foo environment           # Here is where the source happens and we print result
 Printing variable CC
/scif/apps/foo/bin/gcc
Finalizing Singularity container
Calculating final size for metadata...
Skipping checks
Building Singularity FS image...
Building Singularity SIF container image...
Singularity container built: foobar
Cleaning up...

And done. Please give it a run and let me know your thoughts! When we discuss and like the changes this will be merged and released on pip at version 0.0.73.

[bilke]

Thanks a lot it works! Here is my recipe: https://github.com/bilke/ogs.scif/tree/master/centos7

Your explanations make sense to me, I agree with everything!

[vsoch]

Great! Ready for merge then?

[vsoch]

Ohh I really like how you formatted your recipe :)

# ----------------------------------------------------------------------------------