nightly builds

[davidslac]

We need to automate nightly builds. Here are manual steps

source conda_setup

this makes the management code available, and a central conda install

Directory Structure

We'll do this in our own account for development. So make some directories like

mkdir conda-root
cd conda-root
mkdir downloads
cd downloads
mkdir anarel

cd back to conda-root do

pwd

to see what it is. Say it is /reg/neh/home/davidsch/projects/conda-root

Latest Tags

When someone checks in a new tag, we want to first want to check out all the psana source code.

In production, as psreldev, we'll do

ana-rel-admin --force --cmd psana-conda-src --name 9.9.9  

Where we're just picking some crazy version like 9.9.9, and we'll plan on overwritting our previous work, so we use --force.

For development, we can do this

ana-rel-admin --cmd psana-conda-src --name 9.9.9 --basedir /reg/neh/home/davidsch/projects/conda-root --tagsfile /reg/g/psdm/sw/conda/manage/config/psana-conda-svn-pkgs-tst 

You may need to do kinit beforehand. And github has to work.

Psana Recipe

we need to make a recipe to build a psana 9.9.9

cp -r /reg/g/psdm/sw/conda/manage/recipes/psana/psana-conda-opt .

edit the version line to use your 9.9.9

and edit the source file line

for me it is

source:
  fn: /reg/neh/home/davidsch/conda-root/downloads/anarel/{{ pkg }}-{{ version }}.tar.gz

Build Psana

Now do

conda-build --no-locking psana-conda-opt

We want to know that this works on rhel5, rhel6, rhel7

[davidslac]

For development, to make it go faster, try editing the meta.yaml to reduce the buid/run dependencies

[davidslac]

There is this issue, is someone other than psreldev able to run conda-build? At first no, and I made this issue

https://github.com/conda/conda-build/issues/1883

but lately, I am trying it, and it is working?

If it doesn't work, Jacob should install his own miniconda in his home directory, and then build the latest psana-conda on rhel5,6, 7, that means ssh-ing to psdev106 (rhel5) psdev105 (rhel6) and psel701 (rhel7) - note, these rhel 5/6/7 machines are part of the psana filesystem.

You have to be careful about running all three in parallel - you would want to modify the recipe further to have build strings for the psana_conda packages that are unique for the platform.

[davidslac]

We now have a later version of conda-build installed, you should be able to use the --no-locking without a bug.

[davidslac]

Nightly Fails - how to Debug

If the nightly fails, we need instructions for how to debug it.

What we want to do, is to reproduce exactly what conda-build did.

conda-build - high level

At a high level, conda-build does the following

• creates a top level directory
• creates a conda environment for the build in a subdirectory to this top level dir
• creates a working directory where it unpacks the source and runs the recipe build instructions

The details of where to find these, what the names are, this can change from one version of conda to another. We are presently using conda-build 2.10 but conda-build 3.0 is in alpha.

figure out conda-build

To figure out how to debug, I did the following

• build szip
• looked at the conda-build log
• poked around in the directories it built

Right now, the nightly is running conda-build from a user account, like jscott or davidsch. If I do the following

conda-build /reg/g/psdm/sw/conda/manage/recipes/external/szip 

it starts out with a line like

source activate ~davidsch/conda-build/szip_1493307653294/_b_env_placeholder

from this you can see the top level directory is

~davidsch/conda-build/szip__1493307653294

and that the conda environment for building is the long _b_env_placeholder name (it is long to create as much space as possible for the RPATH which conda-build will overwrite in executables and shared object libraries at the end of the build).

You'll also see more stuff, like

+ echo '######## env ## and compilers ######'
######## env ## and compilers ######
+ env
DIRTY=
CPU_COUNT=8

this is getting printed out because in the build.sh for szip:

https://github.com/slaclab/anarel-manage/blob/master/recipes/external/szip/build.sh

I decided to print out the environment present during the conda-build process.

You can add things to the recipe if you need more information, for instance you might want to print the current working directory, or print some parsable marker for automation.

From poking around in the ~davidsch/conda-build/szip_1493307653294 directory, I see a path

work/szip-2.1

If I go in there, I see the szip source that has been unpacked - I also see the file

./conda_build.sh

If I look at that, I see my original build.sh for szip:

https://github.com/slaclab/anarel-manage/blob/master/recipes/external/szip/build.sh

preceded by stuff that conda-build added - namely doing

source activate that _b_env_placeholder environment

Fail Debug Instructions - First Order

So, to first order, if a nightly fails, instructions for debugging will be:

• cd ~davidsch/conda-build/szip_1493307627479/work/szip-2.1
• edit modify
• now run: ./conda-build.sh

note, only jscott will have write permission, we may want to change the permissions on the top level ~jscott/conda-build so that we can all debug conda-build work?

Fail Debug - Second Order

PREFIX environment variable

• your environment may not be exactly the same as what conda-build had. Before running your build.sh via conda-build.sh, conda-build sets a lot of environment variables which build scripts can rely on, they are documented here: https://conda.io/docs/building/environment-vars.html which is one reason I print out the environment in the build scripts.
• The only environment variable build scripts usually need to use is PREFIX. For instance, the szip does

  ./configure --prefix=$PREFIX --enable-shared

  If I follow the first order instructions, at the end my build fails with

  /usr/bin/install -c .libs/libsz.so.2.0.0 /lib/libsz.so.2.0.0
  /usr/bin/install: cannot create regular file '/lib/libsz.so.2.0.0': Permission denied

  because PREFIX is not defined, the build script is trying to install into the system library.

This could be a good thing, you may be in a development cycle where you have to continually re-run the build to debug, and you may not want to install szip in the _b_env_placeholder environment that you are running from.

Debugging conda install

On the other hand, you may have a custom installation into _b_env_placeholder that you need to debug, and you will want to install in the _b_env_placeholder conda environment. That last line of the psana-conda build.sh:

https://github.com/slaclab/anarel-manage/blob/master/recipes/psana/psana-conda-opt/build.sh#L81

executes this custom install. I wrote the build.sh to set PREFIX from CONDA_PREFIX if it not defined:

https://github.com/slaclab/anarel-manage/blob/master/recipes/psana/psana-conda-opt/build.sh#L24

this lets me re-run the script outside conda-build - however once it runs, it modifes the conda-environment, potentially corrupting it depending on what you are debugging.

psana-conda notes

For psana-conda, most of the time we will need to re-run that conda-build.sh, so I'd first comment out the last line that does the conda-install.

You may also want to edit conda-build.sh just to set the environment variables you need, but not do scons, then source your modified conda-build.sh and run scons or scons test yourself.

You may need to activate the long _b_env_placeholder environment and poke around. After activating it, your prompt is horribly long, it may be useful to reset PS1 to something reasonable like

PS1="\[\h: \w $ \]"

conda-build test phase

Recipes that include a test section in the recipe, I'm not sure what happens if you get through the build phase but fail in the test - conda-build might remove that _b_env and create a _t_env for testing. We may want to use special options when running conda-build for the nightly, like --dirty?