Our scons build system allows us to develop new "packages" that become part of the psana-conda package. I say "packages" in quotes because they are not conda or python packages - they are packages as defined by the SConsTools build system we use for psana. By developing a new SConsTools package, we make psana-conda bigger.
Moving forward, we would like to develop new packages as standard python or conda packages when it makes sense. For example, if someone is developing a pure python package that interfaces with psana just through a import psana, I think it wold be better to develop it as a standard python package -- i.e, write a setup.py. @slacmshankar has done this with logbookclient, and now @tjlane has pscache.
We'll go through how to add such package like pscache into the ana environments. All of this will be done as the admin account psreldev.
Create Recipe
as psreldev
cd /reg/g/psdm/sw/conda/manage/recipes/external
cp -r logbookclient pscache
now edit pscache/meta.yaml. For this checkin, there is a tag called v0.1, but the package version, per the setup.py, is 0.0.1 - take care to use the v0.1 with the git_tag and the 0.0.1 for the version.
Also make the build and run dependency psana-conda
for the build script, we are copying what Murali figured out, i.e
the main point is you want to do a standard python install, but not have pip or setuptools install dependencies - you list them in the build/run sections of your conda meta.yaml and let conda mange them.
Build Recipe
execute the command, from a machine with internet access,
you could just do conda-build, but ana-rel-admin will also put the output in the rhelx channel and update the channel index.
Repeat that on a rhel5 and rhel6 machine (psdev106 and psdev105) this is not ideal since pure python will be the same, but for now we are just replicating all packages 3 times.
now we will add pscache to the conda environments we build.
Since pscache depends on psana, and psana is python 2.7 only, we'll follow the logbookclient pattern and specify that we skip it in the py3 environment.
update the package build order
As the number of packages we maintain grows, we need to automate building them all with one command. We also need to build them in the correct order. For example, if we update openmpi, then we will probably update mpi4py. We'd like to edit the two recipes, and then do one command to build openmpi and then mpi4py.
lists all the packages we need to build, the path to their recipe directories (relative to mange) in the order they should be built. Add an entry for pscache there (it won't matter where since pscache only depends on psana, the order is only for things like openmpi -> mpi4py, etc)
You can run
ana-rel-admin --cmd bld-all
it will go through the list and build anything that needs to be built.
Build new ana environments
We should be able to build a new ana environment now, there are separate instructions for that on confluence, or get in touch with admin
Developer Responsibility
maintain your repo pscache, update the git_tag in the conda recipe
Packages - conda, pip, SConsTools
Our scons build system allows us to develop new "packages" that become part of the psana-conda package. I say "packages" in quotes because they are not conda or python packages - they are packages as defined by the SConsTools build system we use for psana. By developing a new SConsTools package, we make psana-conda bigger.
Moving forward, we would like to develop new packages as standard python or conda packages when it makes sense. For example, if someone is developing a pure python package that interfaces with psana just through a
import psana, I think it wold be better to develop it as a standard python package -- i.e, write a setup.py. @slacmshankar has done this with logbookclient, and now @tjlane has pscache.We'll go through how to add such package like pscache into the ana environments. All of this will be done as the admin account psreldev.
Create Recipe
as psreldev
now edit pscache/meta.yaml. For this checkin, there is a tag called v0.1, but the package version, per the setup.py, is 0.0.1 - take care to use the v0.1 with the git_tag and the 0.0.1 for the version.
Also make the build and run dependency psana-conda
for the build script, we are copying what Murali figured out, i.e
you'll also see people do things like
the main point is you want to do a standard python install, but not have pip or setuptools install dependencies - you list them in the build/run sections of your conda meta.yaml and let conda mange them.
Build Recipe
execute the command, from a machine with internet access,
you could just do conda-build, but ana-rel-admin will also put the output in the rhelx channel and update the channel index.
Repeat that on a rhel5 and rhel6 machine (psdev106 and psdev105) this is not ideal since pure python will be the same, but for now we are just replicating all packages 3 times.
Update anarelyaml
I.e, an entry like
https://github.com/slaclab/anarel-manage/blob/master/config/anarel.yaml#L321
now we will add pscache to the conda environments we build.
Since pscache depends on psana, and psana is python 2.7 only, we'll follow the logbookclient pattern and specify that we skip it in the py3 environment.
update the package build order
As the number of packages we maintain grows, we need to automate building them all with one command. We also need to build them in the correct order. For example, if we update openmpi, then we will probably update mpi4py. We'd like to edit the two recipes, and then do one command to build openmpi and then mpi4py.
This config file
https://github.com/slaclab/anarel-manage/blob/master/config/pkg_build_order.yaml#L66
lists all the packages we need to build, the path to their recipe directories (relative to mange) in the order they should be built. Add an entry for pscache there (it won't matter where since pscache only depends on psana, the order is only for things like openmpi -> mpi4py, etc)
You can run
it will go through the list and build anything that needs to be built.
Build new ana environments
We should be able to build a new ana environment now, there are separate instructions for that on confluence, or get in touch with admin
Developer Responsibility
maintain your repo pscache, update the git_tag in the conda recipe