Psteinberg/help submit

[PeterDSteinberg]

Adds a new CLI item for help with submission of packages from the jsonified package tree created in python build2.py ./ split -s package_tree.js. The name of the output of the split file must be package_tree.js so that anaconda build includes it among the dependency files (it check git ls-files and this PR adds it to the ls-files output).

Using it:

• This is how I test it on a local OSX worker that is listening to psteinberg/abc queue

python build2.py ./ submit -json-file-key package_tree.js  libnetcdf   -user psteinberg -queue abc -platforms osx-64

It says:

• open the package_tree.js file and build the packages in the libnetcdf list as well as libnetdf
• create the package under psteinberg username (can be your personal or conda-team)
• build on the -queue named, here abc, but 'build_recipes' by default
• build on the osx-64 platforms, other choices are linux-64, win-64

To submit all packages named in package_tree.js, do:

python build2.py ./ submit -full-json package_tree.js -user psteinberg -queue abc

Either command accepts a -dry argument for dry run.

Example command and output:

$ python build2.py ./ submit -json-file-key package_tree.js  libnetcdf   -user psteinberg -queue abc -platforms osx-64
Running build2.py with args of Namespace(dry=False, full_json=None, json_file_key=['package_tree.js', 'libnetcdf'], path='./', platforms=['osx-64'], queue='abc', user='psteinberg')

-------------------------------
Check to see if psteinberg/protoci-libnetcdf exists: ['anaconda', 'build', 'list-all', 'psteinberg/protoci-libnetcdf']
Using anaconda-server api site http://api.anaconda.org
Getting builds:
        Build # |          Status |        Platform |          Engine |             Env
--------------- + --------------- + --------------- + --------------- + ---------------
            3.0 |         running |          osx-64 |          python |            None
            2.0 |         failure |          osx-64 |          python |            None
            1.0 |           error |          osx-64 |          python |            None

prepare to submit ['anaconda', 'build', 'submit', './', '--queue', 'psteinberg/abc']
TAIL:        binstar-build tail -f psteinberg/protoci-libnetcdf 4

[PeterDSteinberg]

The final commit above prepares build2.py and binstar_template.yml for helping with testing of the entire conda-recipes dir. Doing so is a two step process.

First, it is necessary to split the packages in conda-recipes into independent sub-trees. For me, with conda-recipes cloned in the parent dir of ProtoCI clone dir, I do this to split into sub-trees:

python build2.py ../conda-recipes split -t 12 -s ../conda-recipes/package_tree.js

(splitting into package groups of 12 and saving a json at package_tree.js in there)

Then I can submit the h5py sub-tree like this:

python build2.py ../conda-recipes submit -json-file-key ../conda-recipes/package_tree.js h5py -user psteinberg -queue abc -platforms osx-64

That statement submits the h5py sub-tree to the psteinberg/abc queue that I have running on my laptop.

The latest commit adds code to build2.py's submit action as follows:

• Copy build2.py to the target dir (conda-recipes)
• Create a git branch so that build2.py and the package_tree.js can be committed to conda-recipes. Doing so ensures that build2.py and the package_tree.js are included in the submitted package files. If they aren't in git, they might be skipped if submitting builds outside of the ProtoCI dir.

[msarahan]

I'm a little confused about the workflow here. Does package_tree.js contain all subtree ID's, so that one can programmatically submit them one-by-one?

Is there any way to condense this into a single step, where the split is implicit (and handled according to perhaps some build worker config that can be queried from the queue?) Creating a git branch to ensure presence of files seems like a complex intermediary, and something that could be cleaned up by keeping things in one process.

[PeterDSteinberg]

@msarahan package_tree.js is a json dictionary where the keys are high level packages and the values are lists of packages that should be installed in order from the beginning of the list to the end of the list, followed by the high level package that is the keyname. Regardless of whether we do the split/submit in one step or two, build2.py and package_tree.js need to be in the conda-recipes dir, in the git ls-files, and in the same form as we like them in ProtoCI or else anaconda build submit ./ --queue psteinberg/abc will not find the files or will upload an out of date version. In that case, the remote build starts, but immediately errors out because one or more of these things will happen:

• build2.py is not uploaded and therefore cannot build the subtree with a call like python build2.py ./ build ... remotely
• build2.py is out-of-date relative to recent edits
• package_tree.js absent or does not reflect the package trees we are looking for (this could result in a KeyError on python build2.py ./ build ...

[msarahan]

All right, you have thought this out well. I'm +1 on this - under the assumption that humans are not going to be typing in the keys for the high-level packages, rather that those are controlled by some script that is driving the build2.py script.