A better way to handle conda-supplied dependencies

[mjuric]

Generate individual metapkgs w. EUPS files for pkgs provided by conda.

Instead of having all table files (and other ups/* data) packed in a single package, generate a conda package with the relevant EUPS file for each internal package. They will be named lsst-PACKAGE-eups-configs and contain the contents of the ups/ directory (and be declared as an EUPS product).

Their generation differs from generating other packages only in that they use a special build.sh template (build-internal.sh.template), which copies the ups/* directory instead of doing a full build. This simplifies the internal code logic.

This obsoletes the lsst-product-configs recipe (which we've had to maintain by hand until now). Just adding the product name to internal_products in etc/config.yaml now does everything.

[mjuric]

(@jhoblitt || @jmatt) : Take a peek & review when you get a chance.

[jmatt]

I definitely agree it needs to be improved. Ideally configuration based without changes to multiple repos.

I began looking through this but there is a lot to work through. I couldn't figure out the exact purpose of all the different collections in conda_lsst.config.py. Specifically exactly how internal_products, dependencies and missing_deps work together. I mean how they actually work - not what's documented. I generally understand that.

[mjuric]

I definitely agree it needs to be improved. Ideally configuration based without changes to multiple repos.

Yup, that was the goal here.

... Specifically exactly how internal_products, dependencies and missing_deps work together. ...

Yes, it's a bit convoluted. Roughly:

• config.internal_products is meant to be a dictionary where the key is the EUPS product name, and the value is a dict with two keys: 'build' and 'run'. These hold the package specs that will get written into the eponymous sections of the requirements: clause in meta.yaml. The goal is to find the relevant conda dependency line to write into meta.yaml, for an EUPS package that we don't want to rebuild as conda provides it.
• config.missing_deps is a dictionary analogous to config.internal_products in terms of structure, but different in terms of usage: the keys are considered to be globs (so multiple keys can match the same product). When the code is looking for a list of dependencies it needs to add to meta.yaml so that the package will build, it merges the entries for all keys that match the product names. That allows you to write stanzas that apply to all products which will be merged with product-specific stanzas as well (e.g. like the one for mysqlproxy). N.b.: many of the dependencies entries in config.yaml are actually workarounds for bugs in .table files: e.g., everywhere where we have an astropy entry, that entry should really be in the table file of that package. For others that don't have EUPS stubs (e.g., cython), we'll be able to remove them once EUPS tables begin storing information about dependencies on system packages (which @RobertLuptonTheGood is working on). Btw., once that's in place, we should be able to get rid of many of the stub EUPS packages (like astropy or numpy or python) all together.
• The final bit you'll want to know about is the pin_versions entry. This allows you to pin specific build/run versions for packages referred to in either confg.missing_deps or config.internal_products. The relevant code is in the Config.__init__ method: it basically just traverses all leif nodes of missing_deps and internal_products and anywhere where there's no mention of a specific version (and there's an entry in pin_version) it adds the pinned version.

Hope that helps, for a start!

[RobertLuptonTheGood]

I wrote the eups code on the plane, it's about to be reviewed by Mario.