Improvements to documentation on intermediate level

[redeboer]

It has been pointed out that the documentation for each package lacks explanation on an "intermediate" level. Examples would be "how do I insert my own Flatté shape into the amplitude builder?" or "Why does TensorWaves JIT-compile functions?". This issue is to collect feedback and ideas for this "intermediate" level documentation.

The main documentation pages

• qrules.rtfd.io
• ampform.rtfd.io
• tensorwaves.rtfd.io
• compwa-org.rtfd.io

[redeboer]

@wgradl @sebastianJaeger could you give an example of what you would like to see? Then an attempt can be made to improve things and see if the existing infrastructure is sufficient to tackle such shortcomings.

[wgradl]

One issue that is probably easy to resolve: I usually want to have all packages (qrules, ampform, and tensorwaves) installed in an editable fashion, so my workflow ideally would look like this:

conda create -n tw-new
conda activate tw-new
git clone ...  # all three packages

but then I'm sort of lost how to proceed properly. I've resorted to using something like

pip install -c .constraints/py3.8.txt -e .[dev]

for all three repositories, but I'm not expert enough in all things python to be confident that this is how it should be.

I also note that in many cases the three packages seem to require slightly different versions of other packages, so that running one of these pip commands after the other may even downgrade installed packages (I'll have to do this again to find a concrete example of this, but I'm pretty sure I've seen this behaviour recently).

• How do I know that this operation leaves me with a workable installation?
• Obviously the final set of packages depends on the order in which I install the requirements of each package. This seems not to be desirable.

Releasing a set of tags that use consistent requirements would be of great help here.

[redeboer]

Hmm, dependencies are indeed a complicated issue in general. First note that there are two things going on here:

1. (ComPWA) packages list which packages they require in their setup.cfg, see e.g. here. Version ranges listed here should be as wide as possible as to avoid conflicts when installing the package with other packages. (Wide as possible is roughly: only set e.g. a minimal version if there is a certain breaking change or required feature that was introduced in a certain version.) This allows you to do something like:

   pip install ampform==0.13.* tensorwaves[jax] tensorflow~=2.5

2. To ensure reproducability for each repository itself, we regularly find an exact set of pinned versions that does not result in version conflicts and guarantees that all CI checks keep working when that set of versions is installed. Those are the .constraints/py3.8.txt files. It can indeed be the case though that e.g. TensorWaves constraints some dependencies to a lower version than AmpForm (after all, TensorWaves works with a superset of dependencies some of which may set upper limits). So those .constraint files are not necessarily usable when combining packages in an environment.

• How do I know that this operation leaves me with a workable installation?

So, strictly speaking, if you are writing your own analysis/application with a different combination of packages, you define your own set of dependencies with optional ranges for which your application should work, like in that setup.cfg file (there are other ways to list Python dependencies). If in addition you want to guarantee reproducability (workability?), you can use pip-tools or this GitHub Action to exactly constrain the versions.

• Obviously the final set of packages depends on the order in which I install the requirements of each package. This seems not to be desirable.

Pinning/constraining (or even defining your own docker image) is the only way in which you can avoid this. It's a pain, but afaics it's the only way out of dependency hell.

[redeboer]

I usually want to have all packages (qrules, ampform, and tensorwaves) installed in an editable fashion, so my workflow ideally would look like this:

conda create -n tw-new
conda activate tw-new
git clone ...  # all three packages

As a side note, Conda has its own dependency resolver and works a bit differently. I personally find it a bit obscure (and slow) and therefore prefer to work with pip withing a Conda environment, which also makes it possible to work with those constraint files. In that case, it would be something more like:

conda create -n tw-new "python==3.8.*" pip
conda activate tw-new
git clone ...  # all three packages
pip install -e qrules  # <-- editable install
pip install -e ampform
pip install -e tensorwaves[jax]  # <-- syntax for optional dependencies
# ideally done in one line, so that pip can figure out dependency conflicts

The editable installations here make it impossible to exactly reproduce the environments though, unless you somehow provide instructions as to which Git commits/tags were used.

[redeboer]

Just posting a note here that may be worth investigating: https://documentation.divio.com

[wgradl]

Thank you for the tip regarding conda vs pip ! That works perfectly (except that the current HEAD of ampform requires qrules==0.9. and >=0.9.6, while the current HEAD of qrules provides 0.10.0, but that is to be expected when just checking out the HEADs of three packages, and can be trivially resolved).

Maybe this would be a helpful addition to a generic INSTALL page.

I am not so concerned with reproducing any previous environment, as long as I can then use pip freeze (or similar) to collect the exact versions of the current install. Running this in a container would also be an option, of course.

[redeboer]

That works perfectly (except that the current HEAD of ampform requires qrules==0.9. and >=0.9.6, while the current HEAD of qrules provides 0.10.0, but that is to be expected when just checking out the HEADs of three packages, and can be trivially resolved).

Indeed, this is an example where ampform expects the old qrules interface, not the one on qrules' main branch (upcoming release 0.10). It could be that ampform still runs with qrules 0.10 for most applications though, because most parts of the new interface are not touched.

Maybe this would be a helpful addition to a generic INSTALL page.

Yes, although we were careful to not write too many general tips about software development. Where possible, we refer to sources that provide up-to-date tutorials/tips. Those are collected here: compwa-org.rtfd.io/develop.html, not on the generic install pages for each package.

For this example, I guess a section could be written on how to work with editable installs of multiple repositories (I have my reservations about such a setup though).