August release

[bkpoon]

I still have some cleaning up for the August release as well as getting the configuration of modules with a conda installation working, so the commit for the release will be next week (probably Wednesday or Thursday). Then a week for input on which exact commit to use. The current release branch is at releases/v2020.08 and the release will be tagged by the date of the commit (e.g. v2020.08.19). Point releases, if any, will add another number (e.g. v2020.08.19.1). We can shorten it to vYY.M.D (e.g. v20.8.19 and v20.8.19.1) if that looks better.

The conda package recipe for conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) will go through the approval process after the official release.

[graeme-winter]

If releases are no more frequently than 1/mo surely v2020.08-1 say would make sense as the first 2020/08 version - allows the respin to be -2 which does not suggest a second release on the same day (otherwise the day of the release commit would make more sense? i.e. 2020.08.11 then a week later 2020.08.18 say but then it's not obviously a patch of the previous one)

Would strongly encourage MM not M to enable easier automation, as well as YYYY - "looks better" != good reason ;-)

[bkpoon]

I am not sure about having a "-" in the version. I would prefer one character between numbers.

How about having v2020.08 for the first release and v2020.08.1 for a point release? I suspect most users would use the official releases, so the exact date for the official release would not matter. Also, most branches will not have a point release since there will be monthly releases.

One additional wrinkle is that I was going to tag nightly builds with the date, so something like dev2020.08.29. The dev would differentiate between the official and nightly builds. Or v2020.08.a29 to really separate the last number.

I also created a new repository using submodules to track the different versions of each module. Since these other modules do not change often or are versioned differently (e.g. Boost or SCons), they do not need monthly branches/tags for releases. This would be used to construct the general source tarball for distribution and for conda-build. I added dxtbx, but that and other modules not needed for building cctbxlite will be deleted during the conda-build process.

@graeme-winter Do you want dxtbx in the source release tarball for cctbx? If so, which version do you want or should I just pick the latest commit? If not, I can just remove it.

[Anthchirp]

This is very confusing. Are we talking git branches/tags here, or version numbers?

dxtbx should not be vendored with cctbx.

[bkpoon]

@Anthchirp Those would be version numbers, which would be used for branching/tagging.

@nksauter @phyy-nx Do you want dxtbx in the cctbx source distribution?

[nksauter]

Yes, dxtbx in the cctbx source distribution.

Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713

On Fri, Aug 14, 2020 at 9:38 AM Billy K. Poon notifications@github.com wrote:

@Anthchirp https://github.com/Anthchirp Those would be version numbers, which would be used for branching/tagging.

@nksauter https://github.com/nksauter @phyy-nx https://github.com/phyy-nx Do you want dxtbx in the cctbx source distribution?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cctbx/cctbx_project/issues/515#issuecomment-674160210, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADQ24VWFN3H5UDQPL47ROOTSAVSAHANCNFSM4P2US7CA .

[bkpoon]

Tagging nightly builds seems too excessive, so let's stick with versioning releases.

Let's finalize the release by the end of next week? I'll be testing the builds on the conda-forge infrastructure as well.

[Anthchirp]

ah, by source distribution you don't mean conda releases. I hold no opinion on the former as we are unlikely to ever use them for anything. If you want to bundle dxtbx in releases I recommend using release versions of dxtbx (because that's what they're there for) rather than the latest commit.

Those would be version numbers, which would be used for branching/tagging.

Version numbers should follow PEP440, not start with a letter, and if you want to indicate development versions that can be done via a trailing .devN where N stands for one or more digits. Further I would strongly suggest to avoid leading zeros in version number components. We had this in iota at the start and you may encounter 'interesting' effects.

Branch and tag names of course do not have any of those restrictions.

[bkpoon]

Yes, the version number will be 2020.08 similar to the alpha packages in the cctbx-dev channel. I was typing an extra character. The v is for the branch name from your earlier comment from #451.

(On the release branch name I would suggest adding a non-[0-9a-f] prefix, for example v2020.04 as we recently noticed that pure numeric names can be confused for commit IDs in git.)

Is that an issue if the branch is prefixed with releases/?

I will also be merging some changes tomorrow for building software with cctbx installed as a conda package. Basically, something like

<activate conda environment>
conda install cctbx
mkdir modules
cd modules
git clone <another module or multiple modules>
cd ..
mkdir build
cd build
libtbx.configure <another module or multiple modules>
make (or libtbx.scons on Windows)

will work. Initially, this is just for conda, but I will be generalizing this for any $PREFIX directory with the proper dependencies installed.

[Anthchirp]

Yes, the version number will be 2020.08

I would recommend 2020.8 to avoid an issue where conda doesn't identify 2020.8 and 2020.08 to be the same version, when referenced as a dependency by downstream packages.

Is that an issue if the branch is prefixed with releases/?

Not a problem :+1:

I will be generalizing this for any $PREFIX directory with the proper dependencies installed.

Would it make sense to codify for our modules where the dependencies are defined? So that eg.

<activate conda environment>
conda install cctbx
mkdir modules
cd modules
git clone <another module or multiple modules>
conda install --file=<module1>/conda-requirements.txt [--file=<module2>/conda-requirements.txt [...]]
cd ..
mkdir build
cd build
libtbx.configure <another module or multiple modules>
make (or libtbx.scons on Windows)

[bkpoon]

I would recommend 2020.8 to avoid an issue where conda doesn't identify 2020.8 and 2020.08 to be the same version, when referenced as a dependency by downstream packages.

Sure, that works. The ca-certificates package does not seem to use leading zeroes. I'll change the release branch to follow releases/2020.8.

Would it make sense to codify for our modules where the dependencies are defined?

So once there is a cctbx conda package, dependencies can be installed into the active environment with

conda install --only-deps -c conda-forge cctbx

This is cross-platform, as opposed to the .yml files in libtbx/auto_build/conda_envs (The .yml files are mostly cross-platform, except for packages that are only installed on one platform). The .txt files in libtbx/auto_build/conda_envs are for users/developers that want an explicitly reproducible environment (version and build) with packages backed up on another channel and updated roughly every six months. The plan is to support both the --only-deps option and the.txt files. The .yml files will be removed once cctbx is in conda-forge.

You can do the same with the dials conda package. However, the --only-deps flag would also install the dxtbx conda package, so there will be an issue if you want to to configure both dials and dxtbx. In which case, you can use the .yml approach or your latest .txt files in dials/.conda_envs. Or you can also update your dials-dependencies package and run

conda install --only-deps -c conda-forge dials-dependencies

so that your environment only has the listed dependencies and not the dials-dependencies package. And with this approach, you could also get rid the the conda install cctbx step since cctbx would already be in dials-dependencies.

As for codifying it, I think it will be project-specific, but reducing the number of places that need to updated when dependencies are changed will simplify updates. For phenix, we are using the explicit .txt files that are updated for every release. I may update my custom package that is similar to dials-dependencies on the phenix-project channel.

[bkpoon]

I'm checking with some other groups before finalizing the commit for the release on Monday.

[bkpoon]

Source tarball for 2020.8 is released. The install command or bootstrap.py (more flexible) can be used for building from the sources. Building the conda package for conda-forge is the next step.

[Anthchirp]

Is there a difference on https://github.com/cctbx/cctbx_project/releases/tag/v2020.8 between the tagged source tar.gz and the tar.gz you uploaded?

[bkpoon]

The cctbx-2020.8.tar.gz tarball is the one you should use. It contains all of the modules. The source.tar.gz and source.zip are just the autogenerated copies of cctbx_project. The bootstrap.py in the autogenerated copies will not download the appropriate versions of the other modules for the release (right now it does for some modules), but you can get them with

git clone -b v2020.8 --recursive git@github.com:cctbx/cctbx.git

That repository uses git submodules for tracking the branches in different repositories.

[bkpoon]

Let's talk about the next release in #530.