Release instructions for affiliated packages

[astrofrog]

Since the astropy release mechanism doesn't work for affiliated packages at this point, I have written up some release instructions for WCSAxes:

https://github.com/astrofrog/wcsaxes/blob/master/RELEASE.md

but they are applicable to most affiliated packages. Should we add this file in the package-template repo? Or should we add a page to the main astropy docs along with the other instructions related to affiliated packages?

cc @embray

[cdeil]

@astrofrog Thanks for writing this up!!!

I commented at https://github.com/astropy/astropy-helpers/issues/56#issuecomment-52458429 before I saw this, but I think some instructions on how to set the astropy-helpers sub-module to the latest stable version would fit here (and maybe a standard place to indicate in the release notes which astropy-helpers version is bundled would be useful, in case installation issues occur for some users after release).

I'm :+1: to add this to https://github.com/astropy/astropy/blob/master/docs/development/affiliated-packages.rst

[cdeil]

One thing that's currently missing in the instructions is how to handle the PyPI login info. This was discussed a bit here: https://github.com/astropy/package-template/issues/17#issuecomment-16294852

It's probably sufficient to link to the official Python packaging docs (which are a quick read and pretty good IMO) for explanations and only put the commands that should be used here: https://packaging.python.org/en/latest/tutorial.html#uploading-your-project-to-pypi

[astrofrog]

@cdeil - when you register the first time it will ask for the username and password interactively, so I'm not sure if this requires a special step? Although maybe I need to mention that you need to create a PyPI account before running the register command?

[cdeil]

The recommended procedure is to use the web form on the PyPI website and then to run twine upload ... if this sounds acceptable to you for Astropy affiliated packages you could simply add a link to the instructions here: https://packaging.python.org/en/latest/tutorial.html#uploading-your-project-to-pypi

[astrofrog]

@cdeil - what is the problem with step 11 in the instructions I have given? This will do everything, register, make the source distribution, and upload - is that not recommended?

[astrofrog]

Ah I see, security concerns. Maybe we can leave it as is in the notes I have but add something that says 'if you are concerned about security, then follow these instructions instead? (and link to them).

[cdeil]

Yes, security concerns.

Also I think the instructions should say that when starting the affiliated package one should check that the name is not taken on PyPI and "reserve" it via the web interface. Then come 0.1 release time (which could be a year later) do twine upload.

[astrofrog]

The secure way seems pretty obnoxious, it recommends not even using register and filling out the information by hand. I'll try and update the instructions as you suggest and will mention both the secure and the easy way.

[astrofrog]

(obviously, setuptools needs to be fixed to communicate all this over SSL!)

[cdeil]

I don't think filling out a web form on PyPI and config file is a problem ... this will take a few minutes max, which I think will be a small fraction of the time to make and check a release. But sure, I'm not too concerned with security myself, and for this part there's nothing astropy-specific so just linking to the Python packaging docs that describe the two options should be fine.

[cdeil]

Can you be more specific with the version numbers? Should the release be made as X.Y and then only bugfix releases as X.Y.Z or should we always use the X.Y.Z format?

As an example, should the first release be 0.1 followed by a bugfix release 0.1.1? Or should it be 0.1.0 followed by 0.1.1 and the 0.1 tag would always be moved to the latest bugfix release tag?

[astrofrog]

@cdeil - good point, this needs to be clarified. In astropy core major releases are X.Y and bugfix releases are X.Y.Z. We never move tags, so the v0.1 tag points to the first v0.1 release, v0.1.1 points to v0.1.1 and so on. So we treat v0.1 as v0.1.0 but we just don't write the .0 for the first release in a series. I honestly don't think it matters either way, but that's what we've been doing for the core package so it's as good as any convention. I'll try and incorporate this in the notes.

[cdeil]

@astrofrog It seems problematic to me that users going to the 0.4 version would get the original version from the 0.4.x series and not the latest version in the 0.4.x series which includes bugfixes: https://pypi.python.org/pypi/astropy/0.4 https://astropy.readthedocs.org/en/v0.4/index.html

What do other projects like numpy, scipy, ... do?

Also APE 2, while not being very explicit, seems to suggest the original release in the 1.x series should be 1.0.0 and not 1.0 (without specifying whether the 1.0 is moved as bugfix releases are made.

Maybe clarify this in the APE and then you could link to that explanation for version numbers in the release notes you are writing here?

[astrofrog]

Maybe @embray and @eteq can comment here? (I have no strong feelings about this!)

[cdeil]

Just FYI ... I'm asking because I want to release Gammapy 0.1 (or 0.1.0?) later this week and I want to do it right and consistent from the start with the tags and on PyPI and readthedocs.

[keflavich]

Some more information from my attempts to release astroquery: the MANIFEST.in should specify exclusions (e.g., *.pyc) after the inclusion of astropy-helpers etc.

[astrofrog]

I guess the main thing is to ensure that affiliated packages are up to date with the template (https://github.com/astropy/package-template/blob/master/MANIFEST.in contains the *.pyc exclusion). Now that we have a 'tagged' version of the package template we should more explicitly describe what changes between tagged versions.

[keflavich]

I think that MANIFEST.in is wrong - does it work for you? I had essentially the same one - maybe exactly the same - and the .pyc files from astropy-helpers were being included. I think line 12 needs to be moved to the end. If you agree, I can make that PR

[astrofrog]

@keflavich - oh, you're right! The template needs fixing indeed. Can you do that?

[cdeil]

@astrofrog I followed these instructions and made the Gammapy 0.1 release: https://pypi.python.org/pypi/gammapy/0.1

Now I'm left in this state:

$ git status
HEAD detached at v0.1
nothing to commit, working directory clean

and the v0.1 tag is not pushed yet to Github or registered on readthedocs. Can you add steps to do that to the instructions?

[cdeil]

OK, pushing the tag is easy:

git push origin --tags

For readthedocs I don't know how to do it ... these instructions say I should be able to select the versions from the admin panel, but all I see are "master" (not built) and "latest" (built) ... no idea what "master" is and no "v0.1" that I can select!?

And one more thing for the release instructions: one should create a v0.1.x branch and push it to Github, no?

[keflavich]

For readthedocs, under the admin tab, there is a "versions" sub-tab on the left side. That's where you can activate branches or tags.

[cdeil]

@keflavich - Thanks ... the new tag "v0.1" only appeared in the "versions" sub-tab of the admin settings within the last hour ... maybe readthedocs has a daily cron-job to check for new tags and branches? If that's the case it's pretty annoying because one can only finish the release steps a day later. Should we file an issue "Add synchronize with Github button to versions setting"?

[eteq]

@astrofrog @cdeil - In the comment above to me you were asking about 0.1.0 vs. 0.1, right? In astropy we've been using e.g. 0.4, with the final ".0" implicit. That seems to work fine with all the version number tools. So 0.1, which it looks like @cdeil chose, is indeed the most consistent one. It doesn't really matter that much, though.

[eteq]

@cdeil re: RTD: The way we've been getting around this in astropy is the use of the "stable" branch - RTD will update that immediately if you point it to a commit and re-build.

How long did it take for the new version to appear, @cdeil? My experience in astropy has been it's usually not too long (like < hours). So I had thought it was triggered somehow. But I admit I haven't watched too closely, because of the "stable" branch solution.

[astrofrog]

@cdeil - if you trigger a build, it will fetch new tags at the same time. I'll try and update the instructions with all this here.

[astrofrog]

@cdeil @eteq - see https://github.com/astropy/astropy/pull/2912