release plan

[tacaswell]

This is my plan for ~the next day to get a py38 compatible version of datrie up on pypi

• [x] decide on version number (inclined to go with 0.8.2 because there is a 0.8.1 tag from @sarnold that includes a bunch of changes, including dropping py2 support that I do not think should be done in a patch release and the ambiguity of having two 0.8.1 tags / packages around seems too great)
• [x] get one of the #80 or #78 merged. I am partial to my PR and am willing to self-merge it, but would like some feedback from @ashwinvis @sarnold and @frankier
• [x] either sort out how to make pyproject.toml to work reliably or document that cython is now a build time dependancy
• [x] sort out how to turn the crank on building manylinux wheels (I suspect following https://github.com/pypa/python-manylinux-demo will be enough). I am not going to worry about making it run on CI today.
• [x] find a volunteer to build the mac wheels
• [x] email Christoph Gohlke about making sure the cython change will not break his build machinery
• [ ] document release process
• [x] update release notes done in #80
• [x] manually test that the generated sdist works
• [x] manually test that the linux wheels work

[ashwinvis]

You can also take a look at https://github.com/pyFFTW/pyFFTW-wheels as reference.

[tacaswell]

ah, they are also using multibuild, that is how we build the Matplotlib and h5py wheels as well.

@ashwinvis you want to take a crack at setting that up? I think we would have to get @matthewbrett to set up the RSA key for us? I have the power to create new repos in this org, but would want to check with @kmike before doing that.

[kmike]

FTR, I like how the builds are organized at https://github.com/scrapinghub/python-crfsuite (thanks to the work of @datamade folks). It is all in the same repo, and wheels for OS X / linux / Windows are built on git tags. Setting up Windows builds required additional AppVeyor setup; OS X and Linux were covered by Travis CI.

[kmike]

Feel free to create an additional repo though, if you decide that's the best way forward.

[tacaswell]

@kmike That does look nice! I'm inclined to do this "by hand" today and then adopt an automated solution like this going forward. Do you approve the release plan otherwise?

Also, in small world, I crossed paths with some of the datamade people through the chi hack night!

[kmike]

Do you approve the release plan otherwise?

Yes.

I prefer not to require cython at install time: it is an additional dependency; cython upgrade could break the package, as tests are run at release time, while package can be installed with a different Cython version - older or newer. But I can understand reasons for having it like that, at least it is simpler.

Best of all world seems to be generating C files on CI for the release, but without good automation of releases it may get us into a situation similar to this Python 3.8 problem, when you need to regenerate files to make datrie work with more recent Python.

[tacaswell]

I would argue that if the behavior of datrie changes based on the cython version that is a cython bug and treat that the same as if a patch version of python introduced a regression, but I see your point.

If you include the c-files, but use cython if it is there you are in the same boat as requiring cython and build time in terms of bugs and are still exposed to the issue with the cpython internal changing if you don't have cython. By not including the c-files you only expose your self to 1 failure mode instead of 2 ;). Given that cpython is moving to a 12mo cadence I am personally more worried about cpython moving under us that (hopefully transient).

For reference, h5py has been requiring cython as a build time dependency from at least 2015 (in part because we will build against your local hdf5) and have not to my knowledge had issues with cython breaking under us.

One advantage of hosting the wheel machinery outside is that you don't have to version-rev to get wheels for new versions of Python when they come out.

[kmike]

Fair enough :)

For reference, h5py has been requiring cython as a build time dependency from at least 2015 (in part because we will build against your local hdf5) and have not to my knowledge had issues with cython breaking under us.

Cython 3.0 is going to change default language version from Python 2 to 3; this changes semantics of .pyx code, and can break it. I'm not sure datrie is currently ready for that. It'd be good to set language_level explicitly.

[tacaswell]

ah, good point re the language level. I'll add that to #80 .

[tacaswell]

Update: I have located an osx machine I can use (but have not started to try to use it yet) and am currently playing with the manylinux1 docker (~everything but the py38 one is working~ :sheep: the py38 build was finding local build products rather than the installed version from the wheel).

Have also heard back from CGohlke and this change should not break any of his tooling.

[tacaswell]

Assuming you have conda installed the following will build the mac wheels (basically copying the manylinux demo script). I tried to use python.org python, but could not figure out how to install the pkg file to a local directory, it was not clear to me that it would be happy with installing all of them and I did not want to mess up my wife's laptop which I was using for this (I actually ran this against my fork and branch to test)

#!/bin/bash
set -e -x
TAG=0.8.2
mkdir snakes
conda create -y -p snakes/py27 python=2.7
conda create -y -p snakes/py35 python=3.5
conda create -y -p snakes/py36 python=3.6
conda create -y -p snakes/py37 python=3.7
conda create -y -p snakes/py38 python=3.8
pushd snakes/
git clone https://github.com/pytries/datrie
pushd datrie/
git checkout $TAG
git submodule init
git submodule update
popd

## this is a bit questionable (pip installing into the root conda env!)
pip install delocate

# Compile wheels
for PYBIN in ./py*/bin; do
    "${PYBIN}/pip" install -r datrie/dev-requirements.txt
    "${PYBIN}/pip" wheel datrie/ -w wheelhouse/
done

delocate-wheel wheelhouse/*whl 

# Install packages and test
for PYBIN in ./py*/bin/; do
    "${PYBIN}/pip" install datrie --no-index -f wheelhouse
    "${PYBIN}/python" -m pytest datrie
done
popd

[tacaswell]

docker run --rm -e PLAT=manylinux1_i686 -v `pwd`:/io quay.io/pypa/manylinux1_i686 linux32 /io/travis/build-wheels.sh
docker run --rm -e PLAT=manylinux1_x86_64 -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/travis/build-wheels.sh

will build all of the linux wheels.

[tacaswell]

I noticed one oversight after I merged #80 that we were not excluding the cython build product from the Manifest.in. I added that to directly on master for expediency.

[tacaswell]

I'm going to close this issue as 0.8.2 with wheels for mac and linux have been pushed to pypi :tada:

I will upload the windows wheels when they show up at https://www.lfd.uci.edu/~gohlke/pythonlibs/#datrie

Normally I don't like to upload to pypi until we have all of the wheels built, but as there are not currently windows wheels this does not make things any worse that it is now.

The remaining steps to the release process were:

• create an empty commit with short release notes as the message
• create a tag with the same message
• build the sdist
• turn the crank on making the mac / liunx wheels
• twine upload -s dist/*.tar.gz
• twine upload dist/*whl

[tacaswell]

and I have uploaded the wheels from https://www.lfd.uci.edu/~gohlke/pythonlibs/#datrie to pypi