Release manager needed for pandas 0.24.3

[datapythonista]

The next version of pandas will be 0.25.2, but we'll release another version of 0.24, the 0.24.3 since pip install pandas may fail in Python 2. The pandas 0.24 series is the last compatible with Python 2, but numpy released a new Python 2 compatible version, that can be automatically installed with pip install pandas, and that is not compatible with pandas 0.24.

The proposed solution is to release a new version 0.24.3, with the only change respect to 0.24.2 that requires numpy < 1.17.

See the original issue here: https://github.com/pandas-dev/pandas/issues/27435

Making the change to setup.py is trivial, but releasing a new pandas version not as much (never did it myself). Most of the work will be in understanding the process, and also on building packages (for pip and for conda). Couple of the steps will require the right permissions (access to the server where we have the documentation, and commit rights on the pandas git repo, but someone with those permissions can take care of those steps.

Those are the notes from the meeting where this was discussed:

• pip install pandas works when there’s a binary available.
• pip install --no-binary=pandas pandas numpy==1.16 fails
• pip install numpy && pip install --no-binary=pandas pandas works (if you have a compiler)
• We need a PR to update numpy dependency environment marker for just py 27. Backport to 0.24.x
• Then one of us has to tag 0.24.3 we do the sdsit to github
• Someone builds binaries on CF(?) wheels.
• One of use uploads binaries and sdist to pypi.

[TomAugspurger]

Most of this is semi-automated in https://github.com/pandas-dev/pandas-release (using docker for the local stuff, and making pull requests to GitHub).

[Aya-S]

How realistic do you think is it to have someone who is a noob attempt this?

[datapythonista]

How realistic do you think is it to have someone who is a noob attempt this?

If you're motivated to do it, you should totally do it. It'll require a bit of time to understand things (like, building coda packages...), but there is nothing challenging that you can end up saying "I tried, but I wasn't good enough". :)

And of course you won't be alone, any question we're here to help.

[datapythonista]

@Aya-S if you have some time in the next days, I'm happy to do the release together. I'll have to learn as much as you about it, but I'm sure we'll manage. :)

Let me know if you're interested.

[jorisvandenbossche]

The PR that is needed for the 0.24.3 release (the fix to enable pandas to be installed from source on python 2.7) is https://github.com/pandas-dev/pandas/pull/28511

[Aya-S]

@Aya-S if you have some time in the next days, I'm happy to do the release together. I'll have to learn as much as you about it, but I'm sure we'll manage. :)

Let me know if you're interested.

@datapythonista this sounds like a very good deal! I couldn't bring myself to feel confident enough to attempt this at first. how do we start :D?

[datapythonista]

That's great!

First thing would be to get a branch with the version to release. That is basically branching from 0.24.2 and adding the commit from pandas-dev/pandas#28511 (which is not yet merged).

And after that, create a new release on GitHub, create packages for pip and conda, and update the website/documentation.

But we'll practice a bit before we do it for real. Tom sent us the instructions to do the release: https://github.com/pandas-dev/pandas-release

Do you want to have a look at the document, try to do what's in there, and discuss the questions you have? There are things like pushing to git, uploading the documentation... that you won't be able to do. That's good news, because you can test as much as you want, that you won't be able to break anything. :) Just skip these steps when you find them.

Does this make sense?

[Aya-S]

That's great!

First thing would be to get a branch with the version to release. That is basically branching from 0.24.2 and adding the commit from pandas-dev/pandas#28511 (which is not yet merged).

And after that, create a new release on GitHub, create packages for pip and conda, and update the website/documentation.

But we'll practice a bit before we do it for real. Tom sent us the instructions to do the release: https://github.com/pandas-dev/pandas-release

Do you want to have a look at the document, try to do what's in there, and discuss the questions you have? There are things like pushing to git, uploading the documentation... that you won't be able to do. That's good news, because you can test as much as you want, that you won't be able to break anything. :) Just skip these steps when you find them.

Does this make sense?

ok, great. Then I'll start with the document and let you know how things go!

[Aya-S]

@datapythonista @TomAugspurger I checked out the tag for v0.24.2 `commit cb00deb94500205fcb27a33cc1d0df79a9727f8b (HEAD -> trial_0_24_2, tag: v0.24.2) Author: Joris Van den Bossche jorisvandenbossche@gmail.com Date: Tue Mar 12 22:12:11 2019 +0100

RLS: 0.24.2

`

and was starting to follow the instructions but the makefile I have now, which is the following, doesn't match the instructions, seems like the checkout is not right? any idea what might be causing the discrepancy? Thanks in advance :D `tseries: pandas/_libs/lib.pyx pandas/_libs/tslib.pyx pandas/_libs/hashtable.pyx python setup.py build_ext --inplace

.PHONY : develop build clean clean_pyc tseries doc

clean: -python setup.py clean

clean_pyc: -find . -name '*.py[co]' -exec rm {} \;

build: clean_pyc python setup.py build_ext --inplace

lint-diff: git diff upstream/master --name-only -- "*.py" | xargs flake8

develop: build -python setup.py develop

doc: -rm -rf doc/build doc/source/generated cd doc; \ python make.py clean; \ python make.py html python make.py spellcheck`

[datapythonista]

The Makefile you need to use is not the pandas one, but the one in the repo for the pandas release: https://github.com/pandas-dev/pandas-release/blob/master/Makefile

[Aya-S]

Makes sense (y) will get the right one.

On Sun, Sep 22, 2019, 12:49 AM Marc Garcia notifications@github.com wrote:

The Makefile you need to use is not the pandas one, but the one in the repo for the pandas release: https://github.com/pandas-dev/pandas-release/blob/master/Makefile

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/python-sprints/pandas-mentoring/issues/166?email_source=notifications&email_token=ABUSU7UXM6AWBL6IF5HU7H3QK2QIJA5CNFSM4IVZNKE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7I25VI#issuecomment-533835477, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUSU7VGBXZEFCIY435OHO3QK2QIJANCNFSM4IVZNKEQ .

[Aya-S]

Hello @datapythonista ! I'm at the stage of fetching the PR to merge to the v0.24.2 tag. i used this command: git fetch upstream pull/28511/head:test-branch wondering about 1) how to test if the fetch succeeded 2) how to test the codebase for coherency and make sure the PR was properly fetched

[datapythonista]

In git all repos are the same technically speaking, so you can do that with your fork to test is everything works.

But I'd move forward and check more the packaging part, I think that's the tricky part of the process, more than anything else. What do you think?

[Aya-S]

In git all repos are the same technically speaking, so you can do that with your fork to test is everything works.

But I'd move forward and check more the packaging part, I think that's the tricky part of the process, more than anything else. What do you think?

I'm currently at: make docker-image docker-doc it has been on for a while and not returning, not sure if it is supposed to take so long or if there is a certain bottleneck on my side. Hopefully, it returns by morning so I could proceed :D

[datapythonista]

I don't know much about docker, but I don't think that should take more than some minutes. @TomAugspurger do you know if it's normal that it takes time that step, or what can be the problem?

Thanks @Aya-S for working on this, sounds really promising, I think we'll have a new pandas release soon. :)

[Aya-S]

I don't know much about docker, but I don't think that should take more than some minutes. @TomAugspurger do you know if it's normal that it takes time that step, or what can be the problem?

Thanks @Aya-S for working on this, sounds really promising, I think we'll have a new pandas release soon. :)

fingers crossed :D !

[TomAugspurger]

The docker step can take a while. We install latex for the pdf build which is a big download.

On Sep 24, 2019, at 17:17, Aya Salama notifications@github.com wrote:

I don't know much about docker, but I don't think that should take more than some minutes. @TomAugspurger do you know if it's normal that it takes time that step, or what can be the problem?

Thanks @Aya-S for working on this, sounds really promising, I think we'll have a new pandas release soon. :)

fingers crossed :D !

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Aya-S]

Hey guys! I went through the steps till make conda test the output files of both make pip-test make conda-test are attached conda-test.txt pip-test-out.txt

understandably, I can't execute make push-tag so skipped it.

created a dummy release on my fork at:https://github.com/Aya-S/pandas/releases/tag/v0.24.3

now on the next step: make conda-forge I'm getting this error

File "/home/aya/anaconda2/lib/python2.7/site-packages/conda_smithy/configure_feedstock.py", line 13, in import conda.config ImportError: No module named config

I'm still to resolve it

[jorisvandenbossche]

What version of conda and conda-smithy do you have installed?

[Aya-S]

Conda: 4.7.12 Conda-smithy: 2.3.3

On Thu, Sep 26, 2019, 1:25 PM Joris Van den Bossche < notifications@github.com> wrote:

What version of conda and conda-smithy do you have installed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/python-sprints/pandas-mentoring/issues/166?email_source=notifications&email_token=ABUSU7WCMHQH53KOEIOIIK3QLSL3NA5CNFSM4IVZNKE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7VHUEY#issuecomment-535460371, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUSU7SER7FKTJYFOJS2MY3QLSL3NANCNFSM4IVZNKEQ .

[jorisvandenbossche]

That seems an old release of conda-smithy, they are currently at 3.4.8 (https://github.com/conda-forge/conda-smithy/releases), so I would try to upgrade that, and see if that solves the problem.

[Aya-S]

you're right, that's very weird indeed because my first instinct was to do a conda update and I thought i had the latest version. thanks, will make sure I have the latest version

[datapythonista]

If conda env create and conda env update installs the wrong version when using the environment.yml, may be it make sense to force a minimum version in that file? (e.g. conda-smithy>=3.4)

[Aya-S]

I didn't create a conda env using the environment.yml file actually guess it wasn't mentioned anywhere :D. will try it out

[datapythonista]

Feel free to add it to the instructions. conda env create from the root of the pandas-release repo should create the environment with the dependencies needed for the release.

[Aya-S]

Yup, let me do that. There is a typo in the environment.yml file will edit it as well.

On Thu, Sep 26, 2019, 4:08 PM Marc Garcia notifications@github.com wrote:

Feel free to add it to the instructions. conda env create from the root of the pandas-release repo should create the environment with the dependencies needed for the release.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/python-sprints/pandas-mentoring/issues/166?email_source=notifications&email_token=ABUSU7RQJTE4NGDIW2NZA6LQLS64JA5CNFSM4IVZNKE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7VWA5Y#issuecomment-535519351, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUSU7QHPHWIMBSEAMZMDXLQLS64JANCNFSM4IVZNKEQ .

[Aya-S]

So, after using the conda env based on environment.yml things worked fine (Y)

now i'm at the step that says

Open PRs for each of those

Should I actually do that or leave this for now until we do some further testing on the artefacts of the build?

[datapythonista]

Do you have the packages built then? If that's the case, I think we're ready to start again for real when the PR we want to release is merged. Or am I missing something?

[Aya-S]

well, I am definitely missing something. at which point are the packages built? I assumed, make conda-forge and make wheels are the rules that build the binaries? they succeeded yesterday but afterwardds I couldn't find any .whl packages under the "pandas-release" dir.

and surly enough, upon checking the scripts for both rules no building seems to be taking place, as far as I could tell. as it seems, I need to start PRs for the building to start on a remote server? would be great to get some clarification on this :) @tomAugspurger

[TomAugspurger]

The only package built locally is the sdist. The binaries (wheels and conda packages) are built on CI services. make conda-forge and make wheels create commits and push to a branch, but you would need to open the pull requests on GitHub through your browser once it's time to do this for real.

[Aya-S]

OK great, this makes sense. Thanks @TomAugspurger. I put the built sdist on a dummy release on my fork: https://github.com/Aya-S/pandas/releases/tag/v0.24.3. Can you confirm that there is nothing more to do at this point until the branch is ready for the actual build? what I did was that I executed the commands till make wheels skipping make push-tag and this block: git checkout -b <TAG>.x git push upstream <TAG>.x git checkout master git commit --allow-empty -m "Start <NEXT_TAG>" git tag -a v<NEXT_TAG>.dev0 -m 'DEV: Start <NEXT_TAG> cycle' git push upstream master --follow-tags

[TomAugspurger]

Yep, that sounds about right.

I suspect that we'll want a pandas maintainer to build the sdist, since only we'll be able to upload things to GitHub and push to master.

After that (push-tag), you'll be able to pick things up again.

On Sun, Sep 29, 2019 at 5:37 AM Aya Salama notifications@github.com wrote:

OK great, this makes sense. Thanks @TomAugspurger https://github.com/TomAugspurger. I put the built sdist on a dummy release on my fork: https://github.com/Aya-S/pandas/releases/tag/v0.24.3. Can you confirm that there is nothing more to do at this point until the branch is ready for the actual build? what I did was that I executed the commands till make wheels skipping make push-tag and this block git checkout -b .x git push upstream .x git checkout master git commit --allow-empty -m "Start " git tag -a v.dev0 -m 'DEV: Start cycle' git push upstream master --follow-tags`

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/python-sprints/pandas-mentoring/issues/166?email_source=notifications&email_token=AAKAOIXVXTILCLN4D35DOODQMCAOJA5CNFSM4IVZNKE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD73QEJA#issuecomment-536281636, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKAOIUWAKQQPSMFIB4TK33QMCAOJANCNFSM4IVZNKEQ .

[Aya-S]

when the branch is ready, I can build the sdist(tar) for someone with the needed permission to upload.