Deployment

[michalk8]

Before we put this on PyPI/conda-forge, here's a list of TODOs:

• [x] create README (also add small code sample) #2
• [x] create comprehensive installation intructions for Linux/macOS #2
• [x] add tests for stationary_distribution #3
• [x] use k=m in https://github.com/msmdev/pyGPCCA/blob/main/pygpcca/_sorted_schur.py#L358 - correct @msmdev ?
• [x] check for typos in the docs/fix badge links
• [x] add PYPI_PASSWORD repo secret

After this, we can put in on PyPI and afterwards, we can create a PR to conda-forge.

[msmdev]

One thing that came to my mind: how will we handle versioning?

[michalk8]

One thing that came to my mind: how will we handle versioning?

There's https://github.com/msmdev/pyGPCCA/blob/main/.bumpversion.cfg which creates the bump commit (e.g. bump2version patch) + creates a new tag. You just need to push the commit with the tag to the repo (I have in my git config push.followtags=true so that they are pushed simultaneously). Then CI runs and if it's OK, it will create new build on PyPI.

[Marius1311]

Okay!

[msmdev]

I updated the README (currently just on the update-readme branch). I think we are very close to be able to deploy (and should soon, since pyGPCCA should be ready to use in cellrank until the re-submission of the paper) - so what still needs to be done and who will take care of what?

[Marius1311]

Let me know if I can help in any way - I'm pretty occupied with the revision at the moment and trying to focus on that, but could take over little tasks if needed.

One thing that would be nice is a concept figure in the README - I think repos with a concept figure just look much more appealing and friendly.

[msmdev]

A concept figure is a very nice idea! Do you have anything in mind?

[Marius1311]

Mhm, is there anything in your old papers we could use?

[msmdev]

Hmmm, currently only an old graphical abstract comes to my mind, but I fear it's is to specific...

[Marius1311]

yes, this is too specific to protein folding, we want something more general that focuses on GPCCA rather than an application. I will see whether I can come up with something. However, this is really not essentially, we can put in on pypi without this and add later.

[msmdev]

@michalk8 IMPORTANT: even if we are ready soon, we have to wait with the deployment on PyPI and conda until I have an OK from Berlin... (this should come today/tomorrow)

[michalk8]

Ok, sounds good. Here are the deployment steps you need do:

0. make sure you have push.followtags=true in your git config
1. make sure you do pip install -e'.[dev]' 1.5 git checkout main
2. run bump2version <part> where <part> is one of major, minor, patch (as in {major.minor.patch})* 2.1 sanity check: verify that new tag has been created: git tag 2.2. verify that a new commit with msg like Bump version: 1.0.0 → 1.1.0 has been created 2.3 super extra sanity check: verify the new version has been written in these 3 files: setup.py, pygpcca/__init__
3. run git push
4. let CI run, if the linting/tests are all OK, deployment starts (you can watch it go here: https://github.com/msmdev/pyGPCCA/actions)
5. assuming everything is correct (CI conditions/PyPI credentials), it should be on PyPI

If anything goes awry during commiting, don't push the commit, just revert it (don't forget to also delete the newly created tag [if it was created] as git tag -d <tag_id>. Locally, I've tried bump2version major, worked fine (the new version is 1.0.0, which is expected). If you're also trying this out, you might want to try --dry-run --verbose to see what would be changed. If you don't want to create a new tag, just specify --no-tag. No tags push to main = no new release.

*alternative is to specify --new-version {major}.{minor}.patch

[msmdev]

@michalk8 ok and thank you for the detailed guidance! I will wait for the input regarding credits from @Marius1311 before deploying. Just one question: I would preferably go for bump2version major in step 2. - correct?

[michalk8]

Just one question: I would preferably go for bump2version major in step 2. - correct?

Yes. Also, we need to change the development status classifiers on PyPI: https://github.com/msmdev/pyGPCCA/blob/main/setup.py#L53 - should be Development Status :: 5 - Production/Stable.

Also, if you find any other useful classifier in https://pypi.org/classifiers/, feel free to add. Actually, you can do that in 1 commit when bumping the version as bump2version major --allow-dirty (but try with --dry-run first).

[msmdev]

@michalk8 I also registered that the coverage is just around 76%. How about that?

[michalk8]

@msmdev if you think any lines that we miss here is worth testing for, then let's create a separate issue for that

https://codecov.io/gh/msmdev/pyGPCCA/tree/d5e589ca168684e16f52c14f25a8a86fdcebb6c1/pygpcca (this is the last non-skipped commit on Ubuntu, Python3.8 on add-example branch).

The _checks.py file has ~47% coverage, but most of the stuff we're missing are corner cases (ValueError, etc.).

[msmdev]

Ok, lets leave it for the time being - can be handled in a future version.

[msmdev]

@michalk8 another question: I thought that we originally enforced the data type of the input transition matrix to be double and also ensured that all floating point calculations were double precision, but I couldn't find it - is it still done? Or are we currently just trusting in python/numpy internal type handling here?

[Marius1311]

Thanks guys for pushing this, and sorry for not being super active at the moment, I'm quite occupied with revision work.

[michalk8]

Binder now works as expected.

[msmdev]

Ok, sounds good. Here are the deployment steps you need do:

1. make sure you have push.followtags=true in your git config
2. make sure you do pip install -e'.[dev]' 1.5 git checkout main
3. run bump2version <part> where <part> is one of major, minor, patch (as in {major.minor.patch})* 2.1 sanity check: verify that new tag has been created: git tag 2.2. verify that a new commit with msg like Bump version: 1.0.0 → 1.1.0 has been created 2.3 super extra sanity check: verify the new version has been written in these 3 files: setup.py, pygpcca/__init__
4. run git push
5. let CI run, if the linting/tests are all OK, deployment starts (you can watch it go here: https://github.com/msmdev/pyGPCCA/actions)
6. assuming everything is correct (CI conditions/PyPI credentials), it should be on PyPI

If anything goes awry during commiting, don't push the commit, just revert it (don't forget to also delete the newly created tag [if it was created] as git tag -d <tag_id>. Locally, I've tried bump2version major, worked fine (the new version is 1.0.0, which is expected). If you're also trying this out, you might want to try --dry-run --verbose to see what would be changed. If you don't want to create a new tag, just specify --no-tag. No tags push to main = no new release.

*alternative is to specify --new-version {major}.{minor}.patch

@michalk8 Regarding 1.: Is it done via git config --global push.followTags true?

[michalk8]

@michalk8 Regarding 1.: Is it done via git config --global push.followTags true?

Yes.

[msmdev]

@michalk8 somehow all tests passed, but deploy was first skipped and just started when I clicked it, but failed here resp. was skipped here

[michalk8]

@msmdev it cannot build wheel for some reason. Looking closer at pyproject.toml, it isn't specified as a requirement in the build-system. Can I revert the last commit + delete the tag on upstream and fix this? Then you can again run bump2version.

[msmdev]

@michalk8 ok!

[michalk8]

@msmdev done in https://github.com/msmdev/pyGPCCA/commit/4fac09ffc514354855398cfa98381a93e6f15af0. Make sure you pull from main and locally delete your tag locally as well git tag -d v1.0.0, before running bump2version again.

[msmdev]

@michalk8 should I mb. additionally update anything locally, e.g. pip etc.?

[michalk8]

Not really, if you want, you can double-check (I already did):

pip install build
python -m build --sdist --wheel --outdir dist/

and it should have 0 exit status.

[msmdev]

@michalk8 after git pull origin main I still have the bump commit in my log:

a6125b5 (HEAD -> main) Merge branch 'main' of https://github.com/msmdev/pyGPCCA into main
4fac09f (origin/main, origin/HEAD) Add forgotten 'wheel' in build-requirements
6a44fab (tag: v1.0.0) Bump version: 0.0.0 → 1.0.0
7006afb Improve `GPCCA.optimize()`: Change complex conjugate splitting handling etc (#13)
073cffb Fix rtd (#16)

[michalk8]

Just try git reset --hard 073cffb and git pull should do the trick.

[msmdev]

Jup, worked.

[msmdev]

Will remove the tag now and proceed...

[msmdev]

pip install build
python -m build --sdist --wheel --outdir dist/

Ok, worked it seems.

[michalk8]

Ok, worked it seems.

Great. Let's see now if the PyPI upload also works.

[msmdev]

bump2version worked, tag 1.0.0 was created, commit is also there and in setup.py it is written. Will push now.

[msmdev]

@michalk8 when the tests passed, where should I first click deploy in actions?:

• Bump version: 0.0.0 → 1.0.0, CI #107, main
• Bump version: 0.0.0 → 1.0.0, CI #108, v.1.0.0

[michalk8]

The v.1.0.0, since this one was triggered on tag push.

[msmdev]

ok!

[michalk8]

I've just watched it, the problem seems to be the README and the ..raw directive in .rst.

[msmdev]

Yep, guessed the same x.x

[michalk8]

I will fix this short + 1 more issue, will let you know once it's ready.

[msmdev]

So, same procedure as before? you revert and fix and I try again?

[michalk8]

Yes, that's the plan. We have the check for this in tox as tox -e readme, but I forgot to add it to linting.

[msmdev]

Okdok

[michalk8]

@msmdev done, you can try again, hopefully for the last time. One thing to be careful about: in setup.py, we automatically set the description + long_descript from README (the description is line 35, the long_description is the full file, except for the .. raw, which is removed). For the long_description, I replace |br| with \, which should be the markdown line break.

[msmdev]

@michalk8 ok, I repeated everything and pushed - thumbs crossed.

[msmdev]

@michalk8 oook, everything worked this times besides of invalid authentication info...

[michalk8]

Yes, I've seen it - the username should be __token__ as seen here and the password should be a repo secret named PYPI_PASSWORD. Is this set up correctly? Here's how mine looks like:

[msmdev]

Yes, I've seen it - the username should be __token__ as seen here and the password should be a repo secret named PYPI_PASSWORD. Is this set up correctly? Here's how mine looks like:

hmmm, two things:

• I have set up two-factor auth
• my token is named PYPI_PASSWORD and not __token__

[michalk8]

my token is named PYPI_PASSWORD and not token

This is the issue most likely, we can either rename the user in the CI to also PYPI_PASSWORD or change token name in your PyPI.

[msmdev]

@michalk8 I will delete and name to __token__ on PyPI and add this on GitHub.