search

nils-braun / b2luigi

Task scheduling and batch running for basf2 jobs made simple
GNU General Public License v3.0
17 stars 11 forks source link

Automate deployment #74

Closed meliache closed 3 years ago

meliache commented 3 years ago

Automate the bumpversion, push and publishing to PyPi, which are currently separate steps as described in the development documentation. It previously contained the comment by @nils-braun

At a later stage, I will try to automate this.

But as far as I know this had not been implemented it. I removed that comment in the development PR #73 and decided to turn it into this issue here for more visibility and for keeping the docs up-to-date.

daritter commented 3 years ago

Speaking of this: Can I have a new release on pypi? The last version on pypi depends on luigi<2.7.9 but 2.7.8 is not compatible with a recent jupyter due to conflicting tornado requirements.

The git repository has updated requirements but there's no release for this.

meliache commented 3 years ago

Speaking of this: Can I have a new release on pypi?

@daritter Thanks for the reminder. The tornado issue is why we updated the required luigi version in #52. I'll try to do a minor version bump this week (might wait for PRs #76 and #77 which are close to being merged) and then ask @nils-braun to publish this to PyPi since I don't have admin rights there.

As a new github maintainer I wanted to update the version since a while but kept postponing that b/c there were always new PR's and issues, especially with the new gbasf2 version which I tried to keep up with. But I think I should just do the version bump at some arbitrary point now, we don't have as strict a release schedule as you with basf2 and I can just do a new patch release any short time later.

Then, I guess you might be interested into updating the version in the basf2 externals. I'd say that the local/LSF/htcondor batches are pretty stable and should be updated (not sure if the htcondor one is even in the externals release), for any user of the gbasf2 batch I'd recommend to install the latest release from github to keep up with braking changes in gbasf2.

daritter commented 3 years ago

b2luigi is AFAIK not used in production except for the systematics framework and they depend on gbasf2.

So we can remove b2luigi from the externals and have users install it manually. But if there's a version that works with the systematic framework which can be included in the externals that would of course be easier.

nils-braun commented 3 years ago

@meliache I can make you an author of the pipy repo (what is your pipy username?). But we might also want to fix this issue right away. I will see if I can find some time today evening.

meliache commented 3 years ago

@nils My PyPi username meliache, same as on github, haven't much used that though.

For automizing the publishing I assume we can do it via github actions and a corresponding yaml file? Probably something I could add myself once I have all corresponding rights. But I'm not sure if that's what you had in mind, if you could find some time (today or in the future) this would be great!

meliache commented 3 years ago

@daritter

b2luigi is AFAIK not used in production except for the systematics framework and they depend on gbasf2.

So we can remove b2luigi from the externals and have users install it manually. But if there's a version that works with the systematic framework which can be included in the externals that would of course be easier.

Just checked via git grep and the only place in the basf2 software where b2luigi is used, is the track quality indicator MVA training script combined_quality_estimator_teacher.py which I wrote a long time ago and which was one of the reasons I asked for b2luigi to be included in the externals. But this is used by experts only and they can pip-install b2luigi if they ever need it, the script even prints a nice message how to install the package if it's not installed.

So I'm okay with removing it from the externals :)