Creating next stable release

[kaplun]

@dset0x needs a new stable release of intbitset because he suspects the current release is what leads celery to crash.

@jirikuncar, @tiborsimko, can you publish your vademecum on what is required to make an Invenio package release?

[dset0x]

@kaplun is referring to #13 here. At least in one instance of my issues this was proven to be the problem.

[kaplun]

BTW, have you tried installing the master branch of intbitset to verify if it solves the issue?

[dset0x]

Yes, the issue goes away with pip install 'git+https://github.com/inveniosoftware/intbitset.git'.

[kaplun]

Awesome!

[jirikuncar]

@kaplun since the commit messages are not following the inveniosoftware standards, I can't use the automatic release notes generator, hence it will take more time.

[kaplun]

What I am looking for is actually the list of things I need to do in order to make a release directly myself, following inveniosoftware recommendation.

[kaplun]

I manually completed the CHANGES file, added Travis to Pypi integration and made the release. @dset0x you can now directly use native intbitset.

[jirikuncar]

Released PyPI package is broken and the commit + tag doesn't follow inveniosoftware standards. You could create a PR first before proceeding with release.

[kaplun]

What are the inveniosoftware standards?

[kaplun]

The point of this very issue is the lack of documentation for above mentioned standards. If on the one hand commit logs style are well documented (for developers), on the other hand the releasing process document is still missing, and this makes only 3 people to have the knowledge about what are the Invenio standards.

[kaplun]

commit + tag

What do you mean by commit + tag? About the fact that the tag is called v2.2.0? That was always the case...

[jirikuncar]

There is also a message attached to each tag.

The point of this very issue is the lack of documentation for above mentioned standards.

(1) Well if you are not sure, you can always prepare PR and let somebody to check it (which should be the case for all PR).

If on the one hand commit logs style are well documented (for developers), on the other hand the releasing process document is still missing,

(2) The problem here is, that the commit messages are not following the standard, hence we can not use our toolchain.

and this makes only 3 people to have the knowledge about what are the Invenio standards.

(3) The standards are evolving so one has to check the latest best practices. + point (1)

[kaplun]

There is also a message attached to each tag.

Thanks I will use it in the future.

(1) Well if you are not sure, you can always prepare PR and let somebody to check it (which should be the case for all PR).

But:

• I do want to be sure but I don't have the means to be sure (i.e. a reference document)
• We need control over the release process in order realize concretely what has been decided at the summit, i.e. to distribute the responsibility of package maintainership.

(2) The problem here is, that the commit messages are not following the standard, hence we can not use our toolchain.

Right so we can do this from this release onwards. BTW where is the toolchain so that we can use it as well?

(3) The standards are evolving so one has to check the latest best practices. + point (1)

Right, but since there is no written document for reference, how can we possibly follow best practices without being in your same office?

[jirikuncar]

BTW where is the toolchain so that we can use it as well?

• kwalitee prepare release vX.Y.Z..

Right, but since there is no written document for reference, how can we possibly follow best practices without being in your same office?

You can come every Monday to Invenio Developers Forum where we discuss the best practices.

[kaplun]

kwalitee prepare release vX.Y.Z..

Awesome. Thanks

You can come every Monday to Invenio Developers Forum where we discuss the best practices.

Unfortunately I personally won't be able anymore (family reasons).

OK I guess I will create a vademecum reference document myself by trying to reverse engineering all your different processes. I hope not to miss anything

[tiborsimko]

No need to reverse-engineer anything, for independent packages we follow the usual Python packaging practices with goodies like check-manifest and trying on testpypi in case of doubt. The Invenio add-on layer is rather small and still evolving, e.g. (1) some parts are committed to kwalitee, such as the release notes generator, (2) some parts are not committed to kwalitee yet, such as the authorship verifier (hence my PR #32), and (3) some parts are not automatised yet, such as publishing of GitHub-friendly release notes via API (i.e. not storing everything in the tag) followed by sending email and tweet. Finally, you know well the usual recommended practices of Invenio packages, such as (a) "every commit must be seen by another pair of eyes" (hence PR and signatures) and (b) "a PR is not considered unless it preserves or increases the code coverage" (hence TDD). The former principle might have permitted to catch today's release troubles during review...

[jirikuncar]

[kaplun]

Thanks Tibor, for finally having briefly listed the points I was desperately asking since long time. As you can see from your own words there is more than just the reference python packaging practicing. So it's not that obvious how to make an Invenio module release.

Related to today glitch, who would have inspected my package? The error was only in the sdist part. Not in the code that made the release per se. So I would consider it more rather an unfortunate case, that of course could have been avoided if all the best practices (such as the check-manifest trick that I was unaware of), once known, would have been applied.

[kaplun]

Finally, you know well the usual recommended practices of Invenio packages, such as (a) "every commit must be seen by another pair of eyes" (hence PR and signatures) and (b) "a PR is not considered unless it preserves or increases the code coverage" (hence TDD). The former principle might have permitted to catch today's release troubles during review...

Note that this part is for integrator, but it's unrelated to the problem of "what constitutes an Invenio module release".

[tiborsimko]

Thanks Tibor, for finally having briefly listed the points I was desperately asking since long time.

Yeah, I know I promised to finalise the write-up e.g. summarised in https://github.com/inveniosoftware/invenio/issues/2876#issuecomment-130346421

Related to today glitch, who would have inspected my package?

Ideally you'd find another release peer. Perhaps me or @jirikuncar would have volunteered to eye-ball the release PR, though indeed this is no guarantee of spotting the issue at hand. Sometimes peers are not available or are otherwise busy, so the coordination may require time, which may be frustrating. Other times the time may be simply lacking in zeitnot-like situations. But, as a general principle, there are more pros than cons behind always requiring another pair of eyes behind every PR, and paying for it with time.

WRT the concrete issue at hand, the need to change MANIFEST.in would have ideally been pointed out as part of your earlier PR #26 during the usual second-signature review process. (I see that @jirikuncar left some comments in that PR, but apparently only a posteriori, it seems that you self-merged first.) Anyway, now that check-manifest is always run during Travis CI, this concrete problem won't occur anymore.

P.S. Just some general comments, following an ad hoc forensics, to illustrate the two-signature process's pros and cons. Not to be over-interpreted or over-analysed, I make other mistakes too :smile: Just trying to point towards a set-up where using more eye balls consistently lessens the space for mistakes to appear. This is especially important for "core" packages such as intbitset that are used right at the heart of the Invenio ecosystem.

[pombredanne]

I am closing this in order to clean up old tickets and continue from clean slate.