Support for crossing major-version boundaries

[bitprophet]

I.e. right now, in Paramiko, going from 1.x to 2.x results in: surprise! there's no 2.x displayed anywhere!

Clearly the parse and/or organization logic breaks down in some way, which isn't too surprising, I'm sure there's a lot of assumptions baked in aimed squarely at iterating over minor releases only.

Fixing this may end up solving (fully or partly) #19 and #36.

Remaining todo

• [x] Figure out whether we have solved #19 / #36 or not
• [x] Address whether advanced spec format should imply major/backported behavior or not
• [ ] Implement config option re: issues 'sticking' to latest major release family by default

[bitprophet]

Rambling thoughts to self: This functionality may actually be conceptually incompatible with how Releases currently works!

---

Consider the following scenario, where a project was on a stable 1.5 line, gets some bugfixes, adds some new backwards compatible features for release as 1.6, and also adds some backwards incompatible features for release as 2.0.

They then have a release day, putting out all 3 releases (1.5 bugfix, 1.6 feature, 2.0 backwards incompat feature):

* :release:`2.0.0 <2016-04-25>`
* :release:`1.6.0 <2016-04-25>`
* :release:`1.5.1 <2016-04-25>`
* :feature:`123` Super big huge backwards incompatible feature!
* :feature:`99` Regular feature!
* :bug:`101` Oops, a regular ol' bug.
* :release:`1.5.0 <2016-01-25>`
<prehistory here>

What should happen in this case, re: which "buckets" each item goes into? Currently we do roughly the following:

• Start with just two buckets, unreleased_bugfix and unreleased_feature
• As we walk up the changelog from the beginning of time, :release: roles cause new per-line buckets to be added, going by major+minor version, so :release:1.0.0 generates a 1.0 line bucket, same for 1.1, etc.
• When non-release items are encountered, they get put into all applicable buckets: bugfix items go into every release line + unreleased_bugfix, feature/support go only into unreleased_feature, etc. (With appropriate exceptions for major bugs or backported feature/support items.)
• Encountering bugfix :release roles (defined simply as "releases whose line has already been seen") causes their lines' buckets to get emptied out, so if we've been walking up the timeline after 1.0.0 - consuming bugfixes into a 1.0 bucket - encountering 1.0.1 causes us to dump the contents of 1.0 into the 1.0.1 release.
• Encountering feature :release: roles (defined as "releases whose line we haven't seen before") just empty out unreleased_features.

So, given the above changelog, again ignoring 2.0.0, you'd end up with:

• 1.5.1: bug 101
• 1.6.0: features 123, 99

Great. So what happens when we do want to consider 2.0?

---

In a super naive setup, where major releases follow from minor ones but not at the same time (i.e. if we dropped 1.6.0 from the example) things appear to "work" - but only insofar as it's 2.0.0 that gets features 123 and 99.

Conversely, what happens in the full example is that 1.6.0 "eats" both features out of unreleased_features, leaving it empty when 2.0.0 is encountered. Whoops! (And empty releases don't really count, so they don't get displayed. Bit of a corner case...)

Now, some (maybe even most) projects may happen to fit this scenario, but there's nothing about good versioning practices that says you can't continue putting out 1.x feature releases after 2.0.0.

Additionally, there's the issue of bugs: in a 1.x-and-2.x world, where 2.x is backwards incompatible from 1.x (potentially even a full rewrite!) you can't naively assume all bugs encountered recently belong in both e.g. 1.5.3 and 2.0.1. (Frustratingly, of course, you also can't assume none of them do, either - only in a full-rewrite scenario is that likely the case.)

---

So we clearly need either some serious annotation (if we continue "one timeline rules them all") or consider reorganization somehow (perhaps multiple changelog files and/or sections).

Annotation seems like it'd get annoying fast:

• Requires e.g. everything belonging only to 2.0+ to get tagged with "2.0" or "2" or whatever.
• But it allows one to continue the existing "bend the rules" feature set of major bugs and backported non-bugs; it means one can continue a single timeline without duplicating anything.
• Clearly best suited for a world where 2.0, 3.0 etc aren't hugely divergent from 1.x or each other.

Splitting into multiple sections or files (doesn't matter which, I don't think; either way we'd just be merging them together in the end) allows for easier separation, but means the entire raison d'etre of this library is nullified:

• Now if you have a bug affecting 1.x and 2.x (or, even 1.x, 2.x and 3.x...etc) it must be copied into each section.
• For libraries where the major revisions aren't full-on rewrites, this is real bad, for obvious reasons. Potentially lots of duplication.
• For libraries that are rewrites (hi Fabric 2), it's less bad - but maybe in that case we just want to make it easy to generate multiple, completely distinct changelog files (as in, ones which might even render to separate target files)?

---

tl;dr kind of a classic RDBMS schema (de)normalization problem...

[bitprophet]

More random ideas:

• One could arguably leverage the "explicit release contents" feature (where one puts a comma separated list of bug IDs on the release line) to handle part of this problem?
  • E.g. in tandem with the below idea about "stuff defaults to the most recent major release family", occasional previous-major-family releases could simply indicate "yea I exist and include bugs 15, 78 and 245".
  • However, this starts to fall down a bit more as you stretch across more than 2-3 major release lines, or if your previous-major-family releases are large / frequent.
  • At the same time, compared to annotating "I belong to families 1, 2 and 3" (again as per the following bullet point) all over the place...maybe equivalent or better? Hard to say without seeing it in action.
• We could continue use of implicit behavior - specifically, have similar behavior as now, but limited to the most-recently-seen major release's release lines.
  • I.e. right now the entire setup is implicitly contained within a "1.0" (or whatever you "start" with) major release "context"
  • We can and probably should make that explicit, and then switch up which one is "active" based simply on whatever's most recent. Timeline hasn't gotten to 2.0 yet? everything assumes 1.0. Now 2.0.0 is out? OK, that un-annotated feature is assumed to end up in e.g. 2.1; or, that bug is going to get sucked into 2.0.5 and 2.1.1.
• What needs communication via that annotation is another question. Assuming 3 or more release lines, we'd want to handle up to all of:
  • Single, non-default major release: i.e. "I belong to the 1.x family only"
  • All major releases: i.e. 1.x, 2.x and 3.x
  • Some (presumably contiguous...seriously don't want to even consider non-contiguous, that's crazy) subset of major releases: e.g. 1.x + 2.x, but not 3.x.
  • This strikes me overall as the least likely to appear use case, but still not outside the realm of possibility. E.g. in Paramiko, 1.x + 2.x are actually the same API-wise, with 3.x having actual incompatibility.
    • Thus, I could see fixes or features needing to exist in 1+2 but not 3(+4+...).
• Finally, there's the question of the literal annotation format itself:
  • We already have "keywords" for line items, e.g. :bug:123 major. (And I now regret using "major" for that...heh.)
  • Could add another distinct value for that slot, e.g. 1 meaning "1.x", lines:1,2 meaning 1.x+2.x, that sort of thing. Something easy to distinguish from the existing keywords/flags like major and backported.
  • Could try adding another field of some kind as well, but that can be costly, both cognitively for users and re: potential for bugs (& of course implementation cost). I.e. if one had to declare a "major" bug that needs to appear in both 1.9.0 and 2.2.0, it implies a need for :bug:123 major 1+2.
  • That said, we've got other desired features that push us to expand this crummy mini-language, e.g. #38. It may be worthwhile coming up with an approach that is less costly than arbitrary split()-based parsing - though most things would be much more verbose, i.e. "hey put some JSON in here".

[bitprophet]

Related: I totally forgot we already added a "line" field to the mini-spec; it's currently intended for preventing bugs from going "too far back", so e.g. :bug:57 (1.2+) means "only put this bug into release line buckets for 1.2, 1.3 and so forth - don't put it in 1.0 or 1.1, it doesn't apply there".

Provided the rest of the system is overhauled as above re: major families (including "default to latest major family"), this field could fill our needs?

• Should still work the same as it does now for a changelog spanning only a single major release
• When encountered in a multi-major-release context, it could serve as an indicator to insert the issue into the older line as well as the newer one, and this also lets us indicate a span of contiguous major families.
  • E.g. say Paramiko 3.1.0 is the latest release, and above/after it, we encounter :bug:456 (1.18+). This would cause that bug to copy into the buckets for 1.18, 1.19 etc, 2.0, 2.1 etc, and 3.0 & 3.1.
• The format could be extended to keep things only in older, though how exactly is up for grabs; the "+" suffix is unambiguous but "only before" or "between X and Y" has no similar easy approach fitting in a single suffix character - (2.1-) doesn't make a ton of sense.
  • The obvious move is to just go for regular old dependency version specifiers, i.e. the existing (1.18+) would simply be a shorthand for (>=1.18), then (>=1.18,<2) could mean "only the 1.x line" and so on and so forth as everyone is already used to.
  • Super-ideally, we could rely on an external lib like setuptools or semver or whatever, to handle this. Depends a lot on how abstract they are about it.

[bitprophet]

Yea, semantic_version is something I've used in my Invocations packaging module and it looks like it'll work well for this (+ other spots in here too, honestly, there's bits where we manually parse and should just use semantic_version.Version instead).

Specifically, it has a Spec class that parses strings like the above (>=1.18,<2) and can do various logic things to test if a given Version falls within a Spec or not.

One minor quibble is you can't create Versions to represent release lines (Version('2.1') is invalid) and thus can't answer questions like "which release line buckets should this issue-with-a-spec go into?". Hopefully work-around-able or e.g. subclassable.

EDIT: seems like we were already using distutils.LooseVersion for some of this; I'd bet semantic_version is more useful but will have to see.

[bitprophet]

NOTE: it may still be worthwhile to offer both "versions" of this, i.e. not just annotations but also "one changelog source per major line" (with the config activating/selecting this, also specifying e.g. "1.x is in changelog_1.rst, 2.x is in changelog_2.rst" so each is parsed with different rules). I think this could be a lot nicer for projects with near-100%-incompatible 2.x lines, who'd otherwise have to annotate literally every 1.x (or 2.x depending) related issue.

Good thing is, it should be easy enough to add that after the 1st variant of this feature lands, if and only if it seems truly necessary.

[bitprophet]

ANOTHER NOTE: while I haven't given it serious thought, I am pretty sure all annotations (i.e. major and backported) could be effected by this newer version-spec-using scheme.

E.g. a major bug inserted prior to a 1.1.0 release could be defined instead as :bug:123 1.1+ to keep it out of a 1.0.x bugfix release. (Though I think this would require the logic to change such that annotations override all other rules...) Not sure if this is actually better from a user standpoint though, and again, haven't thought it through all the way, it might not actually be 100% equivalent.

Something to ponder for our own 2.0 tho.

[bitprophet]

Sweet, turns out Version can do partial versions if I...say partial=True. Dunno how I missed that earlier. No workarounds necessary; works great, e.g. Spec('<2.0').filter([Version('1.1', partial=True), Version('1.2', partial=True), Version('2.0', partial=True), Version('2.1', partial=True)]) correctly yields an iterable just containing 1.1 and 1.2.

[MinchinWeb]

What about using :bug:57 (1.6+) to denote versions <=1.6, >2.0 and :bug:57 (1.6++) (double plus) to denote versions <=1.6 including 2.0, 3.0, etc?

That said, as I write this and re-read your comments, using the system already in place elsewhere (like used for setup.py) seems like a bright idea because then it's one less thing to learn.

[bitprophet]

Yea, I specifically wanted to avoid coming up with a new system when the problem domain is already well mapped out :) semantic_version is working well so far.

Have made more progress on this today, running into some implementation hijinx for corner cases, but I think I have a solution in sight.

[MinchinWeb]

Would it also make sense to have a label that denotes a breaking change, similar to a "major" bug? So make your sample changelog from above read:

* :release:`2.0.0 <2016-04-25>`
* :release:`1.6.0 <2016-04-25>`
* :release:`1.5.1 <2016-04-25>`
* :feature:`123 breaking` Super big huge backwards incompatible feature!
* :feature:`99` Regular feature!
* :bug:`101` Oops, a regular ol' bug.
* :release:`1.5.0 <2016-01-25>`
<prehistory here>

[bitprophet]

I've been doing that just by putting a .. warning:: block inside the descriptions for such features. In terms of it indicating how things get rolled up, I'm not sure if it helps any more than using >=2.0/2.0+ would in the same position.

I'll be pushing an update soonish once I am happy with the in-progress implementation but it's basically as described earlier - features just prior to a block of releases containing 2.0.0, and of course any afterwards, would default to "belonging" to 2.0 unless otherwise specified. (So you wouldn't even want to use 2.0+ in the scenario we're discussing - instead, you'd mark any "1.0 but not 2.0" features as being <2.0.)

Said update includes a bunch more explicit examples in the concepts docs, similar to the ones there now.

[bitprophet]

Whee, back to 100% green on the test suite now, including the first meaty test re: new functionality.

Now to fill in the rest of the new skel tests & hope they all 'just work' given the overhaul. And address some TODO's.

[bitprophet]

Surfaced another handful of thorny corner/edge cases while dealing with rest of new tests (& in examining how the changes affected our own changelog). Finally got those all nailed down too.

Will scan for any critical-level TODOs (the rest can wait, I did not anticipate this taking nearly 3 full days of hacking...) and then start applying to Paramiko's upcoming changelog (esp after merging #731 and #733) as a final test.

[bitprophet]

Started poking Paramiko's changelog, some wrinkles or potential ones:

• Marking backported support item 612 (merged to 1.15 and up) as >=1.15 is still only making it appear in 1.17 and 2.0.0.
• Conversely, 649 (major bug intended for 1.17.0 only) isn't appearing at all, because without 'major' and without any 1.17 bugfix releases, there's no bucket for it. (It's not even showing up in e.g. "Next 1.x bugfix release".)
• So clearly "just using a spec" isn't actually a replacement for the major/backported keywords, at least not how it ended up being implemented. Need to double-check the conceptual purity of this and see whether I should consider this a bug or if we do need to allow for up-to-2 keywords...

[bitprophet]

Huh yea looking at the backported stuff, there is actually a regression with that, they're no longer being put in feature release buckets at all. Likely due to how things were cleaned up into is_buglike/is_featurelike. (See #50 as well, this probably is how things should be, but we can't actually change it now w/o breaking things for users.)

[bitprophet]

That part is easily enough fixed.

The previous issue, less so. Core problem here is interplay between "featurelike issues with no spec automatically get assigned to upcoming major releases" and "you can't specify both backported and a spec":

• If the keyword slot is filled by backported, they get stuck to 2.0.0 (because no spec saying otherwise) which is incorrect.
• If the keyword slot is filled by a spec like 1.15+, they will get filed in the 1.x lines...but as regular featurelike issues, not backported ones.

I experimented slightly with letting the specs imply backported behavior (especially as I mentioned earlier in this ticket that we could eventually replace the other keywords entirely by doing so) and this kinda works but it's not compatible with how the feature has been developed & documented so far - i.e. saying :feature:123 (<2.0) to keep something pinned to 1.17. Poof, that spec would then imply backporting as it matches all release lines prior to 2.0; which was not the intent of that example.

So I see two ways out of this:

• Allow both keyword and spec on a single issue.
  • This is probably the least amount of work from here (& I am really sick of this taking forever).
  • My original fears of "growing a custom language/format" aren't really set off by this, it is one single extra potential split item, and I would raise exceptions if someone did something dumb like two keywords or two specs instead of one of both.
• Walk back the current behavior of specs a bit, saying that they're mostly useful for buglike issues, and that featurelike issues must use ==1.17 style pinning (or per-release explicit issue lists) instead.
  • This make sense because it's not like a featurelike issue would ever end up in more than one release either way; giving a 'wider' spec is useless unless you are trying to use it as an implicit backported.
  • The fact that I described it otherwise is testament to a) how over-engineered and complicated this crap is and b) how burnt out I am at the moment...
  • It's a bit more work (have to implement the differing functionality & also scan the docs/tests for the now-incorrect examples). Probably.
  • There may still be other corner cases I haven't considered which this doesn't work for (e.g. - one of our examples/tests puts a feature into both 1.x and 2.x feature releases, via 1.0+; that's harder to disambiguate in this possible new future) - going the "two fields" option helps guard against that.

[bitprophet]

Finally happy with how Paramiko's changelog has turned out. JFC what a boondoggle.