Proposal for Eclipse OMR release process

[xliang6]

@charliegracie , @mstoodle , @0xdaryl and I have had a couple of discussions about what the Eclipse OMR release process would look like. The following is what we came up with as a reasonable starting point proposal for the community to comment on. The purpose of this issue is to document what we discussed and to pose to the community for review ahead of starting the Eclipse release process for the first release.

Given this is a fairly important topic for the Eclipse OMR project, a timeline of 2 weeks has been set for any discussions to take place. We will review and finalize this Eclipse OMR release process document based on the community’s feedback on February 15th, 2018.

Introduction

As Eclipse OMR matures with more downstream consumers, it is necessary to create proper releases for the project. Following the principles of semantic versioning, we would like to support three categories of releases for Eclipse OMR:

• MAJOR releases include API changes (potential for downstream breakage);
• MINOR releases add new functionality, but are API compatible with previous versions; and
• PATCH releases include bug fixes only and include no significant new functionality.

Proposal

We are proposing a MAJOR.MINOR.PATCH format for OMR release version numbers, with additional labels for pre-release and build metadata as extensions to this format. For example, the first release candidate build for the second major release will have version number 2.0.0-rc1. The same version number will be shared among all components within the Eclipse OMR project. As OMR is an Eclipse project, we also have to adhere to the Eclipse project release guidelines. This means that we must engage in a release review for all major and minor releases. Release reviews are not required for bug-fix/patch releases.

Project releases can be time-based or feature-based. For Eclipse OMR, we propose time-based releases. 

1. Every month, milestone builds will be created as git tags with -mX extensions for the version numbers. For example, the version number for the first milestone build for OMR will be 0.0.0-m1. Testing for milestone builds will be minimal, only including OMR continuous integration tests which are performed when a pull request is merged on Github.
2. Every three months, minor releases will be created as git branches. A minor release typically includes API compatible changes for new features and performance improvements. Since Eclipse OMR is a language runtime toolkit that includes many components such as JIT compiler, garbage collector, platform abstraction library, and developer tooling capabilities, its overall set of APIs is difficult to define and guarantee. While some components may be able to offer well-defined and stable API, others cannot. Therefore, the minor release API compatibility guarantee will be on a "best-effort" basis and any major API breakages will be clearly documented in the release notes. We recommend OMR downstream projects picking up an earlier monthly milestone build to perform project relevant testing and creating Github issues for any bugs found. Reported bugs will be prioritized and high-priority bugs will be fixed in the subsequent monthly milestone build. Apart from downstream project testing, OMR minor release testing will include continuous integration tests and additional builds with different build flags, i.e. debug builds with assertions turned on, etc.
3. Patch releases will be created on an as-needed basis as git branches. A patch release includes high-priority bug fixes found in an earlier minor release. Testing for a patch release will be the same as testing for a minor release.

At this point, we are not ready to propose any details on a major release other than the fact that a major release may include API breaking changes. Our immediate goal is to get Eclipse OMR on a release track and start moving forward on that path towards a major release. After a few minor releases, we will be better equipped to figure out what really makes for a major release other than “time” alone.

Thanks for reading and being involved in the Eclipse OMR community!

[fjeremic]

I have several questions:

1. What is the purpose of the milestone builds? If every PR has to pass the same criteria as a milestone build isn't every commit in the master branch by definition a milestone build?
2. The definitions regarding API breakages in the Introduction and Proposal sections for minor releases conflict. The former states API compatibility and the latter states API breakages. Which one is correct?
3. Will all builds for a particular release or milestone or patch be archived? For example to determine which compiler levels were used to build and test, etc. since these may change from release to release? Or can we change build compilers in a minor release for example? Where is the cutoff point?
4. Will we be taking advantage of https://github.com/eclipse/omr/releases?
5. Other than making sure one clones the precise SHA commit will there be a way for an end user to determine what version of OMR they're consuming? i.e. should the version information be encoded somewhere in the repository at the time of branching?

[xliang6]

Thanks @fjeremic for the quick feedback! Let me try to answer some of your questions.

What is the purpose of the milestone builds? If every PR has to pass the same criteria as a milestone build isn't every commit in the master branch by definition a milestone build?

Monthly milestone builds are basically marked points in time on the master branch. We encourage downstream projects to do their project specific testing on these builds (as opposed to a constantly changing code base) and report bugs they find. Patch and minor releases will be branched off one of these monthly milestone builds plus bug fixes. Also, the monthly milestone builds will be archived.

The definitions regarding API breakages in the Introduction and Proposal sections for minor releases conflict. The former states API compatibility and the latter states API breakages. Which one is correct?

Sorry for the confusion. The API compatibility claim of a minor release stated in the introduction section is based on the strict definition from semantic versioning. The "best-effort" API compatibility claim in the proposal section is our pragmatic interpretation given the large API surface area of Eclipse OMR, i.e. we will do our best to maintain any well-defined C API for any Eclipse OMR component if applicable. So, changing such well-defined C API will only happen during a major release, but changes to anything below the C API can happen during a minor release.

Other than making sure one clones the precise SHA commit will there be a way for an end user to determine what version of OMR they're consuming? i.e. should the version information be encoded somewhere in the repository at the time of branching?

I think that's a great idea.

[mstoodle]

Note that "builds" only makes sense for some components and not for others: things like port and thread and jitbuilder can be built as binaries and archived, while other components can't really be "built" as independent libraries (compiler, for sure...not sure what the current state of the GC component and whether it needs to be built together with its glue @charliegracie ?).

But given builds do make sense for some components, I think having milestone "builds" is reasonable. It could also be faster for dependent projects to just grab a milestone source code zip rather than having to clone the entire repository.

[mgaudet]

A couple points:

• There's mention above of C-API stability: Does this mean the C++ ABI will be free to evolve outside the release process?
• Given the definition conflicts about major/minor above, why not do a major release every three months? Or is there an idea here that a 'major' release will be predicated on some milesone (i.e. API stability)?

[Leonardo2718]

A couple points from me as well:

First, regarding the "best-effort" basis for API compatibility in minor releases: is this something that is expected to last, or is the plan to change this once the API becomes more stable? In the latter case, I think it should be stated explicitly. Also, I would like highlight that point 4 in the semantic versioning specification states (emphasis mine):

Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable.

This effectively allows API incompatible changes to happen across both minor and patch releases.

If the initial major release of OMR is 0.y.z, I don't see a need explicitly state that API compatibility across minor releases is on a "best-effort" basis. Instead, we can document which APIs are stable in each minor (and patch) release to account for the different levels of API stability among the different components in OMR. Then, once all APIs are acceptably stable, we can create release 1.0.0.

Second, I am glad to see testing requirements mentioned. However, I would like these to be further refined. Tests can be used to "document" APIs and to measure the stability of APIs. So, defining with a greater level of detail which tests should run for a release can help document what APIs are stable in that release. This would likely have to be done on an "per-release" basis, but I imagine it would part of the release review. In any case, having the process documented will definitely help.

[xliang6]

Thanks @mgaudet and @Leonardo2718 for the comments! Both of you have brought up some valid points.

There's mention above of C-API stability: Does this mean the C++ ABI will be free to evolve outside the release process? Given the definition conflicts about major/minor above, why not do a major release every three months? Or is there an idea here that a 'major' release will be predicated on some milestone (i.e. API stability)?

When you say C++ ABI, do you mean C/C++ compiler version compatibility or changing OMR data structure fields or their order? In general, API and ABI compatibility guarantees can only be made on a patch release with its main release for Eclipse OMR. I understand and appreciate the confusion you have about minor/major releases and API/ABI compatibility guarantees or lack thereof for these releases.

As @Leonardo2718 has pointed out that semantic versioning specification states the following, I would suggest that what is discussed in this proposed Eclipse OMR release process document essentially contains release guidelines for OMR releases < 1.0.0.

Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable. Version 1.0.0 defines the public API. The way in which the version number is incremented after this release is dependent on this public API and how it changes.

First, regarding the "best-effort" basis for API compatibility in minor releases: is this something that is expected to last, or is the plan to change this once the API becomes more stable?

Some of the OMR components can work towards a more stable API while others will not be able to provide such API even after 1.0.0. Therefore, after the 1.0.0 release, we might have to bump the major version number (and hence a major release) say every three months to be consistent with semantic versioning definitions. However, I would like to emphasize that our current goal is to work towards a 1.0.0 release for OMR by starting to create monthly milestone builds and minor releases every three months first.

Second, I am glad to see testing requirements mentioned.

We will refine testing requirements (and update this OMR release document) as we go through a few OMR release cycles and figure out what constitutes a sufficient test suite for a given type of release.

Lastly, I want to point out that this document will evolve over time. We will update it as we go through a couple release cycles and learn about what works and what doesn't. Thanks again for your time and feedback!

[mgaudet]

When I said ABI -- that was a typo (Fingers type ABI much easier than API): I meant C++ API.

[mstoodle]

@DanHeidinga @pshipton @jonathanwoodford any comments from an OpenJ9 perspective?

[xliang6]

Given this is a fairly important topic for the Eclipse OMR project, a timeline of 2 weeks has been set for any discussions to take place. We will review and finalize this Eclipse OMR release process document based on the community’s feedback on February 15th, 2018.

A friendly reminder: The deadline to voice your opinions is in a couple of days. Please let us know what you think soon! Thanks.

[xliang6]

Thank you everyone for spending the time to read and provide feedback on the Eclipse OMR release process proposal. The general consensus is that we can start creating Eclipse OMR releases by following what was proposed in the document till we reach a 1.0.0 release for OMR. At that time, we will revisit and modify the process as needed.

If there are no objections, I will start working on a more concrete plan for the monthly milestone builds. Thanks.

[charliegracie]

Adding a +1 for moving forward with this proposal

[0xdaryl]

+1

[xliang6]

Call for feedback on this proposal is now closed. Please stay tuned for the first Eclipse OMR monthly milestone build soon. Thanks.

[abhi18av]

This seems related https://github.com/eclipse/omr/issues/3309

[mstoodle]

Reopening so we can get this release back on track; we seem to have lost track of our intentions here.

I believe this issue documents that a long time ago we were all in agreement to start doing OMR releases on basically a quarterly basis. Today, I updated the 0.1.0 OMR release paperwork to reset the target date to an extremely arbitrarily chosen "October 4, 2019" to 1) give us some time to discuss it, and 2) give us some time to execute on it (since it's our first release, there will be a lot of accumulated content (e.g. IP log) to deal with. I don't think we need to lock our release cadence or dates in at this time so please don't assume more weight to the choice of "Oct 4" than it deserves: I was just trying to set a not-too-distant straw person date for us to shoot for. If there are good reasons to change it, let's change it. If follow-on releases should fall on different dates, we also shouldn't feel obligated to do the next release precisely 3 months from Oct 4.

I have not submitted the release for review yet. I wanted to make sure the project committers and community are still on board with the approach and that we have rough agreement on the date. You can view the 0.1.0 release information here. Many thanks are overdue to @xliang6, both for starting and driving this issue in the past and for providing most the original content in the actual release documents (I have since tweaked them a bit so I'll take "credit" for any mistakes therein).

Given we've already discussed this topic once before and there will still be a formal 2-week review period once Charlie or I press the button on the release, let's try to surface and address any concerns with the date in the next week.

I would like to set a goal to submit the release for formal review by Aug 30 if possible. Let's get this ball rolling!

@charliegracie @0xdaryl @xliang6 @vijaysun-omr @youngar @Leonardo2718 @fjeremic @rwy0717 @rwy0717 @andrewcraik

[dibyendumajumdar]

Hi, What would be nice if the release could be synced with the version tested with OpenJ9, as I understand OpenJ9 goes through extensive testing. This could be as simple as selecting a commit that was the last OpenJ9 tested version. From a downstream project point of view: a) Having OpenJ9 synced releases would give some assurance of what testing has been done, without (hopefully) adding additional burden to OMR. b) It would be nice to have a list of major features added in each release. c) I don't think api stability matters at this stage.

My suggestion is please keep this as informal and low touch as possible until you are ready for api stability.

[DanHeidinga]

My suggestion is please keep this as informal and low touch as possible until you are ready for api stability.

+1 to that!

@rwy0717 and I were talking about OMR releases on Friday. In addition to working towards API stability, it would also be good to set release goals related to moving the rest of the OpenJ9-specific code out of OMR to help both projects iterate faster.

[mstoodle]

Hi @dibyendumajumdar

Good suggestions, thank you, though I find it a little bit odd to have an upstream project defining its release in terms of a downstream project. I'd like to flip that around, but ensure that OpenJ9 and other downstream projects have enough opportunity to try the new release and to report any problems. In the case of OpenJ9, that's actually pretty easy because OpenJ9 pulls OMR changes almost as they occur and includes them in their regular testing. Any problems they discover that need to be fixed are contributed directly to the OMR project. Right now, OpenJ9 is selecting "arbitrary" commits on which to base their releases. I think it would be better if we can establish a framework where they can choose an OMR release on which to base their releases (even if they have to add a few commits temporarily in their project due to unanticipated late discoveries). Another way to say it: I am hoping the OMR project can leverage the OpenJ9 release testing project as part of the OMR release testing process. That will get harder to accommodate if we have a wide variety of projects, but I guess that's the benefit of getting in early :) . Do you have any thoughts around nj releases that we can try to factor in?

Remember, this is a plan for a 0.1 release. Because it's the "first" release we would need to summarize the major work that's happened over the last 3 years of being open source (yeah, we're late to this "releasing" party :( ). So my thinking was to simply make the 0.1 release a "baseline" for future releases. Once we have that baseline, I think it makes a lot more sense to report major development items as they show up in releases. The full set of changes will be captured in an IP log that we will collect as part of the Eclipse project release process.

The idea behind "API stability" is simply to identify the components that we believe have API stability. It felt reasonable to document where we (the committers) think API stability stands at the moment (again, as a baseline). Over time that chart will change, but we don't intend to make API stability the only focus of releases going forward. I do think the thought process around "what would be required to achieve API stability for component X" may spark some interesting conversations around what we should be doing to go forward in some of our components.

[mstoodle]

@DanHeidinga @rwy0717 I agree completely that this topic deserves attention, but I'm not sure it needs to be spelled out specifically in the 0.1.0 release, unless I'm misunderstanding your suggestion.

[mstoodle]

(sorry, accidental close there, not quite sure what happened!)

[dibyendumajumdar]

Hi,

Good suggestions, thank you, though I find it a little bit odd to have an upstream project defining its release in terms of a downstream project. I'd like to flip that around, but ensure that OpenJ9 and other downstream projects have enough opportunity to try the new release and to report any problems. In the case of OpenJ9, that's actually pretty easy because OpenJ9 pulls OMR changes almost as they occur and includes them in their regular testing. Any problems they discover that need to be fixed are contributed directly to the OMR project. Right now, OpenJ9 is selecting "arbitrary" commits on which to base their releases. I think it would be better if we can establish a framework where they can choose an OMR release on which to base their releases (even if they have to add a few commits temporarily in their project due to unanticipated late discoveries). Another way to say it: I am hoping the OMR project can leverage the OpenJ9 release testing project as part of the OMR release testing process. That will get harder to accommodate if we have a wide variety of projects, but I guess that's the benefit of getting in early :) . Do you have any thoughts around nj releases that we can try to factor in?

I don't have anything specific regarding nj releases but I can merge OMR releases and report back if there are any issues. I am currently merging every 3 months or so, but I am never sure whether I might merge some bad change or other, and not realise it, as my tests don't cover all aspects of OMR.

Remember, this is a plan for a 0.1 release. Because it's the "first" release we would need to summarize the major work that's happened over the last 3 years of being open source (yeah, we're late to this "releasing" party :( ). So my thinking was to simply make the 0.1 release a "baseline" for future releases. Once we have that baseline, I think it makes a lot more sense to report major development items as they show up in releases. The full set of changes will be captured in an IP log that we will collect as part of the Eclipse project release process.

The idea behind "API stability" is simply to identify the components that we believe have API stability. It felt reasonable to document where we (the committers) think API stability stands at the moment (again, as a baseline). Over time that chart will change, but we don't intend to make API stability the only focus of releases going forward. I do think the thought process around "what would be required to achieve API stability for component X" may spark some interesting conversations around what we should be doing to go forward in some of our components.

Okay, I just think that to get started you can just cut a release without having to document all this ... the less hurdle the more likelihood a release will actually be cut ;-)

[mstoodle]

Okay, I just think that to get started you can just cut a release without having to document all this ... the less hurdle the more likelihood a release will actually be cut ;-)

Yep, I'm all for less work required. But the table has already been created, so we may as well use it at this point :) .

Thanks for participating in the discussion!

[Leonardo2718]

@mstoodle I think it's been over a week since the last comment was posted. Shall we move forward with this? :slightly_smiling_face:

[mstoodle]

Sorry, I was off for the weekend and only got back to the office today.

I have, just now, submitted a 0.1.0 release for review. It is still "pending" so I don't think the review can be seen publically yet, but hopefully will be quickly approved for review. I submitted for a review ending Sep 18 (so public comments should be accepted up until that day). I left the release date as Oct 4 since there was no additional input. I have also submitted the IP log for the release with a target date of Sep 11 (it is supposed to be completed a week before the review date). It is in the Eclipse IP team's hands now.

I added some details to the release that weren't there already. Once the release is publically available, please feel free to comment on any aspect of the release. Thanks to everyone for their input so far!

[mstoodle]

The release review is publically viewable here: https://projects.eclipse.org/projects/technology.omr/reviews/0.1.0-release-review (though there is still a comment suggesting it is still "pending" )

Please provide any feedback you think will help make the release better!