Support versionless Jakarta EE/MicroProfile features

[NottyCode]

Description

While the Liberty feature design allows many great features like zero migration and us to support multiple EE versions it does make some things harder. For example if you are not using one of the convenience features and you want to be able to easily move from one EE/MP version to another you have to work out the right combination of new feature versions to specify them.

Proposed is an alternative option. We would allow unversioned EE features to be specified, but in combination another feature must be provided to indicate the platform version required. If no platform versioning feature is provided then there would be a startup failure to ensure we don't end up with zero migration breaks. An example of what this might look like is:

<featureManager>
   <feature>mpHealth</feature>
   <feature>mpRestClient</feature>
   <feature>jsonb</feature>
   <feature>mpPlatform-6.0</feature>
</featureManager>

or

<featureManager>
   <feature>persistence</feature>
   <feature>restfulWS</feature>
   <feature>jakartaeePlatform-10.0</feature>
</featureManager>

As the number of features required grows the value of this increases. It also makes it simpler for operations teams to augment developer provided configurations. For example when deploying into kube you might wish to have a health check capability inserted when a container image is built, but it isn't required by the application. This would enable the right mpHealth to be injected at container build time without needing to worry about what version of Jakarta EE/MicroProfile was used by the developer.

---

Documents

When available, add links to required feature documents. Use "N/A" to mark particular documents which are not required by the feature.

• Aha: Externally raised RFE (Aha)

  • Link the RFE with this issue

• UFO: Link to Upcoming Feature Overview document

  • https://ibm.box.com/s/5fk61j0mfbw4que6jgmmw3vizg3lb6qn (latest)

• FTS: Link to Feature Test Summary GH Issue

• Beta Blog: Link to Beta Blog Post GH Issue

• GA Blog: Link to GA Blog Post GH Issue

• ---

  Process Overview

• Prioritization

• Design

• Implementation

• Legal and Translation

• Beta

• GA

  • Focal Point Approvals

• Other Deliverables

General Instructions

The process steps occur roughly in the order as presented. Process steps occasionally overlap.

Each process step has a number of tasks which must be completed or must be marked as not applicable ("N/A").

Unless otherwise indicated, the tasks are the responsibility of the Feature Owner or a Delegate of the Feature Owner.

If you need assistance, reach out to the OpenLiberty/release-architect.

Important: Labels are used to trigger particular steps and must be added as indicated.

---

Prioritization (Complete Before Development Starts)

The (OpenLiberty/chief-architect) and area leads are responsible for prioritizing the features and determining which features are being actively worked on.

Prioritization

• [ ] Feature added to the "New" column of the Open Liberty project board

  • Epics can be added to the board in one of two ways:
  • From this issue, use the "Projects" section to select the appropriate project board.
  • From the appropriate project board click "Add card" and select your Feature Epic issue

• [ ] Priority assigned

  • Attend the Liberty Backlog Prioritization meeting

• ---

  Design (Complete Before Development Starts)

Design preliminaries determine whether a formal design, which will be provided by an Upcoming Feature Overview (UFO) document, must be created and reviewed. A formal design is required if the feature requires any of the following: UI, Serviceability, SVT, Performance testing, or non-trivial documentation/ID.

Design Preliminaries

• [ ] UI requirements identified. (Owner and UI focal point)
• [ ] ID requirements identified. (Owner and ID focal point)
  • Refer to Documenting Open Liberty.
  • Feature Owner adds label ID Required, if non-trivial documentation needs to be created by the ID team.
  • ID adds label ID Required - Trivial, if no design will be performed and only trivial ID updates are needed.
• [ ] Serviceability Requirements Identified. (Owner and Serviceability focal point)
• [ ] SVT Requirements Identified. (Owner and SVT focal point)
• [ ] Performance testing requirements identified. (Owner and Performance focal point)

Design

• [ ] POC Design / UFO review requested.
  • Owner adds label Design Review Request
• [ ] POC Design / UFO review scheduled.
  • Follow the instructions in POC-Forum repo
• [ ] POC Design / UFO review completed.
• [ ] POC / UFO Review follow-ons completed.
• [ ] POC Design / UFO approval requested.
  • Owner adds label Design Approval Request
• [ ] Design / UFO approved. (OpenLiberty/chief-architect) or N/A
  • (OpenLiberty/chief-architect) adds label Design Approved
  • Add the public link to the UFO in Box to the Documents section.
  • The UFO must always accurately reflect the final implementation of the feature. Any changes must be first approved. Afterwards, update the UFO by creating a copy of the original approved slide(s) at the end of the deck and prepend "OLD" to the title(s). A single updated copy of the slide(s) should take the original's place, and have its title(s) prepended with "UPDATED".

No Design

• [ ] No Design requested.
  • Owner adds label No Design Approval Request
• [ ] No Design / No UFO approved. (OpenLiberty/chief-architect) or N/A
  • Approver adds label No Design Approved
• [ ] Feature / Capability stabilization or discontinuation or N/A
  • Owner adds label Product Management Approval Request and notifies OpenLiberty/product-management
  • Approver adds label Product Management Approved (OpenLiberty/product-management)
  • Note: For stabilized, superseded, and discontinued feature/capability, skip the Beta section of the template (you may delete it). Otherwise, proceed as normal.

FAT Documentation

• [ ] "Feature Test Summary" child task created

  • Use the Feature Test Summary Template
  • Add FTS issue link to the Documents section.

• ---

  Implementation

A feature must be prioritized before any implementation work may begin to be delivered (inaccessible/no-ship). However, a design focused approach should still be applied to features, and developers should think about the feature design prior to writing and delivering any code.
Besides being prioritized, a feature must also be socialized (or No Design Approved) before any beta code may be delivered. All new Liberty content must be inaccessible in our GA releases until it is Feature Complete by either marking it kind=noship or beta fencing it.
Code may not GA until this feature has obtained the "Design Approved" or "No Design Approved" label, along with all other tasks outlined in the GA section.

Feature Development Begins

• [ ] Add the In Progress label

Legal and Translation

In order to avoid last minute blockers and significant disruptions to the feature, the legal items need to be done as early in the feature process as possible, either in design or as early into the development as possible. Similarly, translation is to be done concurrently with development. Both MUST be completed before Beta or GA is requested.

Legal (Complete before Feature Complete Date)

• [ ] Changed or new open source libraries are cleared and approved, or N/A. (Legal Release Services/Cass Tucker/Release PM).
• [ ] Licenses and Certificates of Originality (COOs) are updated, or N/A

Translation (Complete 1 week before Feature Complete Date)

• [ ] PII updates are merged, or N/A. Note timing with translation shipments.

Innovation (Complete 1 week before Feature Complete Date)

• [ ] Consider whether any aspects of the feature may be patentable. If any identified, disclosures have been submitted.

• ---

  Beta

In order to facilitate early feedback from users, all new features and functionality should first be released as part of a beta release.

Beta Code

• [ ] Beta fence the functionality
  • kind=beta, ibm:beta, ProductInfo.getBetaEdition()
• [ ] Beta development complete and feature ready for inclusion in a beta release
  • Add label target:beta and the appropriate target:YY00X-beta (where YY00X is the targeted beta version).
• [ ] Feature delivered into beta
  • (OpenLiberty/release-manager) adds label release:YY00X-beta (where YY00X is the first beta version that included the functionality).

Beta Blog (Complete 1.5 weeks before beta eGA)

• [ ] Beta blog issue created and populated using the Open Liberty BETA blog post template.

  • Add a link to the beta blog issue in the Documents section.
  • Note: This is for inclusion into the overall beta release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

• ---

  GA

A feature is ready to GA after it is Feature Complete and has obtained all necessary Focal Point Approvals.

Feature Complete

• [ ] Feature implementation and tests completed.
  • [ ] All PRs are merged.
  • [ ] All epic and child issues are closed.
  • [ ] All stop ship issues are completed.
• [ ] Legal: all necessary approvals granted.
• [ ] Translation: All messages translated or sent for translation for upcoming release
• [ ] GA development complete and feature ready for inclusion in a GA release
  • Add label target:ga and the appropriate target:YY00X (where YY00X is the targeted GA version).
  • Inclusion in a release requires the completion of all Focal Point Approvals.

Focal Point Approvals (Complete by Feature Complete Date)

These occur only after GA of this feature is requested (by adding a target:ga label). GA of this feature may not occur until all approvals are obtained.

All Features

• [ ] APIs/Externals Externals have been reviewed or N/A. (OpenLiberty/externals-approvers)
  • Approver adds label focalApproved:externals
• [ ] Demo Demo is scheduled for an upcoming EOI or N/A. (OpenLiberty/demo-approvers)
  • Add comment @OpenLiberty/demo-approvers Demo scheduled for EOI [Iteration Number] to this issue.
  • Approver adds label focalApproved:demo.
• [ ] FAT All Tests complete and running successfully in SOE or N/A. (OpenLiberty/fat-approvers)
  • Approver adds label focalApproved:fat.
• [ ] Globalization Translation and TVT are complete or N/A. (OpenLiberty/globalization-approvers)
  • Approver adds label focalApproved:globalization.

Design Approved Features

• [ ] Accessibility Accessibility testing completed or N/A. (OpenLiberty/accessibility-approvers)
  • Approver adds label focalApproved:accessibility.
• [ ] ID Documentation is complete or N/A. (OpenLiberty/id-approvers)
  • Approver adds label focalApproved:id.

  • NOTE: If only trivial documentation changes are required, you may reach out to the ID Feature Focal to request a ID Required - Trivial label. Unlike features with regular ID requirement, those with ID Required - Trivial label do not have a hard requirement for a Design/UFO.

• [ ] Performance Performance testing is complete or N/A. (OpenLiberty/performance-approvers)
  • Approver adds label focalApproved:performance.
• [ ] Serviceability Serviceability has been addressed or N/A. (OpenLiberty/serviceability-approvers)
  • Approver adds label focalApproved:sve.
• [ ] STE Skills Transfer Education chart deck is complete or N/A. (OpenLiberty/ste-approvers)
  • Approver adds label focalApproved:ste.
• [ ] SVT System Verification Test is complete or N/A. (OpenLiberty/svt-approvers)
  • Approver adds label focalApproved:svt.

Remove Beta Fencing (Complete by Feature Complete Date)

• [ ] Beta guards are removed, or N/A
  • Only after all necessary Focal Point Approvals have been granted.

GA Blog (Complete by Feature Complete Date)

• [ ] GA Blog issue created and populated using the Open Liberty GA release blog post template.
  • Add a link to the GA Blog issue in the Documents section.
  • Note: This is for inclusion into the overall release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

Post GA

• [ ] Replace target:YY00X label with the appropriate release:YY00X. (OpenLiberty/release-manager)

• ---

  Other Deliverables

• [ ] Standalone Feature Blog Post A blog post specifically about your feature or N/A. (OpenLiberty/release-architect)

  • This should be strongly considered for larger or more prominent features.
  • Follow instructions in the blogs repo.

• [ ] OL Guides OL Guides assessment is complete or N/A. (OpenLiberty/guide-assessment)

• [ ] Dev Experience Developer Experience & Tools work is complete or N/A. (OpenLiberty/dev-experience-assessment)

• ---

[jhanders34]

https://github.com/OpenLiberty/open-liberty/compare/integration...jhanders34:versionless-features-prototype contains the prototype code that I referenced in our call today. It shows a versionless feature for the persistence / jpa features in Jakarta / Java EE.

Some points from the call today based off of this prototype:

• Presently the prototype uses jakartaPlatform public feature instead of jakartaeePlatform. The reason for this is that io.openliberty.jakartaeePlatform features already exist. They are private features. They would need to be renamed if we wanted to use jakartaeePlatform instead. I actually like jakartaPlatform since it is shorter. If you chose to use jakartaeePlatform, just renaming the existing one to add internal to the feature name.
• You will notice that there isn't a 6.0 Platform feature. That version of the feature would be in WebSphere Liberty. Similarly you will see that there is an unversioned.persistence-2.0 in the tolerates list, but I did not create that feature. It would exist in the WebSphere Liberty repo.
• I would recommend changing to javaeePlatform for 7.0 and 6.0 feature.
• I would commend changing from io.openliberty.jakartaPlatform.internal to io.openliberty.jeePlatform.internal. Then it easily covers both javaee and jakartaee.
• Presently there is a 0.0 version of the unversioned.persistence feature, but that is just so that if you don't specify a jakartaPlatform feature, the prototype will not have a preferred version that starts up an actual feature. In the end if jakartaPlatform, javaeePlatform or mpPlatform feature isn't specified the server should fail to start.
• WLP-Required-Feature: jakartaPlatform, javaeePlatform, mpPlatform in the persistence public feature is just an example of a property that could be in unversioned features to tell the FeatureManager that a required feature is needed for this feature and to fail if it isn't specified. For mp features, it would just list mpPlatform.
• Note that the persistence public feature is NOT a singleton. In this case that is what we want because the persistence feature will be enabled and the persistence-x.y would also be enabled. If the persistence feature was a singleton, a conflict would be triggered. SymbolicNameTest.testSingleton will need to be updated to handle versionless features not being singletons.
• Right now with this prototype both persistence and jpa-x.y / persistence-x.y are both listed in the started features. There could be a change in the FeatureManager to remove the unversioned feature (persistence) if we so chose.
• mpPlatform-x.y features will have a feature dependency on the appropriate jakartaPlatform.internal / jeePlatform.internal feature.
• It needs to be considered if we want to have a jpa. I would recommend not and update the persistence feature to have an attribute of WLP-AlsoKnownAs: jpa.
• This prototype is one way of doing this feature. It could be done other ways. Notably it could be done with auto features instead. One thing to consider with the prototype example is that when doing feature installation of a versionless feature, it would likely install all of the private tolerates to make sure that the feature would work if we installed new versions of jakartaPlatform / javaeePlatform. We could special case that in the repository logic that is used for feature installation. Something to probably discuss with Andrew Rouse. If we used auto features, then that wouldn't be a problem I believe.
• The prototype example doesn't include EE 11 features. When I created the prototype I don't think those had been created yet.
• In general, we should see if we can short circuit private feature changes if we know that a feature in the chain is already included in the list of features in the server.xml feature list. In this example jakartaPlatform / javaeePlatform would be listed in the server.xml which would bring in the jeePlatform.internal feature and that is what is referenced in the tolerated unversioned features so we can easily identify which of the chains matches the list of specified features and not need to test each chain when doing toleration. Only when we can't identify a specific chain based off of the list of features provided would we need to do some toleration permutations.
• internal should probably be added to the unversioned.persistence feature name. I chose to have unversioned in the name right after io.openliberty to easily be able to identify all of the unversioned private features because they will all be together.
• The prototype public feature do not include the required NLS files for the feature description since it is only a prototype. As such VisibilityTest fails currently with the prototype code.
• Thinking more your thought about using versioned features to also pick which feature to pick for versioned features, you could end up having eeCompatible features depend on the appropriate jeePlatform.internal feature, but as I stated in the call I don't think this is the right thing to do.

[rsherget]

I created a draft PR https://github.com/OpenLiberty/open-liberty/pull/26301 showcasing Jared's changes as well as unversioned servlet

[tbitonti]

• Complete UFO
• Open-Liberty:
  • Add Feature Definitions.
  • Beta guarding done by marking the new feature type as beta.
  • New feature resolver error messages
  • Platform selector used without any versionless features.
  • Redundant platform feature used with platform selector feature.
  • Versionless feature used without a platform selector.
  • Versionless feature not available at platform level.
  • Redundant feature used with versionless feature.
  • Update Feature Unit Tests.
  • Add Feature Resolution Unit Tests?
  • Update Feature Resolution FAT Tests.
  • Expected: Feature Introspector impacts.
  • Expected: Minify impacts.
  • Expected: Feature reporting impacts.
  • Expected: Documentation impacts.
  • Expected: Migration impacts (documentation).
  • Beta BLOG and BLOG.
• WebSphere Liberty:
  • Update Feature Definitions
  • Update Feature Unit Tests
  • Add Feature Resolution Unit Tests?
  • Update Feature Resolution FAT Tests

[ayoho]

UFO review (part 1) notes:

Slide 4

• "selector features": perhaps add minor clarification here if feasible? Some ensuing discussion about where to clarify this and maybe being a bit limited by the order of the slides.

Slide 6

• Call out that Jakarta and MicroProfile are the only two platform selectors we plan on implementing to start.

Slide 9

• Fix typo in the "jakartaPlatform-10" feature names - should be "jakartaPlatform-10.0".

Slide 12

• TBD: Almost exactly halfway through the call, something was discussed needing clarification. Will need to check the recording.

Slide 13

• Perhaps create a <platform> or <featurePlatform> element instead of re-using <feature>
• Fix "jakarata" typos.

Slide 17

• How should we document the highest/lowest selection policy? How does an end user know which policy is being used?

Slide 18

• Should use the same selector name as was used earlier in the UFO (e.g. "jakarta-10.0" -> "jakartaPlatform-10.0").
• Still need more discussion about how to handle the environment variable and what to do in that scenario.

Slide 20

• Should maintain "javaee" for javaee selector features; do not reuse "jakarta" for both.
• Need to have good logging in case they say, for example, "jsp" instead of "pages".

[scottkurz]

Re: slide 13:

Perhaps create a <platform> or <featurePlatform> element instead of re-using <feature>

I think it should be considered to add an attribute on featureManager as an alternative..

E.g. <featureManager jakartaPlatform="9.1" microprofilePlatform="5.0">.

The fact that there should only be one of each of these can be captured more efficiently by the basic rules for XML without having to be performed by Liberty-specific validation.

[ayoho]

UFO review (part 2) notes:

Slide 4:

• Plan should probably include how to document or what approach to take for aberrant feature names.
• Clear up and specify what all "not from the command line" covers (WDT?)

Slide 23:

• When using the environment variable, how would that account for adding versionless features for features not part of a programming model?

Slide 36:

• Clarify that installing based off server.xml works, installing by named feature does not work.
• A lot of discussion about what should happen for MVP and beyond for something like the server.xml only having a versionless feature like mpHealth.
• Packaging commands need to account for versionless features as well.

[dlp01]

@tbitonti and I were chatting in the halls of the RTP lab today. As identified in the POC Forum, there is a possibility of multiple possibilities for non-versioned features and that the server start will fail in that situation. The user can avoid this situation by using environment variable(s) that give the preferred list of versioned features to be used when 1) there are multiple possible potential choices for non-versioned features, and 2) the specific versioned features specified don't narrow the choice to a specific version.

I don't have issues with this concept, but do have some requests for the developer experience and use experience that will allow the user to avoid waiting on the server start to discover the error condition. I'll record these here and tag some folks in hopes they'll be able to add details/ideas, or map these to requirements.

1) I think there should be something available in the developer tools (i.e. with auto-completion) that helps the user know that they've set up this condition. Tom was suggesting, as well, that the developer tools could even suggest that the user either set a specific feature level, or set up the environment variable list. @cherylking @scottkurz

2) It would be nice to have some sort of validation, in general, in support of discovering this condition prior to server start. I don't know if that should be a part of the existing set of script. @NottyCode

[tjwatson]

UFO review notes (part 3)

Slide 3

• Use jakartaee-10.0 for the selector name (not jakartaPlatform-1.0)

Slide 17

• By smashing both platforms into the same attribute, you make it impossible to merge two separate elements, e.g. one in the main server.xml and one in a separate dropin. Note that tooling generally uses configDropins. Resolution: Need to make sure we merge the platforms from multiple sources of server.xml (e.g. configDropins).

Slide 18

• If ambiguous feature cannot be determined then act like a conflict and don't install any features
• Discussion around requiring PREFERRED_FEATURE_VERSION setting for cross platform versionless features (e.g. servlet-5.0, mpHealth). Conclusion: must either specify the platform selector or use PREFERRED_FEATURE_VERSION to break current and possible future ambiguity when versionless features cross a platform boundary. Otherwise we risk breaking zero migration.
• Note that ambiguity within the same platform then the lowest version must be selected, no need to specify the preferred version.

Slide 25

• The env var will be included if it is needed.
• What happens if the environment is set in the environment and not the server.env. Should packaging append the env if needed to the archive? More discussion needed here
• For minify we only include the feature versions that matched (just like other features).

[tjwatson]

Slide 49 for InstantOn: There is an InstantOn concern around the use of env values to configure the feature manager with PREFERRED_FEATURE_VERSIONS. The Semeru runtime returns null for most all environment variables except ones specified with WLP_IMMUTABLE_VARS before checkpoint https://github.com/OpenLiberty/open-liberty/blob/e2e768871ba4095a2ed128c6e05807f8354d3425/dev/com.ibm.ws.kernel.boot.ws-server/publish/bin/server#L95

My initial thought is we will want to add WLP_IMMUTABLE_VARS to the list of defaults so users do not have to set it themselves. That is done at https://github.com/OpenLiberty/open-liberty/blob/e2e768871ba4095a2ed128c6e05807f8354d3425/dev/com.ibm.ws.kernel.boot.ws-server/publish/bin/server#L905-L906

[tjwatson]

UFO review notes (part 4)

Slide 19

• [ ] Should we add meta-data that identifies the feature as belonging to a specific platform?
• [ ] This can be used to auto-generate documentation

Slide 26

• [ ] Clarify that if it is not set in the server.env then the variable must be specified when running the minified server
• [ ] Read server.env to verify if it is set or not when the variable value is found. Warn if the variable is not spec'ed in the server.env

Slide 36

• [ ] Note that info from 27 need to be put on command line page and installUtility will not support installing versionless features.

Slide 27

• [ ] Need a tools epic/issue to address tolerate or add support for versionless features.

Slide 31

• [ ] Talk about updating guides to use platform selector and versionless features

Slide 37

• [ ] Update the Jakarta Starter to suggest platform selector and versionless features

Slide 42

• [ ] platform config element is marked internal/beta until GA

Slide 43

• [ ] Exist rules for repeat on EE/MP platforms should transform the <platform/> element

Slide 46

• [ ] Packaging is often discussed here, are versionless features are included in core/base/nd? Versionless should not be in the platform specific packages e.g. Jakarta 8,9,10 etc.

Slide 48

• [ ] Need more detail on messages for serviceability focal approval.
• [ ] Should the feature template be updated for versionless features

General comment:

• [ ] Need to cache the platform selector values from one start to the next to validate that it didn't change

[cbridgha]

All comments from above have been resolved and changed in the UFO document: https://ibm.ent.box.com/file/1484909310089?s=5fk61j0mfbw4que6jgmmw3vizg3lb6qn

[hlhoots]

Beta BLOG issue is #28656