Configuration options to change the MP OpenAPI endpoints

[Azquelt]

Description

Add configuration options to the MicroProfile OpenAPI features which allow the user to change:

• the endpoint which serves the OpenAPI yaml/json document (default /openapi)
• the endpoint which serves the Swagger UI interface for browsing the documentation (default /openapi/ui)

For example, we could add MP Config properties like this:

mp.openapi.extensions.liberty.path=/openapi
mp.openapi.extensions.liberty.ui.path=/openapi/ui

One scenario where this feature is important is where a user has several services on different servers behind a Kubernetes or Openshift ingress point, with a different path mapped to each service. If the endpoints are not movable, it's impossible to expose the UI served by different servers from the same ingress point.

An important consideration is that the UI is a javascript application which runs in the browser and it needs to be able to find and call the endpoint which serves the OpenAPI yaml document. If that endpoint is configurable, we need to pass through its location when serving up the UI.

---

Documents

When available, add links to required feature documents. Use "N/A" to mark particular documents which are not required by the feature.

• Enhancement request: #17573

• UFO: https://ibm.box.com/s/i05uidcj09hhqe848io9wnpq0xo7l9ju

• FTS: #25071

• Beta Blog: #26222

• GA Blog: #27046

• ---

  Process Overview

• Prioritization

• Design

• Implementation

• Legal and Translation

• Beta

• GA

  • Focal Point Approvals

• Other Deliverables

General Instructions

The process steps occur roughly in the order as presented. Process steps occasionally overlap.

Each process step has a number of tasks which must be completed or must be marked as not applicable ("N/A").

Unless otherwise indicated, the tasks are the responsibility of the Feature Owner or a Delegate of the Feature Owner.

If you need assistance, reach out to the OpenLiberty/release-architect.

Important: Labels are used to trigger particular steps and must be added as indicated.

---

Prioritization (Complete Before Development Starts)

The (OpenLiberty/chief-architect) and area leads are responsible for prioritizing the features and determining which features are being actively worked on.

Prioritization

• [x] Feature added to the "New" column of the Open Liberty project board

  • Epics can be added to the board in one of two ways:
  • From this issue, use the "Projects" section to select the appropriate project board.
  • From the appropriate project board click "Add card" and select your Feature Epic issue

• [x] Priority assigned

  • Attend the Liberty Backlog Prioritization meeting

• ---

  Design (Complete Before Development Starts)

Design preliminaries determine whether a formal design, which will be provided by an Upcoming Feature Overview (UFO) document, must be created and reviewed. A formal design is required if the feature requires any of the following: UI, Serviceability, SVT, Performance testing, or non-trivial documentation/ID.

Design Preliminaries

• [ ] UI requirements identified. (Owner and UI focal point)
• [x] ID requirements identified. (Owner and ID focal point)
  • Refer to Documenting Open Liberty.
  • Feature Owner adds label ID Required, if non-trivial documentation needs to be created by the ID team.
  • ID adds label ID Required - Trivial, if no design will be performed and only trivial ID updates are needed.
• [x] Serviceability Requirements Identified. (Owner and Serviceability focal point)
• [x] SVT Requirements Identified. (Owner and SVT focal point)
• [x] Performance testing requirements identified. (Owner and Performance focal point)

Design

• [x] POC Design / UFO review requested.
  • Owner adds label Design Review Request
• [x] POC Design / UFO review scheduled.
  • Follow the instructions in POC-Forum repo
• [x] POC Design / UFO review completed.
• [x] POC / UFO Review follow-ons completed.
• [x] POC Design / UFO approval requested.
  • Owner adds label Design Approval Request
• [x] Design / UFO approved. (OpenLiberty/chief-architect) or N/A
  • (OpenLiberty/chief-architect) adds label Design Approved
  • Add the public link to the UFO in Box to the Documents section.
  • The UFO must always accurately reflect the final implementation of the feature. Any changes must be first approved. Afterwards, update the UFO by creating a copy of the original approved slide(s) at the end of the deck and prepend "OLD" to the title(s). A single updated copy of the slide(s) should take the original's place, and have its title(s) prepended with "UPDATED".

No Design

• [ ] No Design requested.
  • Owner adds label No Design Approval Request
• [ ] No Design / No UFO approved. (OpenLiberty/chief-architect) or N/A
  • Approver adds label No Design Approved
• [ ] Feature / Capability stabilization or discontinuation or N/A
  • Owner adds label Product Management Approval Request and notifies OpenLiberty/product-management
  • Approver adds label Product Management Approved (OpenLiberty/product-management)
  • Note: For stabilized, superseded, and discontinued feature/capability, skip the Beta section of the template (you may delete it). Otherwise, proceed as normal.

FAT Documentation

• [x] "Feature Test Summary" child task created

  • Use the Feature Test Summary Template
  • Add FTS issue link to the Documents section.

• ---

  Implementation

A feature must be prioritized before any implementation work may begin to be delivered (inaccessible/no-ship). However, a design focused approach should still be applied to features, and developers should think about the feature design prior to writing and delivering any code.
Besides being prioritized, a feature must also be socialized (or No Design Approved) before any beta code may be delivered. All new Liberty content must be inaccessible in our GA releases until it is Feature Complete by either marking it kind=noship or beta fencing it.
Code may not GA until this feature has obtained the "Design Approved" or "No Design Approved" label, along with all other tasks outlined in the GA section.

Feature Development Begins

• [x] Add the In Progress label
• [x] #25072
• [x] #25073
• [x] #25074
• [x] #25075
• [x] #25076
• [x] #25367
• [x] #25375
• [x] #25559
• [x] #26966

Legal and Translation

In order to avoid last minute blockers and significant disruptions to the feature, the legal items need to be done as early in the feature process as possible, either in design or as early into the development as possible. Similarly, translation is to be done concurrently with development. Both MUST be completed before Beta or GA is requested.

Legal (Complete before Feature Complete Date)

• [x] Changed or new open source libraries are cleared and approved, or N/A. (Legal Release Services/Cass Tucker/Release PM).
• [x] Licenses and Certificates of Originality (COOs) are updated, or N/A

Translation (Complete 1 week before Feature Complete Date)

• [ ] PII updates are merged, or N/A. Note timing with translation shipments.

Innovation (Complete 1 week before Feature Complete Date)

• [x] Consider whether any aspects of the feature may be patentable. If any identified, disclosures have been submitted.

• ---

  Beta

In order to facilitate early feedback from users, all new features and functionality should first be released as part of a beta release.

Beta Code

• [x] Beta fence the functionality
  • kind=beta, ibm:beta, ProductInfo.getBetaEdition()
• [x] Beta development complete and feature ready for inclusion in a beta release
  • Add label target:beta and the appropriate target:YY00X-beta (where YY00X is the targeted beta version).
• [x] Feature delivered into beta
  • (OpenLiberty/release-manager) adds label release:YY00X-beta (where YY00X is the first beta version that included the functionality).

Beta Blog (Complete 1.5 weeks before beta eGA)

• [x] Beta blog issue created and populated using the Open Liberty BETA blog post template.

  • Add a link to the beta blog issue in the Documents section.
  • Note: This is for inclusion into the overall beta release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

• ---

  GA

A feature is ready to GA after it is Feature Complete and has obtained all necessary Focal Point Approvals.

Feature Complete

• [x] Feature implementation and tests completed.
  • [x] All PRs are merged.
  • [x] All epic and child issues are closed.
  • [x] All stop ship issues are completed.
• [x] Legal: all necessary approvals granted.
• [x] Translation: Feature may only proceed to GA if it has either Translation - Complete or Translation - Missing label
  • If all translation has been delivered to release branch, feature owner adds label Translation - Complete.
  • If missing translation does not cause a break in functionality, nor a security or production outage risk, feature owner adds label Translation - Missing.
  • Once all missing translations are delivered, the Translation - Missing label is replaced with Translation - Complete.
  • If missing translation could cause a break in functionality or a security or production outage risk, feature owner adds the Translation - Blocked label.
  • Featues with Translation - Blocked may NOT proceed to GA until the label has been replaced with either Translation - Missing or Translation - Complete.
  • For further guidance, contact Globalization focal point or the Release Architect.
• [x] GA development complete and feature ready for inclusion in a GA release
  • Add label target:ga and the appropriate target:YY00X (where YY00X is the targeted GA version).
  • Inclusion in a release requires the completion of all Focal Point Approvals.

Focal Point Approvals (Complete by Feature Complete Date)

These occur only after GA of this feature is requested (by adding a target:ga label). GA of this feature may not occur until all approvals are obtained.

All Features

• [x] APIs/Externals - Externals have been reviewed or N/A. (OpenLiberty/externals-approvers)
  • Approver adds label focalApproved:externals
• [x] Demo - Demo is scheduled for an upcoming EOI or N/A. (OpenLiberty/demo-approvers)
  • Add comment @OpenLiberty/demo-approvers Demo scheduled for EOI [Iteration Number] to this issue.
  • Approver adds label focalApproved:demo.
• [x] FAT - All Tests complete and running successfully in SOE or N/A. (OpenLiberty/fat-approvers)
  • Approver adds label focalApproved:fat.

Design Approved Features

• [x] ID - Documentation is complete or N/A. (OpenLiberty/id-approvers)
  • Approver adds label focalApproved:id.

  • NOTE: If only trivial documentation changes are required, you may reach out to the ID Feature Focal to request a ID Required - Trivial label. Unlike features with regular ID requirement, those with ID Required - Trivial label do not have a hard requirement for a Design/UFO.

• [x] InstantOn - InstantOn capable or N/A. (OpenLiberty/instantOn-approvers)
  • Approver adds label focalApproved:instantOn.
• [x] Performance - Performance testing is complete or N/A. (OpenLiberty/performance-approvers)
  • Approver adds label focalApproved:performance.
• [x] Serviceability - Serviceability has been addressed or N/A. (OpenLiberty/serviceability-approvers)
  • Approver adds label focalApproved:sve.
• [x] STE - Skills Transfer Education chart deck is complete or N/A. (OpenLiberty/ste-approvers)
  • Approver adds label focalApproved:ste.
• [x] SVT - System Verification Test is complete or N/A. (OpenLiberty/svt-approvers)
  • Approver adds label focalApproved:svt.

Remove Beta Fencing (Complete by Feature Complete Date)

• [x] Beta guards are removed, or N/A
  • Only after all necessary Focal Point Approvals have been granted.

GA Blog (Complete by Friday after GM)

• [x] GA Blog issue created and populated using the Open Liberty GA release blog post template.
  • Add a link to the GA Blog issue in the Documents section.
  • Note: This is for inclusion into the overall release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

Post GA

• [x] Replace target:YY00X label with the appropriate release:YY00X. (OpenLiberty/release-manager)

• ---

  Other Deliverables

• [ ] Standalone Feature Blog Post - A blog post specifically about your feature or N/A. (Feature owner and OpenLiberty/release-architect)

  • This should be strongly considered for larger or more prominent features.
  • Follow instructions in the blogs repo.

• [ ] OL Guides - OL Guides assessment is complete or N/A. (OpenLiberty/guide-assessment)

• [ ] Dev Experience - Developer Experience & Tools work is complete or N/A. (OpenLiberty/dev-experience-assessment)

• ---

[malincoln]

@tevans78 asked that this feature replace https://github.com/OpenLiberty/open-liberty/issues/17573 on the roadmap board.

[scottkurz]

Notes from UFO review on 2023-05-19:

Design Issue

The biggest action item needed is to open a design issue regarding the use of MP Config Properties for configuration that logically applies to the server as a whole, rather than an individual app.

On the call today it was mentioned there is already a precedent for using this type of config today. When these existing particular server-wide properties are found in an "application-level" config source (this would be the microprofile-config.properties file as well as the appProperties element in server.xml as documented) the properties are simply ignored.

We should consider whether to continue with this pattern, or move away from it to some more Liberty-centric config solution. Whatever we decide should apply to this current epic, naturally.

Other comments

• Slide 18 - Overview - include a visual of the config parameter matching the custom/location in the example
• slide 28 - DevEx tools - Liberty-specific MP config props Though this is a nice idea, this would require enhancements in a couple directions: e.g. having the tooling be aware of Liberty-specific extension properties (rather than just MP-spec'd properties) as well as Liberty-specific config sources (e.g. server.env). The design issue above would also be relevant (it might be nice to warn about server MP props in "app" sources). Will open up an issue for internal discussion for the future, but for the scope of this epic this is probably therefore N/A.
• slide 35, 37 - write specific SVT test scenario from k8s/OpenShift-ingress perspective. (Did we say some doc too? Or maybe that doesn't need to be called out in addition).
• slide 37 - There was a concern raised that the text: ... documentation to be exposed through a single ingress path. was overloading the word "path" at the end... and that maybe something like "flow" would be better? (I didn't totally get this point).
• slide 39 (serviceability) - consider what to do when a conflict is detected with another ctx root
• performance - check performance on resume after a checkpoint using InstantOn

Doc updates

• Check whether AdminCenter links to the OpenAPI documentation - if so it will need to update based on this config
• There are other features which have APIs for which we provide OpenAPI documentation - how would this feature impact them?

[Azquelt]

Changes made:

• Slide 18: (End User Overview) Added the config parameter
• Slide 28: (DevEx) changed to N/A
• Slide 35: (SVT) Added SVT scenario for using MP OpenAPI on two servers behind a Kubernetes ingress
• Slide 36: (Performance) Added test for performance regression when resuming from snapshot
• Slide 37: (Platform/Cloud) Updated with better terminology.

Still to do:

• Check whether AdminCenter links to OpenAPI documentation
• Investigate other features which provide OpenAPI documentation
• Define what messages are output when a conflict occurs

[Azquelt]

Changes made:

• changed MP Config properties to server.xml config elements based on the decision in #25327
  • <mpOpenApi docPath="/openapi" uiPath="/openapi/ui" />

[Azquelt]

Changes made:

• Change MP Config properties to server.xml config on slide 17 (missed in previous update)
• Added some explanatory text to the As-Is kubernetes ingress scenario (slides 10-13) to make it easier to follow without watching the presentation replay
• Added that existing endpoints /openapi/platform will not be moved when the configuration is changed (slide 22)
  • These endpoints document restConnector-2.0 APIs to retrieve config and check database connectivity
  • They're designed for use by system admins, not by end users, so we don't have the same motivation to allow them to be configured

Other tasks:

• Checked with the admin center team that they're not aware of anywhere they reference the Open API documentation or UI
• Searched the codebase for any other references and didn't find any.
  • We had an SPI for the older openapi-x.x features to allow other features to contribute to the generated OpenAPI documentation when the docs for all applications were aggregated together, but this was not carried across to the newer mpOpenAPI-x.x features.

[Azquelt]

Changes made:

• Added slide 28 stating the behaviour when the context root of an app conflicts with one of the mpOpenAPI endpoints (unchanged from current behaviour)

  The behaviour here isn't great, the app doesn't start and the mpOpenAPI endpoint is also removed. This is consistent with the behaviour when two apps are deployed at the same endpoint. I've raised #25855 to query it.

[abutch3r]

@OpenLiberty/demo-approvers Demo done in EOI 23.20

[donbourne]

Serviceability Approval Comment - Please answer the following questions for serviceability approval:

1. UFO -- does the UFO identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?

2. Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. L2, test team, or another development team).
   a) What problem paths were tested and demonstrated? b) Who did you demo to? c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that L2 should be able to quickly address those problems without need to engage L3?

3. SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered. a) Who conducted SVT tests for this feature? b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that L2 should be able to quickly address those problems without need to engage L3?

4. Which L2 / L3 queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective L2/L3 teams know they are supporting it. Ask Don Bourne if you need links or more info.

5. Does this feature add any new metrics or emit any new JSON events? If yes, have you updated the JMX metrics reference list / Metrics reference list / JSON log events reference list in the Open Liberty docs?

[abutch3r]

@donbourne the current state of the questions is

1. UFO covers most likely issues and are addressed either within the feature itself or another area where the messages have been checked. Those that apply to the implementation have been checked and each new message is tested for. Where handling of issues applies to the WebContainer in cases of endpoint conflicts, the behavior and messages have been checked and in this case had a fix supplied by the WebContainer team. The following scenarios are called out in the UFO:
   • Invalid characters - Error Messages include valid characters that can be used
   • Conflict with another applications endpoints - WebContainer errors cover this and include useful information about the cause of the conflict.

2. a. During demo scenarios covering invalid characters and defining the same path as a customer application were shown b. Demo to EoI attendees and Microprofile UK team c. No questions about the messages were raised during the demo and our only remaining issues are for us to validate the messages do cover all the scenarios that the team can think of

3. Will follow up with Brian
4. The existing queues used for OpenAPI will continue to be used
5. No new metrics or JSON events have been added.

[tevans78]

@OpenLiberty/demo-approvers Demo done in EOI 23.10

You meant EOI 23.20

[natalie-bernhard]

The following attachment contains accessibility issues for the Swagger UI that need to be addressed. One file is an XLSX and one is HTML, but both contain the same information, just organized differently. Accessibility Issues for Swagger UI.zip

[abutch3r]

@natalie-bernhard The Swagger-UI is an OSS dependency we have and we and even they acknowledge that the UI is not fully accessibility compliant as they have a number of issues open on them for it.

A query about the report you have provided is, Would the report that has been generated be accepted by an external third-part - the WCAG parts I believe would be, but the IBM parts may not be and we may have to strip out IBM references.

Given timeframes to get fixes in upstream so they can be maintained through future releases will take a significant amount of time that this feature in no way warrants giving.

Also as the document itself can be downloaded and accessed in a users own tooling, therefore it does not provide the only method by which someone can obtain the information about the API. As such the core functionality provided by the feature is not blocked by an inaccessible UI.

So to get the approval in time for the next release what do we need to put in place. Can we either

• State that we will create fixes to put in the upstream component, that would be included in the next update of the Swagger-UI version to address primarily the critical failures, which we do periodically - though likely occur to when we do the next mpOpenAPI version
• Get an exception for this component explicitly

[tjwatson]

I'm approving with the expectation that we get the testcase #26966 added this week for InstantOn.

[abutch3r]

@donbourne

SVT approval has now been given. the tests were done by Brian Hanczaryk

As for serviceability, the main issue encountered was with examples in the UFO having the wrong case compared to the supporting metatype. As such the values when provided were ignored. Once we had worked this out then the issue was resolved and nothing else in regards to the features error messages was flagged

This is a wider issue with types, not just for this feature. As currently a mistype is not flagged in anyway such as an unknown type or a close match being highlighted. The only fix is ensuring the documentation and examples are correct. The beta blog does use the correct example and will be reused for the GA blog and the documentation supporting this will be generated from the metatype, so will match. This does need flagging under the STE that types are case sensitive.

Are there any other areas under serviceability that need additional information

[abutch3r]

@OpenLiberty/ste-approvers WAS L3 site page here: https://github.ibm.com/WASL3/site/pull/61

[gnadell]

@abutch3r ,L2 has requested STE slides for this feature. The STE template can be found at the links below. You can use either one to create the education. Slide Template: https://ibm.box.com/s/1an42g7zdgmaj84w7dft0indqfgi8ffm

Github Template: https://pages.github.ibm.com/WASL3/site/STE/about

Please upload the completed slides to the same 'STE Archive' BOX folder and let @tngiang73 or me know they're complete or provide us the Github link. Thanks!

[abutch3r]

@gnadell @tngiang73 I have uploaded the slides to https://ibm.ent.box.com/file/1368308474181

Also, when I try to go to the github pages site I get accessing token endpoint (https://login.w3.ibm.com/oidc/endpoint/default/token) failed: address not available

[gnadell]

@abutch3r , I checked with Eric Covener. It appears to be a github issue. Please retry the link later.