Build Spring Cloud Gateway as a replacement for the current Gateway

[balhar-jakub]

Use Case

As an API ML engineer, I want to limit the number of old dependencies to prevent security breaches.

As a decision maker, I want to be certain that we use modern tools with limited risk of security issue.

Technical Requirements

• Zuul won't be used anymore
• Spring Cloud Gateway will be used as a replacement
• The replacement Gateway will be built side by side.
• Deprecate Current Gateway to be removed in V3
• Find the right place for Authentication functionality

Linked issues

• 3165

• 3109

• 3282

• 3088

• 3110

• 3111

• 3112

[balhar-jakub]

There are plenty of areas that we can rebuild and verify piece by piece before fully replacing API Gateway with the Spring Cloud Gateway. Below I am trying to outline the areas that we could split as separate issues.

For the time being, we can keep the authentication and authorization in the current implementation and therefore transform the current Gateway to the Authentication and Authorization Service.

Routing

• Route REST request through the gateway - The cornerstone functionality requires the Gateway to understand the metadata and be able to issue the REST call to the southbound service with the same parameters received by the Gateway.
  • The first step could be to support one instance of the service. The second step could properly load balance between multiple instances of the service
• Route WebSocket request through the gateway - The metadata must understand that a specific pattern belongs to the WS. The second step would be to verify the support of the STOMP protocol.
• Properly retry when a problematic response from southbound service (Ribbon + Retry) - If there is more than one instance of the service if 500 or other retryable status code happen in southbound service, the Gateway tries to issue the code against the other instances one by one.
  • There are some areas that are related to this topic. If some service doesn't respond multiple times within a given time frame, the Gateway needs to at least temporarily stop issuing requests against the instance.

Authentication

• Support the protection of the routes via the JWT token. - This is the first step to provide the current authentication functionalities. After the first step, there are the exchanges of the token based on what the services require. I explain these areas below.

Advanced functionality

These are the areas that we currently support and are more or less supported by the SCG as well, it's just that we need to properly document how to approach them. And also they aren't per se required for the basic functionality.

• Support AT-TLS - This should happen in the background in a transparent way. Needs verification in the SCG.
• Allow CORS delegation - Make sure that the proper headers are set and that the services can configure how exactly and for which origins.
• Compression on behalf of the southbound service - If the service requires the functionality the Gateway compresses the response on behalf of the southbound service. The requirement comes from ZSS where the compression implementation is painful.
• Configure connection timeouts - This is relevant to allow for specific scenarios such as a high amount of requests or a specific low amount of long-running requests.
• Sticky Sessions (Needed by more services, originally came from IBM)
  • We support either that the specific authenticated user will go to the same instance of the service or that the northbound services can decide what instance will get which request.

Authentication and Authorization

It's a core part of the Gateway functionality as it is currently used to integrate all southbound services in an SSO with MFA experience of all of the Mainframe services. Therefore there are two important methods that are available for the northbound caller to authenticate (X509 and JWT token) and then there are five different ways to authenticate towards the southbound service. These methods are outlined below and can be implemented one by one.

Could be done in another service - Take into account time taken by HTTP request itself, This part still needs a bit more testing and verification. Using this approach could allow us to just repurpose current gateway to the AAS.

• Support X.509 Authentication - X509 information are passed via the headers to the southbound service.
• Support SAF Authorization - The southbound services receive valid SAF IDT token.
• Support Passtickets - Generate PassTicket for southbound service and pass it along instead of password.
• Support JWT token validatable by the new Gateway. - Provide in cookie the JWT token that can be validated based on the setup either via Gateway or also via zOSMF.
• Support ByPass - Authentication is fully ignored.

[balhar-jakub]

A more technical breakdown that was done by David in https://github.com/zowe/api-layer/issues/1769

The key section is copied below.

Proxy approach

• What are the benefits of this contrary to Big Bang migration?
• Introduce proxy service and gateway service and migrate between the two.
• No breaking changes anticipated ? Maybe not true but maybe the breaking changes can be implemented first in the old gw and released like this.
• There has to be a proxy, separated by a communication layer
• This layer would be REST
• Which service to be the proxy and which should be the new one
• How to manage certificate?
• How to manage trust between the services? Generated token or available certificate?
• Discovery client synchronization issues will happen, how will this affect the reverse proxy? Can we be seeing route from scgw > gw for a route and then when route gets to scgw it would take over? We should prevent that. That means potentially routing has to be moved in one go from GW to SCGW.
• Concerns about the proxification
• Localhost resolution: will need configuration Might not need to be onboarded to Gateway

> scgw > apimlgw >

• Advantage is that nothing is implemented and the proxy logic is not tainted by anything
• Disadvantage is that security has to be put in place to handle certificates equally
• Seems that certificate is the only thing
• AT-TLS? - should be ok if we move responsibility to the scgw and figure out certificate handling
• To migrate something means: implement it in spring cloud gateway and stop the forward
• This will create a version of gateway with proxied authentication logic for free
• What is easy to migrate:
  • Static endpoints like actuator, version …
• Medium effort:
  • Authenticated endpoints, especially with Certificate
• High effort:
  • routes
• At that moment, everything gw reverse proxy specific has to go
• When would GW require changes?
  • Once to estabilish x509 interop
  • Once to estabilish trust

Proxification implementation stages

This approach facilitates safe process where migration can be done on fully working gateway.

1) Production grade proxy

• Solve security of communication proxy - gw
• Solve deployment
• [SOLVED] Solve connectors
• Solve AT-TLS

Stories needed (10):

• Solve security of proxy to gateway communication (3)
• Package configure and deploy SCG together with APIML (5)
• Move AT-TLS processing to SCG (2)

2) Migrate Gateway routing

• leave other endpoints in GW
• leave zaas in GW
• proxy calls to zaas

Breaking changes

• depending on how successfull are we in migrating features to sc gw. This is where we could face breaking changes

Stories needed (18):

• RouteLocator and DiscoveryClient introduction (3)
• Configuration of SCG features for CircuitBreaker (1)
• Configure SCG for retry (1)
• Configure SCG to handle CORS (1)
• Create SCG Zaas client (3)
• Reimplement Authentication remapping (3)
• Reimplement load balancing (3)
• Reimplement whatever is needed from original filter chain (1)
• Error handling for new exceptions (2)

These stories can be front loaded before 1). They hold most of the risk. Compared to that, 1) is fairly low risk. If we are able to complete these reasonably fast, we are able to migrate for V2.

(following is optional)

3a) Migrate other endpoints to SCGW

• migrate other endpoints
• migrate zaas

3b) Cleanup GW and make it into ZAAS microservice

• client certificate already solved from 3)
• cleans gateway and decouples from zaas startup & config & logic

Breaking changes

• Potential path change for api consumers
• Breaking of GW and ZAAS might influence assumptions about cluster health

What breaking changes might happen

• [SOLVED] External vs internal port

• Auditing plugin will break

• [SOLVED] Client with and without certificate switching

  Seems like there is quite easy way to achieve this in the end. It is prototyped in the POC. I have tested it's stability for memory usage and thread counts and it's very stable.