[Epic] Multi-param DestinationSpec for K8S Gateway API HttpRoutes

[DuncanDoyle]

Gloo Edge Product

Enterprise

Gloo Edge Version

1.17.0

Is your feature request related to a problem? Please describe.

Context

In our current implementation to support function parameters for Function Upstreams in K8S Gateway API, we use an ExtensionRef to a virtual Parameter resource to pass in, for example, the logical name of the Lambda function to call. See this GitHub issue comment, which contains the following example:

backendRefs:
    - name: lambda
      namespace: gloo-system
      group: gloo.solo.io
      kind: Upstream
      filters:
        - type: ExtensionRef
          extensionRef:
            group: "gloo.solo.io"
            kind: Parameter
            name: echo

Since this is an unnamed parameter, we only support one Function Upstream parameter to be specified. There are however destinationSpec in the Gloo Edge API that support multiple parameters: https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/options.proto.sk/#destinationspec

The aws destinationSpec for example supports fiels like wrapAsApiGateway, unwrapAsApiGateway, invocationStyle, etc.: https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/options/aws/aws.proto.sk/

With the current K8S Gateway API implementation, we can't/don't support these additional parameters.

Describe the solution you'd like

Requirement

Provide support for multiple destinationSpec parameters in the K8S Gateway API for Function Upstreams. We must support these parameters for the following Function Upstream (https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/options.proto.sk/#destinationspec):

• aws
• azure
• rest

Since the grpc destinationSpec is deprecated (see: https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/options/grpc/grpc.proto.sk/#destinationspec), we should not add support for it in our K8S Gateway API implementation.

Describe alternatives you've considered

n.a.

Additional Context

After some research, there is some additional context:

Lambda

For the DestinationSpec use-case for AWS Lambda, there is a "sort of" workaround to set these properties when using K8S Gateway API. In the Upstream you can specify a destinationOverrides option, which is itself of type DestinationSpec, and hence you can set these options directly on the Upstream, like so:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: my-aws-lambda-upstream
  namespace: gloo-system
spec:
  aws:
    region: eu-central-1
    secretRef:
      name: aws-creds
      namespace: gloo-system
    destinationOverrides:
      unwrapAsApiGateway: true

The UX is questionable, as that field is intended as an override setting, and here we're "overriding" something that has not actually been set. See also: https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/options/aws/aws.proto.sk/#upstreamspec

Azure

The Azure DestinationSpec only contains a single field (functinonName, see: https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/options/azure/azure.proto.sk/#destinationspec), hence you can argue that the current solution of specifying a Parameter in an ExtensionRef is sufficient. However, from an API design perspective, the DestinationSpec API design for K8S Gateway API must be aligned across the 3 use-cases/supported areas (i.e. aws-lambda/azure/rest).

### Tasks
- [ ] https://github.com/solo-io/gloo/issues/9838
- [ ] https://github.com/solo-io/gloo/issues/9814
- [ ] https://github.com/solo-io/solo-projects/issues/6672
- [ ] https://github.com/solo-io/gloo/issues/9815
- [ ] https://github.com/solo-io/gloo/issues/9816

┆Issue is synchronized with this Asana task by Unito

[nrjpoddar]

@sam-heilbron to look at this issue for scope/estimation.

[DuncanDoyle]

It seems that you can use destinationOverrides on the Upstream to add additional configuration logic, at least for AWS Lambda Upstream, in a K8S Gatewat API use-case. I.e. with the following HttpRoute:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: lambda-example-com-route
  namespace: default
  labels:
    app: apis
spec:
  parentRefs:
    - name: gw
      namespace: ingress-gw
  hostnames:
    - "lambda.example.com"
  rules:
    - matches:  
      - path:
          type: Exact
          value: /
      backendRefs:
      - name: my-aws-lambda-upstream
        namespace: gloo-system
        group: gloo.solo.io
        kind: Upstream
        filters:
        - type: ExtensionRef
          extensionRef:
            group: "gloo.solo.io"
            kind: Parameter
            name: QuarkusLambdaDdoyle

I can set the unwrapAsApiGateway parameter as follows on the Upstream:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: my-aws-lambda-upstream
  namespace: gloo-system
spec:
  aws:
    region: eu-central-1
    secretRef:
      name: aws-creds
      namespace: gloo-system
    destinationOverrides:
      unwrapAsApiGateway: true

So there seems to be somewhat of a workaround to support these additional parameters for AWS Lambda Upstreams, but the UX is questionable (in particular because we are defining that we're overriding a destinationSpec that does not really exist.

[sam-heilbron]

When we deliver this Epic, our focus is the design plan first (which needs to incorporate all of the sub-tasks), and then lambda support second. After those are completed, we should regroup with @DuncanDoyle