Determine if customers can access Kibana for their brokered ES instance

[markdboyd]

Background

See https://gsa-tts.slack.com/archives/GS7RNLM1N/p1699292497648179

To do

• [ ] Test to see if customers can access Kibana for a brokered instance
• [ ] If so, write a knowledge base article for customers on how to do this

[markdboyd]

I did some initial investigation here. Findings:

• You can create a tunnel that allows you to access the VPC host for your ES domain, however you need to use /etc/hosts or SOCKS5 proxying to access the domain host over the tunnel, otherwise you will get certificate errors

• Even if you create the tunnel successfully, the domain security config doesn't allow incoming traffic from anonymous users, so you would get an error like:

  User: anonymous is not authorized to perform: es:ESHttpGet

It should be possible to use the brokered credentials to sign requests to the ES domain, but other than using a proxy I'm not sure how to sign the requests to Kibana for the domain.

[markdboyd]

Reverse proxy

After discussion with the team, it was suggested to use a reverse proxy deployed as an app which could talk to the ES/OS instance and display its Kibana/Dashboard.

AWS even has an example proxy which handles the request signing properly and can be configured to use brokered credentials for an instance:

https://github.com/awslabs/aws-sigv4-proxy

The only problem with using this proxy is that works almost too well: because the requests are signed by the brokered IAM credentials, they don't require any credentials to log in to the Kibana/Dashboard itself.

The AWS docs even call out that you can't use both IAM credentials and fine-grained access controls at the same time:

If a resource-based access policy contains IAM roles or users, clients must send signed requests using AWS Signature Version 4. As such, access policies can conflict with fine-grained access control, especially if you use the internal user database and HTTP basic authentication. You can't sign a request with a username and password and IAM credentials. In general, if you enable fine-grained access control, we recommend using a domain access policy that doesn't require signed requests.

However, I don't think we want to remove the IAM users from our brokered instances access policies, since that is how we guarantee that customers can only talk to their specific instances.

Dashboard SSO

Another, completely different option is using Dashboard SSO which is a built-in feature of the service broker API. I'm still grasping how it works, but it seems like it would be something like:

• Brokered instance is registered with a dashboard_url property
• When a user goes to access the dashboard URL, they have to log in with UAA. Then the broker logic can verify that they have access to the dashboard for the specific service
• Once login succeeds, the user could be sent to the dashboard URL, which in this case could be a proxy

Questions/thoughts

• The proxy would have to handle proxying to any brokered instance
• How would the proxy get the credentials to sign requests for any brokered instance?

[markdboyd]

After discussion with @spgreenberg and @mogul, we determined that the dashboard SSO is the only viable path forward here because:

• The AWS service proxy uses V4 signed headers to authenticate requests with brokered credentials, which is not compatible with using fine-grained access controls to manage access to Kibana/Dashboards. Thus, there would be nothing actually enforcing authentication and anyone who had the proxy URL would have unfettered access to Kibana, which is unacceptable
• Even if using fine-grained access controls were possible, it would require manual management of users by customers
• Dashboard SSO will integrate with CF authentication and allow very specific checks that the authenticated UAA user has access to view/manage a service instance
• Dashboard SSO is a built-in feature of the open-service broker API, so it's good to leverage
• Dashboard SSO links will appear in Stratos for easy discovery and can be easily integrated in the new dashboard solution

I have taken some initial steps towards this implementation:

• Proved that the AWS service proxy can use IAM credentials with read-only access to all Elasticsearch/OpenSearch domains to sign requests
• Created a basic Golang implementation of OAuth with UAA: https://github.com/markdboyd/dashboard-service-proxy

The next steps would be:

• [ ] Improve the Golang OAuth/UAA integration:
  • Add verification of the JWT received from UAA before decoding it, possibly using the JWKs provided by UAA. See https://uaa.fr.cloud.gov/.well-known/openid-configuration.
  • Add unit tests
  • Create a route for handling proxying to Elasticsearch/Opensearch services:
  • If not logged in, should redirect for auth
  • If logged in, should show list of ES/OpenSearch services
  • Once an instance is selected, should use AWS proxy to show Kibana/Dashboards for instance
• [ ] Deploy dashboard service proxy as app in dev
• [ ] Update AWS broker in dev to include dashboard_client pointing at the new dashboard service proxy
• [ ] Verify that links for dashboard appear in Stratos/etc and then when clicking the link, the user goes through the auth process and is able to successfully view their instance Kibana/Dashboards

[markdboyd]

Notes on verifying JWTs:

• https://pkg.go.dev/github.com/golang-jwt/jwt/v5#example-Parse-Hmac
• https://stackoverflow.com/questions/51834234/i-have-a-public-key-and-a-jwt-how-do-i-check-if-its-valid-in-go
• JWT validation with JWKs: https://stackoverflow.com/questions/61850992/jwt-validation-with-jwks-golang

[mogul]

I don't think you need this:

• If logged in, should show list of ES/OpenSearch services
• Once an instance is selected, should use AWS proxy to show Kibana/Dashboards for instance

The dashboard_url that is provided to clients along with the other instance details should be specific for that instance, not identical for all of them.

For instance:

dashboard_url: https://dashboards.fr.cloud.gov/SERVICEGUID/INSTANCEGUID

If you later decide you do want to offer folks some kind of navigation/chooser, then you can make this functional...but that's not what gets supplied in dashboard_url:

dashboard_chooser: https://dashboards.fr.cloud.gov/SERVICEGUID

Or even

dashboards: https://dashboards.fr.cloud.gov

But that goes into UX issues that are best left to the overall platform UI. And you can support that by only implementing the first case, so don't overdo it!