Add optional access control to API

[jraddaoui]

Refs #957.

Use Keycloak instead of Dex in dev env

Dex allowed us to implement OIDC authentication in front of an OpenLDAP instance in the development environment. However, it does not provide the IAM tools to manage access control in Enduro.

• Remove Dex and OpenLDAP from the Kubernetes environment.
• Add Keycloak to the environment:
  • Use development server.
  • Listen in 7470 to avoid collisions.
  • Import a custom enduro realm with:
  • Clients for Enduro, Temporal and MinIO.
  • Minimal OIDC built-in scopes.
  • Custom client scopes for Enduro and Minio.
  • Custom user attributes and claims mapped to those scopes.
  • Add three default users with different sets of attributes.
  • Without persistence to recreate the custom realm on each start.
  • Update docs to use keycloack instead of dex in /etc/hosts.
  • Update access section to indicate new credentials and service.
• Set up MinIO and Temporal to SSO with Keycloack.
• Update default Enduro configuration to work with Keycloack.
• Remove offline_access from the requested scopes (a following commit/PR will make scopes configurable).
• Update Pulumi project and related Github action.
• Add TODO note in dashboard to end sessions in the provider.

Add optional access control to API

Optionally enable Attribute Based Access Control (ABAC) in the API. When enabled, the API will look for a custom claim in the access token and extract the attributes relevant to Enduro. If a custom scope needs to be requested to get that claim, it has to be configured in the dashboard.

• Add OIDC ABAC configuration. Allows to enable or disable access control entirely and set a claim to get the attributes. That claim could be nested and include values unrelated to Enduro.
• Extend claims and token verifier to parse ABAC attributes from the access token based on configuration.
• Allow custom scopes in dashboard and configure it to work with the Keycloak instance for the dev. env. by default.
• Add scopes, forbidden error and security to all endpoints in Goa's API design.
• Check scopes against attributes in API requests. As a first access control check, the scopes configured on the API design need to be included in the user attributes.
  • This check allows attributes with wildcards, so:
  • package:read will only provide access to that action(s).
  • package:* will provide access to all package related actions.
  • * will provide full access to all actions.
• Add claims with the extracted attributes to the context on each request. After that initial check is passed, the claims are included in the context to make them available for other possible access control checks on the endpoints implementation.
• Re-order API design methods in storage service:
  • Keep locations and packages methods together.
  • Move submit to indicate update is related to that method and not to create.

TODO

Follow-up work that may or may not happen in this pull request.

• Determine the attributes used for each endpoint and user action [done].
• Consider access control in the dashboard:
  • Add similar configuration and attribute extraction.
  • Add similar checks to hide/show elements/pages.
  • Consider websocket connection and access control (disable it for now when access control is enabled) [done].
  • Add landing page for users without the required attributes. Right now the landing page is the packages list.
• Add administration section to the documentation:
  • Add notes to configure OIDC ABAC in the API and the dashboard, explaining the current behavior.
  • Add notes to build and serve the dashboard, considering that configuration.
  • Document API, including the attributes/scopes needed.

Note: moving to a BFF auth. flow could simplify some of these changes and documentation, but that won't happen in this PR.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 85.71429% with 15 lines in your changes missing coverage. Please review.

Project coverage is 53.06%. Comparing base (07ac6f6) to head (673c572).

Files	Patch %	Lines
internal/api/auth/token_verifier.go	91.11%	3 Missing and 1 partial :warning:
cmd/enduro-a3m-worker/main.go	0.00%	3 Missing :warning:
cmd/enduro-am-worker/main.go	0.00%	3 Missing :warning:
cmd/enduro/main.go	0.00%	3 Missing :warning:
internal/api/auth/claims.go	91.30%	1 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #965 +/- ## ========================================== + Coverage 52.16% 53.06% +0.90% ========================================== Files 100 101 +1 Lines 5590 5661 +71 ========================================== + Hits 2916 3004 +88 + Misses 2428 2408 -20 - Partials 246 249 +3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[jraddaoui]

@djjuhasz @mcantelon @sevein @sbreker

While this is still a work in progress and I'm working in a document explaining the existing OIDC implementation, I think there is enough information in this PR description and in code for you to see where this is going and provide some initial feedback.

I'd appreciate if you can take a look when you have some time. It will be better if you review each commit individually, they have a detailed message (the same as in the PR description), and also check the TODO section. Thanks!

[jraddaoui]

Thanks @djjuhasz! I addressed most of your feedback, I'll ping you for another review after adding all the scopes we discussed yesterday and some of the dashboard TODOs.

[sbreker]

This looks amazing @jraddaoui! :star_struck: I do not have anything to add beyond the others' comments. Thanks for your efforts with this!

I agree that follow-on work might work best as a second PR after this is merged...

[jraddaoui]

Thanks @sevein and @sbreker!

I think I have addressed all feedback and added a few more commits. I'll leave it disabled for now until we add the dashboard side and the documentation, which as you said will happen in another PR.

[djjuhasz]

@jraddaoui I tried testing the authentication out locally, but I'm getting an "NS_ERROR_UNKNOWN_HOST" error from keycloak when I click the Enduro Dashboard "Sign In" button. The only config change I made was to change api.auth.oidc.abac.enabled = true in "enduro.toml".

[djjuhasz]

@jraddaoui nevermind, I forgot to add keycloak to my /etc/hosts. RTFM moment. :facepalm:

[djjuhasz]

After updating my /etc/hosts I logged into the Enduro Dashboard fine with the "readonly" account, and I navigated around the read pages without any issues. I also logged into the Temporal UI and MinIO without any issues.

When I tried to process a SIP, I got an error in the upload activity though:

{
  "message": "[storage create]: invalid value expected *storage.CreatePayload, got &{79dc514f-d42e-40b4-a17b-2c2cc325f88f ZippedBag.zip <nil>}",
  "source": "GoSDK",
  "applicationFailureInfo": {
    "type": "ClientError"
  }
}

It's possible the error is unrelated to your changes, but I've never seen it before.

[jraddaoui]

Thanks @djjuhasz! That's totally related to these changes, I fixed that issue in the tests, but I thought the service was created differently in the main functions. Will fix!

[jraddaoui]

No, it doesn't. That was just a cosmetic commit:

Re-order API design methods in storage service

• Keep locations and packages methods together.
• Move submit to indicate update is related to that method and not to create.