Security improvements in the Iceberg REST specification

[snazy]

Proposed Change

Following up on the mailing list discussion, we propose the following changes to Apache Iceberg. The following summary chapter is a summary of the initial message on this topic on the iceberg-dev mailing list.

We think that the ‘/v1/oauth/tokens’ endpoint in the Iceberg REST spec poses potential security and OAuth2 compliance issues, and excessively restricts how authorization should be implemented.

• As an open table format, it would be good for Iceberg to focus on the table format / catalog and not how authorization is implemented. The existence of an OAuth endpoint pushes implementations to adopt authorization using only OAuth, whereas the implementers might choose several other ways to implement authorization (e.g. SAML). In our opinion the spec should leave it open to the implementation to decide how authorization will be implemented.
• The existence of that endpoint also pushes operators of Iceberg REST endpoints into the authorization service business. Clients might expose their clear-text credentials to the wrong service, if the (correct) OAuth endpoint is not configured (humans do make mistakes).
• (Naive) Iceberg REST servers may proxy requests received for ‘/v1/oauth/tokens’ - and effectively become a “man-in-the-middle”, which is not fully compliant with the OAuth 2.0 specification.

The goals of this proposal are:

1. Secure the Iceberg REST specification by preventing accidental misuse/misimplementation.
2. Prevent that Iceberg REST to get into dictating the “authorization server specifics”.
3. Enable flexibility for Iceberg REST servers to opt for other authorization mechanisms than OAuth 2.0.
4. Enable REST servers to opt for integrating with any standard OAuth2 / OIDC provider (e.g. Okta, Keycloak, Authelia).

Proposed "milestones" are: M1: Deprecate the /v1/oauth/tokens endpoint, targeting Apache Iceberg 1.7.0 M2: Update & clarify documentation, asap M3: Define a pluggable REST client authorization framework, before M4 M4: Reference client authorization implementation(s) in Java, targeting Iceberg 1.8.0 or 2.0 M5: Removal the /v1/oauth/tokens endpoint, targeting Iceberg 1.9.0 or 2.0

Details in the linked document.

Proposal document

https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ/

Specifications

• [ ] Table
• [ ] View
• [X] REST
• [ ] Puffin
• [ ] Encryption
• [X] Other

[adutra]

Should we add this to the "Iceberg REST Catalog" milestone?

https://github.com/apache/iceberg/milestone/46

[ajantha-bhat]

Should we add this to the "Iceberg REST Catalog" milestone?

I have added now.

[ajantha-bhat]

There is also a weekly sync planned for REST catalog. We have one on next Monday here the second point in agenda is to discuss the same. https://lists.apache.org/thread/yy2qjbwk5lzd1ro2opl8zvb3lflw1pd8