aws / aws-sdk

Landing page for the AWS SDKs on GitHub
https://aws.amazon.com/tools/
Other
68 stars 13 forks source link

Support for Iceberg tables in the create-table command from the aws glue cli #549

Closed sfgarcia closed 1 year ago

sfgarcia commented 1 year ago

Describe the feature

In Athena it is possible to create Iceberg tables through DDLs (explained here https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-creating-tables.html). But this same functionality isn't possible to replicate through the aws glue cli in the create-table command (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glue/create-table.html) , because the TableType option doesn't allow to specify Iceberg.

Use Case

The main use case is to be able to create Iceberg tables as IaC, instead of just creating them through DDL queries inside Athena. This new feature would open the possibility to Cloudformation team to add the support to create Iceberg tables from the AWS::Glue::Table resource.

Proposed Solution

No response

Other Information

No response

Acknowledgements

CLI version used

v2

Environment details (OS name and version, etc.)

MacOS

indrora commented 1 year ago

Could you specify the full version of the CLI that you're using? you'll find it by running aws --version.

The CLI depends on keeping boto3 up to date. If you haven't updated your CLI since the introduction of the feature, the CLI won't have any knowledge or understanding of changes to the API.

Edit: I've just checked with the API documentation and the AWS Glue API itself does not allow the creation of these tables, so the CLI won't be able to create those tables without some way to pass DDL through to the CLI.

Closing.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

tim-finnigan commented 1 year ago

Checking in — I saw this documentation on using Iceberg tables in AWS Glue: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html

If what you're requesting isn't supported then we can forward that feature request to the Glue team as they own the CreateTable API. The CLI create-table command corresponds to the API functionality as mentioned above.

We recommend reaching out through AWS Support for API feedback/requests if you have a support plan but we can also forward those on your behalf. Please let us know.

sfgarcia commented 1 year ago

Hi @tim-finnigan thanks for your answer. Yes I would like if you could forward the API request to the Glue team please, is there any Github repository where I can see the roadmap of the Glue team?

tim-finnigan commented 1 year ago

Hi @sfgarcia thanks for following up. We don't have a GitHub repository for the Glue roadmap, but we can transfer this issue to our cross-SDK repository since the AWS APIs are used across SDKs. I'll reach out to the Glue team with this feature request and share any updates here on the issue. (ref: P90852964)

tim-finnigan commented 1 year ago

A backlog item for this feature request has been added for the Glue team's further review and consideration. We can't provide any ETAs or guarantees but please feel free to check back in here for updates in the future. We recommend reaching out through AWS Support if you have a support plan.

github-actions[bot] commented 1 year ago

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.