search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.25k stars 400 forks source link

Glue Crawler Error: "Internal Service Exception" when integrating Delta with Glue Catalog #2937

Closed marcoaanogueira closed 4 hours ago

marcoaanogueira commented 4 hours ago

Environment

Delta-rs version:0.20.0

Environment:

Bug

What happened: While attempting to integrate Delta Lake tables with the AWS Glue Catalog, I encountered a generic error: Crawler Error: Internal Service Exception. This error occurred during the process of cataloging Delta Lake tables, and I couldn't find much information on how to resolve or troubleshoot this issue.

What you expected to happen: The Glue Crawler should successfully catalog the Delta Lake tables without encountering any errors.

How to reproduce it: 1 . Create a Delta Lake table.

  1. Set up an AWS Glue Crawler to scan and catalog the table.
  2. Run the Crawler.

More details: Error:

Captura de Tela 2024-10-10 às 14 13 48

CloudWatch Error:

Captura de Tela 2024-10-10 às 14 14 58
rtyler commented 4 hours ago

Please contact AWS Support for this, delta-rs does do anything with Glue Data Catalog and I cannot provide any advice here.

That said, I have successfully used their crawlers in the past for Delta tables. I don't typically use them because they're bad and expensive though :laughing:

marcoaanogueira commented 3 hours ago

Thank you very much