Add our expire snapshots to fix Athena ICEBERG_CANNOT_OPEN_SPLIT

matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS

https://matano.dev

Apache License 2.0

1.42k stars 98 forks source link

Add our expire snapshots to fix Athena ICEBERG_CANNOT_OPEN_SPLIT #175

Closed shaeqahmed closed 11 months ago

shaeqahmed commented 11 months ago

The athena implementation has a bug that can cause orphaned pointers to manifest/data files and break the tables. Move back to our lambda based expire snapshots flow.