matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
https://matano.dev
Apache License 2.0
1.42k stars 98 forks source link

Faulty Metadata `file_path` Pointers #180

Open damon-edstrom opened 10 months ago

damon-edstrom commented 10 months ago

We're experiencing a problem appears to be resolved recently in the 3a4967d commit with Athena queries returning the ICEBERG_CANNOT_OPEN_SPLIT error when it hits a faulty file pointer. Is the fix for this retroactive and will be resolved once the maintenance task is ran? Or, is it a going forward type fix where it shouldn't happen anymore but existing occurrences will still fail?

If it's the latter, is there something I can do to resolve the issue or should we plan on replaying the data to have a clean run?