fix: vacuum more runs needed error

[Jrmyy]

When there is too many files to process during a vacuum, dbt model fails with this error: "ICEBERG_VACUUM_MORE_RUNS_NEEDED: Removed 20000 files in this round of vacuum, but there are more files remaining. Please run another VACUUM command to process the remaining files."

We apply therefore the same logic as we did for the optimize. There is also an attempt to gather the code since they have the same logic.

Description

Models used to test - Optional

Checklist

• [X] You followed contributing section
• [X] You kept your Pull Request small and focused on a single feature or bug fix.
• [X] You added unit testing when necessary
• [X] You added functional testing when necessary

[nicor88]

@Jrmyy do we have a way to test this in the CI?

I can imaging to setup an iceberg table with vacuum_max_snapshot_age_seconds set to 1 second, then insert many times to the same table to lead to have a situation where iceberg has many snapshot to expire, and finally try to run the vacuum on an iceberg table with many commit.

PS: code looks good, I re-triggered the CI that randomly failed due to a functional test where we run concurrent iceberg inserts

[Jrmyy]

I can try what you suggest ! Since the VACUUM fails when there are more than 20000 files to remove, it means we will have to insert a lot of lines ahah 🙈

I will give it a try and let you know if it works 🔥

[nicor88]

@Jrmyy I totally understand that reproducing a failure of a vacuum can be cumbersome, if we don't manager to reproduce it, well leave it like that.

[Jrmyy]

I tried to create some sql queries which maximise entropy in order to generate as much files as possible to be sure vacuum will perform several times. But it did not scale very well. With 100 epochs of doing this query, I managed to get "only" 1000 files, which is not close to the 20k files needed to an other round of vacuum. And it tooks near 7 minutes ... So I think testing it will be counter-productive 😞

[nicor88]

Thanks @Jrmyy let's leave it as it is.