When dropTable is called with purgeRequested, the Polaris server will create Tasks in order to delete data files associated with the table being dropped. However, if the service dies before the tasks are completed they may not resume gracefully on startup. Some of that table's data files may not be cleaned up.
To Reproduce
Call dropTable with purgeRequested, and kill the Polaris service before the relevant Tasks complete.
Actual Behavior
Tasks are not resumed on startup
Expected Behavior
Tasks associated with a purge should reliably complete, or the user should be made aware that they cannot be completed
Additional context
I've met with @collado-mike and others to discuss a long-term solution here, but it may take some time. In the interim, we discussed some workarounds and stopgaps.
Is this a possible security vulnerability?
Describe the bug
When dropTable is called with
purgeRequested, the Polaris server will createTasks in order to delete data files associated with the table being dropped. However, if the service dies before the tasks are completed they may not resume gracefully on startup. Some of that table's data files may not be cleaned up.To Reproduce
Call
dropTablewithpurgeRequested, and kill the Polaris service before the relevantTasks complete.Actual Behavior
Tasks are not resumed on startupExpected Behavior
Tasks associated with a purge should reliably complete, or the user should be made aware that they cannot be completedAdditional context
I've met with @collado-mike and others to discuss a long-term solution here, but it may take some time. In the interim, we discussed some workarounds and stopgaps.
System information
n/a