IntelLabs / vdms

VDMS: Your Favorite Visual Data Management System
MIT License
84 stars 31 forks source link

S3 objects not cleaned up on transaction failure #196

Open ifadams opened 1 month ago

ifadams commented 1 month ago

Describe the bug

Currently, a failed VDMS transaction will not lead to the associated image being removed, causing a storage leak.

To Reproduce

Add multiple images, include an invalid operation (or garbage data) that will trigger a failure on the second (or later) image add.

Expected behavior

If a transaction fails, the S3 objects should be removed as well.

Additional context

We need to have a broader discussion on VDMS transactions vs individual query transactions, because when we start including things like remote storage it gets very messy.