aai-institute / lakefs-spec

An fsspec implementation for the lakeFS project.
http://lakefs-spec.org/
Apache License 2.0
37 stars 4 forks source link

Warning message when ending a transaction while having uncommited objects #264

Closed janwillemkl closed 4 months ago

janwillemkl commented 4 months ago

What is the motivation and/or use case?

Consider the following scenario: A user starts a new transaction and adds an object to lakeFS. Without making a commit (forgetting to add it), the user ends the transaction.

A temporary branch is created for the scope of the transaction. The objects are uploaded to the temporary branch. Without a commit, there is nothing to merge back into the source branch, and the temporary branch is deleted.

This means that there is no 'evidence' left in lakeFS that any objects were uploaded. This may lead to accidental data loss, and without any messages or indication within lakeFS, it is very difficult to debug.

Showing the user a warning that objects were uploaded, but not committed, could help a great deal.

How can we implement this feature?

When closing the transaction, check via the lakeFS API if there are uncommited files, if yes, display warning.