search

codalab / codalab-worksheets

A collaborative platform for reproducible research (web interface and CLI).
Other
155 stars 83 forks source link

Some make bundles' data does not get stored in the bundle store #996

Closed bkgoksel closed 1 year ago

bkgoksel commented 5 years ago

As reported by Robin, some of his make bundles such as 0x4ce5b53c4e7440aab0aca24cb913630f look ready on the UI (and in the database, with the correct data_size field, and in the server logs with no errors), but they are not actually stored in the bundle store (i.e. no directory for their contents etc).

The end result is the same as if someone ran cl rm -d <uuid>: the bundle looks ready but any run that tries to depend on it fails with an uninterpretable dependency download failure message, and clicking the download button on the front end UI redirects to an ugly Path not found in bundle error page.

Few things to consider here:

1- Why do some make bundles lose their data randomly? 2- How should we deal with bundles whose data is lost somehow? 3- Should we even give the users the option to silently delete the data behind a bundle when it causes us so much trouble downstream to handle? 4- If we really want to give that option, do we want a deleted or truncated bundle state so people can keep the metadata while preventing attempts to download or depend on that bundle?

percyliang commented 5 years ago

On Thu, Nov 8, 2018 at 5:05 PM Kerem Goksel notifications@github.com wrote:

As reported by Robin, some of his make bundles such as 0x4ce5b53c4e7440aab0aca24cb913630f look ready on the UI (and in the database, with the correct data_size field, and in the server logs with no errors), but they are not actually stored in the bundle store (i.e. no directory for their contents etc).

The end result is the same as if someone ran cl rm -d : the bundle looks ready but any run that tries to depend on it fails with an uninterpretable dependency download failure message, and clicking the download button on the front end UI redirects to an ugly Path not found in bundle error page.

Few things to consider here:

1- Why do some make bundles lose their data randomly? 2- How should we deal with bundles whose data is lost somehow? 3- Should we even give the users the option to silently delete the data behind a bundle when it causes us so much trouble downstream to handle? 4- If we really want to give that option, do we want a deleted or truncated bundle state so people can keep the metadata while preventing attempts to download or depend on that bundle?

3 is a not well-known feature. I think in practice, it might be less easy to use. You might want to delete files selectively (which is theoretically possible by doing a cl make on the files you want to keep, but not super ergonomic).

I think 4 is a great idea!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/codalab/codalab-cli/issues/996, or mute the thread https://github.com/notifications/unsubscribe-auth/AAakuPI_os6c5zqkF8CgbuhaLPDNoZC4ks5utNTIgaJpZM4YVxlO .