Storage and clean up

[Didayolo]

Implementing a storage quota and effectively managing capacity could be crucial for enhancing Codabench's scalability and ensuring long-term service availability in production.

Features

Deleting files from the MinIO

@OhMaley: Add a button, in admin analytics page, to manually starts a script removing all orphans files.

• [x] Nothing is never erased from the MinIO. Add a button to remove all orphan files manually (#1523, #1575)
• [x] To check: #1319

Quota

We need to:

• [x] Impose a storage quota to each user (e.g. 15GB, platform setting) #1254
• [ ] Idea: separated quota for each joined competition
• [x] Admins should be able to manually increase the quota of any user (on Django admin interface) #1254
• [x] Have an user interface to manage and delete submissions Solved by: #918
• Actually, submissions can't be removed if there are part of a benchmark (which makes sense to avoid spamming submissions when there is a daily limit).
• Solution: submissions are counted on the quota of the participants, but they can remove a submission (even if it's part of a benchmark) after a delay of e.g. 7 days.
• WARNING: disallow removing leaderboard submissions
• Other idea: organizer phase setting to allow or disallow deleting submissions
• [x] Be able to delete tasks when deleting a benchmark #810 Partially Solved by: #918
• [ ] Be able to re-use past submissions #632
• [ ] Failed submission when deleted still remains as Data (deleted from leaderboard/submission panel but can be found in submissions in resources)
• [ ] The dumps should be counted too
• [ ] Refresh the quota figure automatically when submisison/dataset is uploaded in resource interface
• [ ] Readable format for the quota in the user table in admin interface .e.g MB

Light/archive mode after competition end

Option to clean competitions when there are completed. It would for instance keep only leaderboard submissions.

Automatic cleaning

• [ ] Failed submissions, orphans datasets, useless benchmarks, etc. should be marked and cleaned automatically. Users could manually unmark the object to keep it (from the resources interface). Deleted object should be kept in a trash can for one month. This feature needs to be discussed.
• [ ] #302 (schedule)

Monitoring and statistics

• [x] Fix analytics page download CSV buttons: #649 Solved by: #988
• [x] Improve analytics page to include storage statistics. Migrate analytics features from CodaLab. Solved by: #1248
• [x] date of storage calculation like codalab Solved by: #1265

Per benchmark submission size limit

Migrate the CodaLab feature on Codabench. This may be less important once the quota feature is implemented.

• [ ] Limit the size of submissions
• [ ] Django admin interface to define the limit independantly for each benchmark

Interface

@ihsaan-ullah

Improve resources interface

The interface for managing datasets, submissions and tasks could be improved. Indeed, improving the resources interface is also part of helping participants to manage their storage space.

• [x] Add the size of the files Solved by #895
• [x] Submission details: in which benchmark Solved by #896
• [x] make benchmark name clickable link Solved by #896
• [x] Submission: name it using the filename Solved by #896
• [x] Rename datasets tab in "Datasets and programs" #840, Solved by #898
• [x] Have a separate tab for submissions #596 (and show the title given by the user for datasets and programs, and other consideration in the last comment of #896) Solved by #899
• [x] Public datasets should have their own tab or a "show/hide" button (see #758) Solved by #906
• [x] Public datasets should not have delete button if they do not belong to us (see #758) Solved by #906
• [x] submission detail, make created_by a clickable link to user profile Solved by #910
• [x] The search bar for the tab "submissions" should search by filename Solved by #917
• [x] We need more filtering options. For instance, in the tab "submissions", we should be able to filter a specific competition. Solved by #917
• [x] UI for quota management and cleanup (A "progress bar" for the storage quota) Solved by #918
• [x] Delete unused tasks Solved by #918
• [x] Delete unused datasets and programs Solved by #918
• [x] Delete unused submissions Solved by #918
• [x] Delete failed submissions Solved by #918
• [x] Being able to order rows by clicking on the column name (e.g. size) Solved by #919

As shown in the screenshot below, we can manage and remove submissions from the "resources" interface. However, we do not track what the benchmark on which the submission was made, so it is hard to know what we are removing.

Some issues

• [ ] #145
• [ ] #637
• [ ] #552
• [ ] #862 (impose a limit on dumps for each benchmark)
• [ ] #163

[ihsaan-ullah]

@Didayolo

Public datasets should have their own tab

Is this only for public datasets (now datasets and programs) or also for submissions

Right now we have these tabs:

1. Submissions (shows my submissions and public submissions)
2. Datasets and Programs (shows my datasets and public datasets) Note: datasets are everything except submissions
3. Tasks

[Didayolo]

@ihsaan-ullah

I think my idea was about public datasets that you don't own. It should be either clearly indicated, either on a separated tab.

EDIT: oh and the problem is that it could become cluttered by public datasets in the future when we have many users. As submissions can also be made public, maybe different tabs is not the good way to go. I see that in the "Tasks" tab, there is a "Show public tasks" option. Maybe we need the same option for datasets and submissions, so we can show/hide the public ones.

[Didayolo]

Side remark on the resources interface, usernames could be clickable link to the profiles (when clicking on details in any tab):

In the leaderboard, the clickable link is made in this way:

<a href="{submission.slug_url}">{ submission.owner }</a>

https://github.com/codalab/codabench/blob/ce98a1f49c9233119a2997721113a668f29f60af/src/static/riot/competitions/detail/leaderboards.tag#LL59C57-L59C113

[ihsaan-ullah]

Side remark on the resources interface, usernames could be clickable link to the profiles (when clicking on details in any tab):

Added this as one todo in the interface todos list

[Didayolo]

I added the same TODO in the "job status" issue #744

[ihsaan-ullah]

• [x] Have an user interface to manage and delete submissions
• Can we confirm that the submissions are in the "resources" interface even if you are not organizer of the benchmark?
• Actually, submissions can't be removed if there are part of a benchmark (which makes sense... this needs discussion).
• [x] Be able to delete tasks when deleting a benchmark Task deletion #810

@Didayolo I think these two issues under the Quota heading are solved in the Interface solved issues by #918

[Didayolo]

• [x] We need to check that objects are really deleted when removed from the resources interface.

EDIT: it is not the case.

[Didayolo]

Actually, submissions can't be removed if there are part of a benchmark (which makes sense... this needs discussion).

About this point:

• We don't want participants to be able to freely remove submissions because that could allow some sort of cheating
• But if a participant have many big submissions and no storage left, what can he do? Some ideas: submissions could be counted on the quota of the competition organizer (instead of the participant) or not count in the quota at all.