codalab / codabench

Codabench is a flexible, easy-to-use and reproducible benchmarking platform. Check our paper at Patterns Cell Press https://hubs.li/Q01fwRWB0
Apache License 2.0
76 stars 28 forks source link

Submissions Stuck for Competition 4430 #1664

Closed archettialberto closed 4 days ago

archettialberto commented 1 week ago

Dear Codabench Team,

We are the moderators of competition 4430 and are facing significant issues with the platform's performance. These issues are impacting our ability to efficiently manage the competition, particularly given the scale of our challenge, which involves almost 700 students. Below, we outline the main concerns:

1. Submissions Stuck in the "Submitting" Phase

Submissions are often stuck in the "submitting" phase for extended periods. For example, this happened last night (7 PM to 7 AM). New submissions began processing and scoring properly after 7 AM but old ones did not. Worker logs show no significant issues except for occasional failures to unpack bundle data:

an2dl-worker3  | [2024-11-15 11:11:07,170: INFO/ForkPoolWorker-11] Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/prediction_result/2024-11-15-1731666877/be333d307345/prediction_result.zip?AWSAccessKeyId=EASNOMJFX9QFW4QIY4SL&Signature=y%2B5f998wuOiJIl%2Ff7YXQ%2B65FdAw%3D&Expires=1732100954 to unpack @ input/res
an2dl-worker3  | [2024-11-15 11:11:07,302: INFO/ForkPoolWorker-11] Failed. Retrying in 60 seconds...

Our server resource usage (RAM, CPU, disk, network) appears normal during these periods. Of course, we tried restarting the workers but with no effect. How can we determine if the delays are due to Codabench platform issues or worker-related problems? What actions can we take to resolve these delays?

2. Performance Issues in Submission Management

The "Submissions" section got extremely slow to load. The issue is still present when filtering submissions by user or status, making it impractical to properly sort and manage them.

3. Student Quota Management

Codabench enforces a 15GB storage limit for each user. However, there is no option to delete old submissions to free up storage space. We understand that this feature is still under development, but we ask if there is a way to address deletions at this scale adequately.

These issues are critical to maintaining service uptime for our students. Thank you in advance for your assistance.

Best regards,
Alberto

Didayolo commented 4 days ago

I close this issue to keep track of it on #1666