search

codalab / codabench

Codabench is a flexible, easy-to-use and reproducible benchmarking platform. Check our paper at Patterns Cell Press https://hubs.li/Q01fwRWB0
Apache License 2.0
76 stars 28 forks source link

Submission appear to fail after 24h #1639

Open jpbrunetsx opened 4 weeks ago

jpbrunetsx commented 4 weeks ago

Hi, In our competition we are running into an issue with failed submission with what appears to be a timeout. We did increase the running time to 72h in the task itself.

Competition : 3282 https://www.codabench.org/competitions/3282/ Exemple submission with a problem : 95068

Description: The submission fails after around 24h with the following error : 

`https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/submission_details/2024-10-22-1729621719/e8ba39ca3b12/prediction_ingestion_stdout.txt?AWSAccessKeyId=EASNOMJFX9QFW4QIY4SL&Signature=RenMoB7mHR2cs23sT7lGFLoAGz8%3D&content-type=application%2Fzip&Expires=1729708119

[2024-10-23 19:32:13,673: INFO/ForkPoolWorker-3] *** PUT RESPONSE: ***
[2024-10-23 19:32:13,674: INFO/ForkPoolWorker-3] response: <Response [403]>
[2024-10-23 19:32:13,674: INFO/ForkPoolWorker-3] content: b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>submission_details/2024-10-22-1729621719/e8ba39ca3b12/prediction_ingestion_stdout.txt</Key><BucketName>coda-v2-prod-private</BucketName><Resource>/coda-v2-prod-private/submission_details/2024-10-22-1729621719/e8ba39ca3b12/prediction_ingestion_stdout.txt</Resource><RequestId>18012BD6DA5A37FD</RequestId><HostId>e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</HostId></Error>'`

A shorter version running sub 24h does not trigger the issue.

Didayolo commented 4 weeks ago

Hi, thank you for reporting this issue.

@ObadaS Do you think we have some kind of 24h timeout setting in MinIO?

ObadaS commented 4 weeks ago

Hello, The MinIO link in the error does seem to have an expiry date at the end. I think that links created by the code have a set expiry time which seems to be lower than 24 hours. I think you would need to alter something in the code to request a longer expiry.

I will continue searching on my side and will update this issue if I find something else.

ObadaS commented 4 weeks ago

I seem to have found the code that creates the URLs https://github.com/codalab/codabench/blob/1bd7c6fdb6f2e457187f6f6e6bb994f74e2cca1a/src/utils/data.py#L76

The duration is 24h, so it should be possible to change it there, or make it variable somehow https://github.com/codalab/codabench/blob/1bd7c6fdb6f2e457187f6f6e6bb994f74e2cca1a/src/utils/data.py#L46

Didayolo commented 3 weeks ago

Very nice catch. Should we simply put a higher limit? For instance 4 days? Or can this have bad consequences?

ObadaS commented 3 weeks ago

Could it be possible to link it to the competition ? So that they can decide to make it higher if needed (up to a hard limit of course) without having every link for all competitions be upgraded to 4 days since it's rarely needed

Didayolo commented 1 week ago

Should be fixed by #1659, soon to be deployed