Closed johanneskruse closed 4 months ago
Hello, I am having the same problem. Any possible solution? or is it a bug? Thank you very much!
@Didayolo I think this is because of the latest changes
Do you think 600 seconds is not enough? MAX_EXECUTION_TIME_LIMIT=600
Hi @johanneskruse @aporrasc,
Indeed, we just merged the following change: #1154
It limits the execution time on the default queue (the queue of workers shared by default by all competitions), in order to avoid congestion with long jobs.
The limit is currently set to 600 seconds, maybe it is a bit too extreme?
For now:
Your feedback on this matter is appreciated in any case. What is the usual duration of the jobs in your competitions?
Post-it: the Execution time limit exceeded
message should be more informative in the case the limitation comes from the use of the default queue.
I added a clarification in the front page message: https://www.codabench.org/
Hi @Didayolo,
Thank you for getting back to me on this matter.
I understand why this is being done; however, it is very unfortunate for our ongoing competition. Our competition demands more in terms of evaluation time (approximately 3 hours), and since it is already ongoing, we cannot optimize or change the scoring program at this point.
We have set up a remote worker (thank you for the easy step-by-step tutorial!), but it is quite slow because each submission is processed sequentially on a single worker. Is there a way to utilize multiple remote workers for the same competition, or to parallelize the jobs on the single remote worker? This would be a great help!
Again, thank you for the great Codabench platform!
Any help on setting the compute worker?
I am able to run the compute_worker container using: _docker run -v /codabench:/codabench -v /var/run/docker.sock:/var/run/docker.sock -d --env-file .env --name computeworker --restart unless-stopped --log-opt max-size=100m --log-opt max-file=3 codalab/competitions-v2-compute-worker:latest in Shell.
A couple of questions raise: 1) Should there be something in a local folder called "codabench"? 2) I submit a test but never starts. I am using the same Broker URL.
Any easy-to-follow tutorial?
Thanks in advance!
@aporrasc
.env
I have to say I put "BROKER_USE_SSL=True" in the .env file.
Any advice?
Thanks.
Finally, I got it.
But even now in my own computer as compute worker the time limit is set to 600 seconds.
Where can I modify this? Becuase I only know one way to do it, in the competition.yaml.
Thanks.
You can change the time limit in your phase settings
@johanneskruse Sorry for the inconvenience for your competition, and thank you for your understanding.
You absolutely CAN setup several workers in order to parallelize the computation. You can simply follow the exact same procedure on several computers (VMs or physical machines). Use the same BROKER_URL
, so the different workers will be listening to the same custom queue (see schema).
Hi,
The 'Execution Time Limit' is fixed at 600 seconds, and I cannot increase it. My competition has been running for a few weeks with the limit set to more than 600 seconds, but today my submissions started to fail.
These are two valid submissions, but from one hour to the next, they cannot exceed 600 seconds:
I am still able to decrease the 'Execution Time Limit':
Is there an explanation or a fix for this?
Link to Ekstra Bladet News Recommendation Competition
Thank you very much!