search

carla-simulator / leaderboard

CARLA Autonomous Driving leaderboard
MIT License
156 stars 76 forks source link

Submission pending #145

Closed jdmdj1999 closed 2 months ago

jdmdj1999 commented 1 year ago

Hello, my new submission has been pending for a long time, and it still takes up my computing time. Is there a problem with the leaderboard?

Sou0602 commented 1 year ago

@jdmdj1999, I am trying to submit to the leaderboard and have been facing the same issue for a few days now. Upon trying to run run_evaluation.sh locally on the docker, the leaderboard evaluator times out as it cannot connect to the CARLA server. I am not sure how the leaderboard evaluation procedure connects to the server. When I run locally, I run the CARLA server on one terminal (./CARLAUE4.sh) and the run_evaluation.sh on the other terminal. But in the docker container, I don't see a (./CARLAUE4.sh) in the standard make_docker.sh file to test the container locally. Can someone give more details on how the leaderboard submission connects to a carla server, and which routes and scenarios files are used to test locally, if there is a problem with the leaderboard?

jdmdj1999 commented 1 year ago

@Sou0602 , I think 'run run_evaluation.sh locally on the docker, the leaderboard evaluator times out as it cannot connect to the CARLA server' is normal, I was in the same situation before, but I was able to submit the image and run the test normally

Sou0602 commented 1 year ago

@jdmdj1999 so were you able to submit the image to the leaderboard without any pending status and finish the test with a reported score ? I keep encountering a problem where the test runs for a long time(~150 hours) and says finished with an incomplete progress

Sou0602 commented 1 year ago

@jdmdj1999 I resolved the error of running the evaluation on docker locally, but I am still facing the problem of submission pending to the leaderboard submission. How did you overcome that?

jdmdj1999 commented 1 year ago

@Sou0602 I haven't submitted it successfully until last week. The administrator says they have a maxmimum amount of submissions running at the same time for all users, so if others have also submitted one, it might take a bit for all submissions to start. That's what you see as pending. And please do wait a bit, the submission queue is in general occupied, so you'll have to wait for some time before your submission is able to start. I haven't tried to submit it recently.