search

facebookresearch / habitat-challenge

Code for the habitat challenge
https://aihabitat.org
MIT License
309 stars 56 forks source link

"standard_error" on submission #49

Closed devendrachaplot closed 4 years ago

devendrachaplot commented 4 years ago

Hi, My submission to the evalai portal failed and it is showing just "standard_error" in the Stderr File. Could you help in figuring out the error?

mathfac commented 4 years ago

Hi Devendra, that's not expected behavior of EvalAI and will be fixed. We will try our best to provide you error logs.

RishabhJain2018 commented 4 years ago

Hi @devendrachaplot , your submission to the minival phase is timing out as it isn't able to run through all the episodes in 30 mins. (We will fix the error message at our end to be more meaningful.)

Please close this issue if this resolves your query.

dhruvbatra commented 4 years ago

It is easy to bring your run-time down? We can bump the time-out limit but it will reduce your iteration cycles when you get to test-std (which is 100x larger than minival).

devendrachaplot commented 4 years ago

It will be great if the time-out limit can be increased! The local evaluation takes around 16 mins on my machine. Is there a way to see the output of the model? Our submission prints FPS. I am trying to figure out the ratio of FPS from my machine to remote evaluation.

dhruvbatra commented 4 years ago

We're working on all of these points.

devendrachaplot commented 4 years ago

Thanks, the time limit issue was mostly because of inconsistent docker base image (https://github.com/facebookresearch/habitat-challenge/issues/33#issuecomment-629741382). Fixing the base docker image reduced the runtime by around 50%.

Also, just for your reference, the remote eval (with the fixed docker base image) is now taking around 2x time as compared to my local evaluation on with 1080 Ti GPU (14 mins vs 7 mins).