search

ATR-DBI / ScanQA

Other
102 stars 3 forks source link

EvalAI Submission Error #11

Closed haomengz closed 1 year ago

haomengz commented 1 year ago

Hi,

Thanks for the nice work.

I followed the instructions in README file and tried to submit the json files trained from your original code. The EvalAI shows the normal stdout but there is output from stderr file and the status is shown as failed. I am wondering whether you could check the error. Thanks!

miyatai2 commented 1 year ago

Hi,

Thanks for letting us know. The leaderboard worked fine until 2/5, but it seems not to recently have work. We do not know what the cause is. We are investigating it now.

haomengz commented 1 year ago

Thanks for the reply. Please let me know when it is fixed.

shuheikurita commented 1 year ago

Hi haomengz,

Thank you for having interests in our ScanQA paper! I'm an author and maintainer of ScanQA leaderboard. We notice that the java used in the SPICE evaluation suddenly fails after 5th Feb. I stopped the SPICE evaluation (it always becomes 0 for now...) Can you try the leaderboard? Hope you can obtain scores except SPICE now.

haomengz commented 1 year ago

Hi haomengz,

Thank you for having interests in our ScanQA paper! I'm an author and maintainer of ScanQA leaderboard. We notice that the java used in the SPICE evaluation suddenly fails after 5th Feb. I stopped the SPICE evaluation (it always becomes 0 for now...) Can you try the leaderboard? Hope you can obtain scores except SPICE now.

Hi Shuhei, I have tried the leaderboard again. It works well now. Many Thanks.