search

askforalfred / alfred

ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
MIT License
360 stars 77 forks source link

Question regarding the leaderboard.py code #74

Closed soyeonm closed 3 years ago

soyeonm commented 3 years ago

Hello,

I have a question on submitting the performance of a certain model to the leaderboard using the leaderboard.py code.

Can we write our own leaderboard code as long as

  1. It receives the same inputs
  2. It outputs the same json format?

I am asking this because I have a model that is significantly different from the lstm model of the original paper, and the following lines

m_out = model.step(feat) m_pred = model.extract_preds(m_out, [traj_data], feat, clean_special_tokens=False) m_pred = list(m_pred.values())[0]

do not really hold for my model. Also, I would like to add other lines to the leaderborad.py, because of how I have implemented the memory of my model.

Thanks!

MohitShridhar commented 3 years ago

Can we write our own leaderboard code as long as

  1. It receives the same inputs
  2. It outputs the same json format?

Yes! As stated in the Leaderboard Rules, as long as you use the same inputs (RGB + Goal & Step-by-Step Language) and the output format of the JSON is consistent with leaderboard.py, it should be good to go.