Why do I get different SR with SR score in the paper?

soyeonm / FILM

Official repository of ICLR 2022 paper FILM: Following Instructions in Language with Modular Methods

114 stars 27 forks source link

Why do I get different SR with SR score in the paper? #22

Open uyeongkim opened 1 year ago

uyeongkim commented 1 year ago

I ran the experiment just like the command on github, and I used trained model weight on Google drive. However, I could not reproduce the number shown in the paper. Is there room for random numbers to change in code? If not, I would like to know why I cannot re-produce the number reported in the paper. For reference, I got SEEN/UNSEEN SR: 26.48/25.05 from the applied (high+low instruction) and new_best semi-mapping weight setting.

TopCoder2K commented 1 year ago

@uyeongkim, I was just looking around and encountered you question. I'm not sure about FILM's deterministic execution (I also wasn't able to reproduce the results), but AI2THOR itself is a source of randomness. See: https://github.com/askforalfred/alfred/issues/131 and https://github.com/askforalfred/alfred/issues/134#issuecomment-1427749977 ...