search

mila-iqia / babyai

BabyAI platform. A testbed for training agents to understand and execute language commands.
BSD 3-Clause "New" or "Revised" License
700 stars 146 forks source link

BabyAI 1.1 #96

Closed dyth closed 4 years ago

maximecb commented 4 years ago

Looks good David 👍

I think we don't need the PyQT dependency anymore with the new MiniGrid btw. We could probably remove the relevant bits from the README.

dyth commented 4 years ago

Thanks @maximecb We didn't have PyQT in setup.py or environment.yaml, so I'm assuming that's a minigrid thing. In any case, I've removed the PyQT sections in README.md

dyth commented 4 years ago

@rizar My tests to reproduce the BabyAI 1.1 benchmarks have been successful. They consist of:

  1. RL, small arch, GoToRedBallGrey: original, original_endpool, original_endpool_res, bow_endpool_res, pixels_endpool_res
  2. IL, small arch, GoToRedBallGrey: original, original_endpool_res, bow_endpool_res
  3. IL. big arch, GoTo: original, original_endpool_res, bow_endpool_res
dyth commented 4 years ago

RL sample efficiency: ../beluga/logs/rl/GoToRedBallGrey/original_test 159999.0 ../beluga/logs/rl/GoToRedBallGrey/original_endpool_test 164785.0 ../beluga/logs/rl/GoToRedBallGrey/pixels_endpool_res_test 163982.0 ../beluga/logs/rl/GoToRedBallGrey/bow_endpool_res_test 164538.0 ../beluga/logs/rl/GoToRedBallGrey/original_endpool_res_test 162186.0

IL performance: ../beluga/logs/il/GoTo/original_endpool_res/10000_test 0.80322265625 ../beluga/logs/il/GoTo/bow_endpool_res/10000_test 0.95671875 ../beluga/logs/il/GoTo/original/10000_test 0.67263671875 ../beluga/logs/il/GoToRedBallGrey/original_endpool_res/5000_test 0.9903515625 ../beluga/logs/il/GoToRedBallGrey/bow_endpool_res/5000_test 0.9926953125 ../beluga/logs/il/GoToRedBallGrey/original/5000_test 0.99029296875