search

beyretb / AnimalAI-Olympics

Code repository for the Animal AI Olympics competition
Apache License 2.0
573 stars 84 forks source link

resolution changes have huge impact #61

Closed dan9thsense closed 5 years ago

dan9thsense commented 5 years ago

Bit of a rant here. When the competition rules were announced, they stated that the resolution would be 84x84 for submissions and that was fixed and not changing. Also, a time limit was set. To decide halfway through the competition to change such a fundamental parameter is unfair. Huge amounts of time have gone into training at 84x84, along with a lot of testing and other aspects of the code. Similarly, the time constraint was a consideration right from the start. Now that the basic aspects of the code are working, the second half of the competition time was planned to be spent exploring interesting avenues using that basic code. Instead, a tremendous amount of time and resources will have to be used to redo that initial work at various resolutions. Furthermore, this benefits groups with large computing resources and budgets. At 256x256, the network training needed for the existing code is very hard to do with our machines. I understand that we may get $500 for Amazon AWS, but that funding was going to be used to explore interesting new areas rather than just re-training at higher resolution. Furthermore, at 256x256, we will burn through that $500. pretty quick.

Unhappy here...........

jaggiK commented 5 years ago

Agree, it was starting to get interesting. Dont want to go back at this stage. It is best to stick to earlier setup.

ironbar commented 5 years ago

Can you elaborate more why do you think resolution can have a huge impact? From my point of view if the frames have enough information increasing the resolution won't give any advantage. The question is if 84x84 is enough for playing, I believe it is because I have played myself on that resolution and it seems enough.

mdcrosby commented 5 years ago

We thought that allowing for more resolutions would only increase possibilities for people entering the competition and that 84x84 would still be a good resolution to use, therefore not changing anyone's plans. However, we overlooked that this would be unfair to anyone who had invested time in 84x84 that would now want to have explored the other options. We had not intended this change affects those that have already been engaging with the competition and are therefore going to remove this from the evaluation script. You can still use multiple resolutions for training, but evaluation will require 84x84.

The time increase will remain. This should have very minimal effect as very few entries come even close to the time limit. It also allows for a wider variety of approaches to be entered. Lastly, it potentially helps those that have access to less resources compared to the resolution change which primarily helped those with access to more resources.

Please be aware that in our rules the final evaluation (In November not for AWS) may include extra variations on the tests (resources permitting, we will run them more than once with minor variations to get a more robust score) and therefore the final score may not be identical with the score on the current leaderboard.

dan9thsense commented 5 years ago

Thanks for responding, appreciate your understanding, sounds good.