How is the accuracy calculated ? Its mentioned that it is averaged over 100 episodes, but averaged over which 100 episodes ? Is it from 1 - 100 ?

devendrachaplot / DeepRL-Grounding

Train an RL agent to execute natural language instructions in a 3D Environment (PyTorch)

https://sites.google.com/view/gated-attention/home

MIT License

237 stars 37 forks source link

How is the accuracy calculated ? Its mentioned that it is averaged over 100 episodes, but averaged over which 100 episodes ? Is it from 1 - 100 ? #5

Closed rshivansh closed 6 years ago

devendrachaplot commented 6 years ago

I am not sure if I understand your question, but accuracy is averaged over 100 separate test episodes at the end of training.

rshivansh commented 6 years ago

what is the accuracy returned by running python a3c_main.py --evaluate 1 --load saved/pretrained_model Sorry for such basic questions ! I am still a beginner.

devendrachaplot commented 6 years ago

The accuracy can be anywhere between 0.8 and 1.0 because the episodes are generated randomly.

rshivansh commented 6 years ago

Yes this is what i wanted to confirm ! So it means once my model is trained , while testing it I test it over any 100 test episodes and then report the avg value right ? Thanks

devendrachaplot commented 6 years ago

Yes, that's correct.

rshivansh commented 6 years ago

Thanks @devendrachaplot

rshivansh commented 6 years ago

A silly question, @devendrachaplot it is not an issue, just wanted to know that I am currently training my own model for same work (visual grounding) on your modified environment and I am receiving avg reward as 0 for quite a long time now, so could you let me know if this actually happens at the start of training time. Thanks

devendrachaplot commented 6 years ago

The average reward shouldn't be 0. If it's not an issue with the code, you can email me with your questions, attaching the log file and the command you used to run the code.