search

duckietown / gym-duckietown

Self-driving car simulator for the Duckietown universe
http://duckietown.org
Other
51 stars 19 forks source link

Evaluation script needed #67

Closed jzilly closed 5 years ago

jzilly commented 6 years ago

Currently I cannot find an evaluation script of how good a Duckiebot is performing according to the AI-DO rules. The rewards are only a proxy to that score unfortunately. Will look into this myself but would appreciate some help from someone who knows the duckietown gym better.

For now an evaluation script for the lane following task is a good target.

maximecb commented 6 years ago

Sure, I can answer your questions regarding this.

Florian might want to be in the loop too. This script may need to be used by the launcher/server. Currently, @fgolemo, do you return the simulator's reward function to the client as-is? I guess this might be useful for RL, so we might want to keep that, but also have a mode where the AIDO score is evaluated?

jzilly commented 6 years ago

For now I wrote a simple logger that logs the relevant data (see pull request). With that data evaluation of the performance metrics and rules should be possible. As far as I can tell the code might work as is with the robotarium as well.

fgolemo commented 6 years ago

@maximecb yeah I'm just sending back the reward as-is. We should grab it definitely from the agent's side and not the server, because the server can keep running in between agent connections, thus making it impossible to trace which episode belongs to which agent.

bhairavmehta95 commented 5 years ago

Getting handled elsewhere.