Evaluating Agents against Environment at Scale

[krtkvrm]

Evaluating AI Agents against Environment at Scale

Description

Talk about my project that will help users to evaluate Agents against Environment at Scale.

Duration

• [x] 30 min

Outline

• The rise of reinforcement learning based problems or any problem which requires that an agent must interact with an environment introduces additional challenges for benchmarking. In contrast to the supervised learning setting where performance is measured by evaluating on a static test set, it is less straightforward to measure generalization performance of these agents in the context of the interactions with the environment. Evaluating these agents involves running the associated code on a collection of unseen environments that constitutes a hidden test set for such a scenario. This project deals with seting up a robust pipeline for uploading prediction code in the form of Docker containers (as opposed to test prediction file) that will be evaluated on remote machines.
• Presentation: http://bit.do/gsoc-cloudcv

Additional notes

• Product Engineer at Gojek, previously worked as Google Summer of Code Developer at CloudCV and as intern at BookMyShow

[Sangarshanan]

Hey @vkartik97 Would you available to talk on 18th ?

[krtkvrm]

Yes!

[TrigonaMinima]

@vkartik97 will you be available for the talk on 29th Feb?

[krtkvrm]

@TrigonaMinima I am not available for talk on that date.