Command line tool for evaluation of interaction/motion prediction challenge metrics

[lassepe]

Are there any plans to add code for a binary that computes the metrics for the interaction prediction challenge (or, likely equivalently, the motion prediction challenge) for a given serialized prediction in the MotionChallengeSubmssion format?

Specifically, there is this command line tool for the detection challenge which takes the prediction.bin and a labels.bin and returns the metrics results for the detection challenge. Will there be a similar command line tool for the interaction/motion prediction challenge in the near future (within the time frame of the ongoing challenge)?

[s-ettinger]

Hi Lasse, Thanks for the feedback. We did not provide this tool partly because the size of the ground truth data makes this difficult to compute in a command line tool. I can recommend two alternatives for now. The submission server is now open on the Waymo Open Dataset website for both challenges. Each submission will provide you with a results page under the "My Submissions" menu. There is not a limit on how often you can submit a validation set submission. You are limited on the test set to 3 times a month. The other possibility is to use the tensorflow metrics op wrapper provided in the code (see the tutorial) to evaluate metrics during training.

We will consider the request to provide a command line tool.