search

waymo-research / waymo-open-dataset

Waymo Open Dataset
https://www.waymo.com/open
Other
2.66k stars 608 forks source link

Number of scenarios for Motion Forecasting Challenge test submission #744

Open ArielleZhang opened 10 months ago

ArielleZhang commented 10 months ago

Hi,

When I submit test results to the Motion Forecasting Challenge, I noticed if you drop several scenarios, for example 30, the tester will not give an error, but if you drop too many, for example keep only 128 scenarios, it would error out. Is there a buffer zone for the number of dropped scenarios? If there is, what number exactly it is?

Best,

Arielle

scott-ettinger commented 10 months ago

You should submit with all of the scenarios in the dataset. The server has a very small allowance for missing scenarios. This will likely be removed in near term updates. Could you tell us why you want to submit entries with missing scenarios?