Camera images for the SDC?

[atnachkov]

Is there a way to get camera images for the log trajectory of the self-driving car in each scenario?

I imagine that the expert trajectories are the same as those on which the Waymo sensor dataset is based? So, in principle, one should be able to obtain the camera images for the expert SDC trajectory.

If yes, what are the steps to merge the images with the other Waymax state data?

[nicomon24]

Hi @atnachkov,

As Waymax is based on the Waymo Open Motion dataset, we have not release the camera images for the expert SDC trajectory as a whole.

In a recent change to WOMD, we have added both camera (embeddings, not full images) and lidar for the first second of history (but this hasn't been integrated into Waymax). We have tutorials on how to use this data (https://github.com/waymo-research/waymo-open-dataset/blob/master/tutorial/tutorial_womd_camera.ipynb and https://github.com/waymo-research/waymo-open-dataset/blob/master/tutorial/tutorial_womd_lidar.ipynb).

[atnachkov]

Hi @nicomon24, thank you for your reply. From this reference I see that the camera embeddings have been obtained using a pretrained VIT-VQGAN model. Do you know if it's available to the public? Can we use it to decode the tokens back into images?

[nicomon24]

Hi @atnachkov, we are not currently planning on releasing the weights of this model, so you only have access to the embeddings. We are discussing future releases of full camera images but I don't have a timeline for that.