[Question] Compatibility with VectorNet

[hzyjerry]

Question

It mentions in the paper that the state vector follows conventions of the VectorNet.

I was wondering if there's any attempt/code of training a VectorNet-based policy with Nocturne? Thinking about the amount of engineering it takes to hook nocturne up with some latest trajectory prediction models.

Thanks,

[eugenevinitsky]

Great question! We actually haven't tried this but I've really wanted someone to do it; I think it makes sense though it will slow down RL training a lot. I think if you wait long enough someone from my lab will probably implement it but it could be quite a while.

[hzyjerry]

Thanks! That makes sense.

Just curious is the code for processing WOMD data into vectorized states (VectorNet format) included? And in general, given a vectorized state, is it possible to convert it back to non-vectorized state formats (is any information lost)?

[eugenevinitsky]

Yeah it is! There's a set of tutorials on this on the lab fork of this: https://github.com/Emerge-Lab/nocturne_lab/blob/main/examples/01_data_structure.ipynb. There's an exact equivalence between the vectorized and non-vectorized state except for some additional padding in the vectorized state

[hzyjerry]

Thanks for the swift response. When running the dataloader in imitation learning the state dimension is 35110. In the paper, it mentions that the state dimension is 6727. Is there a place to study how the waymo dataset is converted to either of these two formats?

[eugenevinitsky]

Ah! By default in imitation learning the config stacks a history of 5 states so I think that's roughly what's going on there. Btw, the lab fork above should have a lot of other useful tutorials / documentation / organization that might be helpful for you

[hzyjerry]

Yup, the notebooks are super great! I wish other simulators have this :D

Just some follow-up:

1. Is there a way to turn off the occlusion?
2. Is the dataset essentially the same as WOMD? Has there been any processing/subsampling/vectorization on top of the map data in WOMD?

[eugenevinitsky]

1. At the moment, no. It'd be easy to turn off in the c++ (like a couple of lines) but I'm not certain when I'll have a chance to get to that.
2. It is identical to WOMD as far as I know with one exception; the dataset we have uploaded does not contain any of the scenes with traffic lights because the WOMD traffic lights are incomplete and are only observed from the perspective of the AV. Also, by default the cyclists and pedestrians are removed from the scenes but there's an arg to put them back. We subsample the lines at 0.1 m like they do. However, there's a PR that should land soon that replaces this subsampling for straight lines and should decrease the size of the state space by an order of 10 + sharply improve the simulator speed.

[hzyjerry]

That's very nice. Thanks for the response!