search

rainmaker22 / SMART

[NeurIPS 2024] SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction
Apache License 2.0
62 stars 9 forks source link

Details about map tokens #16

Open OrangeSodahub opened 3 days ago

OrangeSodahub commented 3 days ago

Hi, I want to figure out some details about your map tokens.

I checked the map_traj_token5.pkl file which contains two keys:

"traj_src": shape (1024, 11, 2),
"sample_pt": shape (1024, 3, 2)

Could you explain more about what these attributes stand for? I'm confused about the second dimension: 11 and 3. Here 3 also comes from the map tokenization process.

And what is the difference between traj_src and sample_pt, could you give some advice?

cc @rainmaker22 @ziyanGao

OrangeSodahub commented 1 day ago

Another question is, next token prediction for map seems not be used in current version of codes. Is that right?

e.g.: here map_next_token_prob and map_next_token_idx_gt are never used. According to your paper, you need this prediction (given predecessor map token) to supervise model learning topology of maps. https://github.com/rainmaker22/SMART/blob/a329361b63082359be56c9bfaa7e76336c19115f/smart/modules/map_decoder.py#L133-L139

cc @rainmaker22 @ziyanGao