Predictive Streamers

[wangxiyang2022]

Excellent work! Combining Kalman filtering with trajectory prediction has always been something I wanted to delve into. I have tried in the past to use the predicted trajectory as the observation and directly update it in the CV model, and then use the updated state as the predicted trajectory. The experimental results were not very noticeable. The idea of introducing DF to learn observation noise in the paper is great, very reasonable. May I ask when the training method and configuration for Predictive Streamers will be released?

[ziqipang]

@wangxiyang2022 Thanks for your nice words and interest! If you need a reference for implementing a differentiable filter, you could check df.py in my code. My old server had a glitch so I might take more time to write the docs/training the checkpoints. Thst's why I leave that section as TBD in readme. But you can basically check this file I mentioned to have a sense of implementing differentiable filters for trajectories. Hopefully, this will be of help. Please feel free to ask if you have further questions.

[wangxiyang2022]

Thank you for the patient response. I currently have three questions about the df.py function.

Firstly, do single_modal_A and dynamics_A in lines 135 and 136 represent the state transition matrices? What model is being used (CV or CA or ...)? I have derived them, but they don't seem to conform to either case（Perhaps it's a misunderstanding on my part.）.

Secondly，actually, I believe that using the Kalman filter to update the predictive trajectory of the model should be iterative by frame. For example, when predicting the next 30 frames, the agent's current frame (t) should be used in the motion equation to predict the position of the next frame (t+1). Then, the position of the t+1 frame in the 30 frames predicted by the model should be used as an observation to update the Kalman filter's prediction. This iterative process is to be continued to eventually get the final trajectory, which is updated after df of the model's predicted trajectory. However, in the code, it directly predicts the next 30 frames and then updates, there seems to be no iterative process.

Thirdly, it was mentioned in the paper that the whole model training was divided into three stages. First, a normal predictor was trained, then it was fine-tuned using tracking data, and finally the prediction model was frozen and the df module was trained independently. During the training of the df module, is there any supervision on the learnable observation noise? Or is it just to refine the trajectories predicted by the model into the df module, and finally only supervise the refined trajectories?

Thank you again for your reply!

[ziqipang]

@wangxiyang2022 Thanks for the follow-up!

For the first part, I'll answer from two aspects:

• single_modal_A and dynamics_A in lines 135 and 136 represent the same state transition matrices. If you look down at line 144, the dynamics_A is cloned from single_modal_A. Then I applied this dynamic matrix to all the multi-future branches of motion forecasting.
• The dynamic matrix is described in Eqn. 8 in the paper, with the explanation for the design.

For the second part:

• My KF treats the whole future trajectory as the hidden state of KF, instead of just single-frame positions. More specifically, the state_dim of line 128 has dimension $T\times 2$ instead of $2$.
• So you might wonder where does the iterative update happen? The forward function in df.py is the operation on a single frame, and I call this function on each frame via here.

For the third part: Yes, I only supervise the final trajectories just like other differentiable filtering works, because the gradient flows back to the noise matrix.

[wangxiyang2022]

Thank you for your reply, it's very enlightening!

[ziqipang]

@wangxiyang2022 Do you mind if I pin this issue since a lot of people might be interested in the questions you raised?

[wangxiyang2022]

No problem！