about FDE calculation

[ares89]

Thanks for sharing the work! The calculation of fde uses the last timestamp of gt trajectory , but it may not exist. https://github.com/roggirg/AutoBots/blob/3a61ad9f80603f99ab1e9cc535acccf72c5d6bea/utils/train_helpers.py#L97

Will this have a bad effect？

[roggirg]

Hi @ares89,

Yes that is correct, it would not work if you have incomplete trajectories. Note that this is for the ego-version of the model. If you are trying to have a model that predicts for an ego agent but without complete trajectories, I invite you to look at how the masking is done in the joint version: https://github.com/roggirg/AutoBots/blob/3a61ad9f80603f99ab1e9cc535acccf72c5d6bea/utils/train_helpers.py#L193

[ares89]

Thank you for your reply. In the joint version, the size of agent_mask is batch*time*agent, and agent_masks[:, -1:, :] also uses the last timestamp of the gt trajectory. For incomplete trajectories, it may not exist. So I think it's necessary to get the last exist timestamp from agent_mask. https://github.com/roggirg/AutoBots/blob/3a61ad9f80603f99ab1e9cc535acccf72c5d6bea/utils/train_helpers.py#L194

[ares89]

I try to add some codes to deal the incomplete trajectories scene.

1. add this code here https://github.com/roggirg/AutoBots/blob/3a61ad9f80603f99ab1e9cc535acccf72c5d6bea/datasets/nuscenes/dataset.py#L286

       ego_last_index = np.nonzero(out_ego[:, -1] == 1)[-1][-1]
       out_ego_fd_mask = np.zeros(out_ego.shape[:-1])
       out_ego_fd_mask[ego_last_index] = 1
       out_agents_fd_mask = np.zeros(out_agents.shape[:-1])
       for i in range(out_agents.shape[1]):
           res = np.nonzero(out_agents[:, i, -1] == 1)[-1]
           if len(res) > 0:
               out_agents_fd_mask[res[-1], i] = 1
       out_fd_mask = np.concatenate([np.expand_dims(out_ego_fd_mask,0).T, out_agents_fd_mask], axis=1)

2. add this code here https://github.com/roggirg/AutoBots/blob/3a61ad9f80603f99ab1e9cc535acccf72c5d6bea/utils/train_helpers.py#L194

   if out_fd_mask is not None:
       fde_loss = torch.zeros(data.shape[0], pred.shape[0]).to(pred.device)
       tmp = pred.permute(2, 1, 3, 0, 4)
       for i in range(data.shape[0]):
           fd = tmp[i, ...][out_fd_mask[i] == 1][..., :2]
           fd_data = data[i][out_fd_mask[i] == 1][..., :2].unsqueeze(1).repeat(1, 6, 1)
           fde_loss_batch = torch.norm((fd - fd_data), 2, dim=-1).mean(0)
           fde_loss[i] = fde_loss_batch
   else:
       fde_loss = (torch.norm((pred[:, -1, :, :, :2].transpose(0, 1) - data[:, -1, :, :2].unsqueeze(1)), 2, dim=-1) *
                   agent_masks[:, -1:, :]).mean(-1)

3. add extra code in train.py

[roggirg]

Hi,

Can you first clarify which dataset you are working with, and what version you are working on (ego vs. joint)? I'm guessing it's joint.

For all the datasets with a joint version, we pad all agent trajectories such that all agents end up (artificially) having the same number of timesteps, and each have an existence mask. So using the agent_mask as done in the joint version would work since if the agent does not exists in the last timestep, then it would not contribute to the FDE calculation.

Now, if you want to compute variable-length FDE, i.e., if some trajectories have 12 timesteps while others have 10 timesteps and you want to just compute the DE for the final timestep no matter what the length is, then yes you should probably do what you are suggesting. Have you tried implementing and training a joint (or ego?) model with these modifications? If it improves things, I'd be happy to see a PR.

Thanks!

[ares89]

Thank you for your reply. I am trying to work with the Waymo Motion Dataset using the joint mode.