Test set performance

[Sungmin-Woo]

Hi @ZikangZhou Could you describe the training recipe used for the submission? I got the test results with your model as below but there seems quite large gap compared to the result you got.

minFDE (K=6): 1.277136966106996 MR (K=6): 0.14637267573551055 minADE (K=6): 0.8215039475137975

As it is hard for me to reach your result by just increasing the training epochs, it will be very helpful if you can provide some other hints. I am looking forward to your reply!

[XianrenYty]

same question, here is my result:

• minFDE (K=6)": 1.2320899802561445
• "MR (K=6)": 0.1369028575816132
• "minADE (K=6)": 0.7993995795805748

[MasterIzumi]

Hi, @XianrenYty @Sungmin-Woo Could you please provide the corresponding results of the validation set? I am curious about the performance gap between val set and test set for the same model. By the way, are the results you provide come from the repo (the provided ckpt) or re-training? Thanks!

[Sungmin-Woo]

Hi, @XianrenYty @Sungmin-Woo Could you please provide the corresponding results of the validation set? I am curious about the performance gap between val set and test set for the same model. By the way, are the results you provide come from the repo (the provided ckpt) or re-training? Thanks!

I verified similar results on validation set as reported in the paper: minADE - 0.66, minFDE - 0.97 For the test set, both the provided checkpoint (ckpt) and the re-trained model exhibit similar performance, which is quite inferior.

[YueYao-bot]

Hi @Sungmin-Woo ,

I also used the HiVT-128 checkpoint provided in this github. I got very similar results with the test split as you mentioned:

FDE k6: 1.2816903181056214 MR k6: 0.147588395633646 ADE k6: 0.8216046525410116

This is not comparable with the results in the paper and I'm curious what could be the reason.

Have you found out the reason?

[Sungmin-Woo]

Hi @Sungmin-Woo ,

I also used the HiVT-128 checkpoint provided in this github. I got very similar results with the test split as you mentioned:

FDE k6: 1.2816903181056214 MR k6: 0.147588395633646 ADE k6: 0.8216046525410116

This is not comparable with the results in the paper and I'm curious what could be the reason.

Have you found out the reason?

No, I failed to find out the reason and moved on to other project recently. Please let me know here if you find out.

[Porphy]

Hi @Sungmin-Woo , I also used the HiVT-128 checkpoint provided in this github. I got very similar results with the test split as you mentioned: FDE k6: 1.2816903181056214 MR k6: 0.147588395633646 ADE k6: 0.8216046525410116 This is not comparable with the results in the paper and I'm curious what could be the reason. Have you found out the reason?

No, I failed to find out the reason and moved on to other project recently. Please let me know here if you find out.

Would you like to provide your Leaderboard Submission Code？Thanks a lot @MasterIzumi @XianrenYty @Sungmin-Woo @YueYao-bot

[YueYao-bot]

Hi @Porphy, you could try this:

import matplotlib.pyplot as plt
import numpy as np
import os

from itertools import permutations, product
from typing import Tuple, List, Dict
import json
from tqdm.auto import tqdm

import torch

from argoverse.map_representation.map_api import ArgoverseMap

import sys
sys.path.append('HiVT')
sys.path.append('HiVT/datasets')

from models.hivt import HiVT

from datasets.argoverse_v1_dataset import process_argoverse, get_lane_features, ArgoverseV1Dataset
from utils import TemporalData

from argoverse.map_representation.map_api import ArgoverseMap
from argoverse.data_loading.argoverse_forecasting_loader import ArgoverseForecastingLoader

checkpoint_path = 'checkpoints/HiVT-64/checkpoints/epoch=63-step=411903.ckpt'
model = HiVT.load_from_checkpoint(checkpoint_path=checkpoint_path, parallel=False)

split = 'test'
dataset = ArgoverseV1Dataset(root='data', split=split, local_radius=model.hparams.local_radius)

output_all_k6 = {}
probs_all = {}
for i, inp in enumerate(tqdm(dataset)):
    x = inp.x.numpy()
    y = inp.y.numpy()
    seq_id = inp.seq_id
    positions = inp.positions.numpy()

    # the location of the ego vehicle at TIMESTAMP=19
    origin = inp.origin.numpy().squeeze()

    # the index of the focal agent
    agent_index = inp.agent_index

    # ego_heading at TIMESTAMP=19 
    ego_heading = inp.theta.numpy()

    ro_angle = inp.rotate_angles[agent_index].numpy()

    # Global rotation to align with ego vehicle
    rotate_mat = np.array([
        [np.cos(ego_heading), -np.sin(ego_heading)],
        [np.sin(ego_heading), np.cos(ego_heading)]
    ])

    R =  np.array([
                    [np.cos(ro_angle), -np.sin(ro_angle)],
                    [np.sin(ro_angle), np.cos(ro_angle)]
                ])

    # we recover the agent trajectory from the inputs, just as a sanity check
    offset = positions[agent_index, 19, :]
    hist = (np.cumsum(-x[agent_index, 20::-1, :], axis=0)[::-1, :] + offset) @ rotate_mat.T + origin
    fut =  (y[agent_index, :, :] + offset) @ rotate_mat.T + origin

    res, res_pi = model(inp)
    agt_res = res[:, agent_index, :, :].detach().cpu().numpy() # [6, num_agents, 30, 2]

    probs = torch.softmax(res_pi[agent_index], dim=0)

    agt_res_origin = (agt_res[:, :, :2] @ R.T + offset) @ rotate_mat.T + origin

    probs_all[seq_id] = probs.detach().cpu().numpy()
    output_all_k6[seq_id] = agt_res_origin

from argoverse.evaluation.competition_util import generate_forecasting_h5

output_path = 'competition_files/'

generate_forecasting_h5(output_all_k6, output_path, probabilities= probs_all, filename = 'HiVT64') #this might take awhile

[Porphy]

Thank you most sincerely. @YueYao-bot

[Porphy]

same question, here is my result:

• minFDE (K=6)": 1.2320899802561445
• "MR (K=6)": 0.1369028575816132
• "minADE (K=6)": 0.7993995795805748 @XianrenYty Would you like to provide your Leaderboard Submission Code？I have tried the code that YueYao-bot provided for me, but the result is much worse than yours. And I'm curious about the reasons.Thanks.

[YueYao-bot]

Hi @Porphy , the code I provided is HiVT-64. However, the results we listed are HiVT-128, just in case, if you didn't notice it.

[Porphy]

Hi @Porphy , the code I provided is HiVT-64. However, the results we listed are HiVT-128, just in case, if you didn't notice it.

Thanks. I did notice that. But I'm very curious about why his results are significantly better than yours, and I don't think it's because of training instability.

[YueYao-bot]

Hi @Porphy, I guess this might be achieved by increasing training epochs or using "train + val" data for training. I am currently trying this out.

[Porphy]

Hi @Porphy, I guess this might be achieved by increasing training epochs or using "train + val" data for training. I am currently trying this out.

Thanks a lot. Please let me know the results if you finish trying this strategy.

[YueYao-bot]

@Sungmin-Woo , @Porphy , I notice that the model should use "model.eval()" to deactivate all dropout layers before validation. With this, the result improves a lot.

[Porphy]

@YueYao-bot I just tried it out and was able to achieve the desired results as XianrenYty provided. Thank you very much.

[Sungmin-Woo]

@YueYao-bot Thank you for sharing it!

[wangqc1001]

@YueYao-bot Hello, is this the test code you tried to write by yourself? Could you share with me how to use the trained model and visualize the predictions while reading the test set? There is no test.py file in the open source of the paper, so does the test process need to be found in Argoverse-API?