Problem with rotation in custom dataset class

[FrancescoMandru]

I'm dealing with a custom class for my problem which simplify my life. It doesn't change anything and I call it after EgoDataset invocation and before loading the final result into a data loader.

class TransformDataset(Dataset):
    def __init__(self, dataset, cfg):
        self.cfg = cfg
        self.dataset = dataset
        self.W = self.cfg['raster_params']['raster_size'][0]
        self.H = self.cfg['raster_params']['raster_size'][1]
    def __getitem__(self, index):
        batch = self.dataset[index]
        return self.transform(batch)

    def __len__(self):
        return len(self.dataset)

    # Here batch is just 1 element
    def transform(self, batch):

        return (batch["image"].astype(np.float32),
                batch["target_positions"].astype(np.float32),
                batch["target_availabilities"].astype(np.float32),
                batch['history_positions'].astype(np.float32),
                batch['history_yaws'].astype(np.float32),
                batch['target_yaws'].astype(np.float32),
                batch['target_velocities'].astype(np.float32),
                batch['history_velocities'].astype(np.float32),
                batch["centroid"].astype(np.float32),
                batch["world_to_image"].astype(np.float32)
                )

The reference code is here https://github.com/d-eremeev/SC-GAN/blob/971c169899bd76beee532bcbc8620370a3906f61/utils.py#L400. Theysaid to keep attention for newest versions of l5kit and I removed the changes of coordinates that they have done because probably in the previous versions there were some bugs which now are ok.

However, during a training session I noticed something wrong, maybe I missed something about the trajectory rotation? They seems to be flipped.

In the last one it is also visible a double ground truth trajectory.

[lucabergamini]

it might be (and it's very speculative) an issue with the flag set_origin_to_bottom in the config. Have you tried to invert it? I'm positive about trajectories being well aligned in master, so it looks like you're doing a wrong transformation somewhere..

[FrancescoMandru]

@lucabergamini the problem is that the model I'm using works with an older version of l5kit (don't know which one) but this probably can help you. When they have to plot the trajectories and the predictions they do the following:

gen_batch = predictor(image, actor_state, noise)

xy = gen_batch[batch_id].cpu().detach().numpy()  # trajectory

W = cfg['raster_params']['raster_size'][0]
# agent's position (in pixels)
agent_position_image = np.array([cfg['raster_params']['ego_center'][0] * W,
                                 cfg['raster_params']['ego_center'][1] * W])

# transform to pixels by division on r and then translate
r = cfg['raster_params']['pixel_size'][0]
target_positions_pixels_pred = agent_position_image + xy/r  # (target_size, 2)
target_positions_pixels_history = agent_position_image + history_positions[batch_id].cpu().numpy()/r
target_positions_pixels_future = agent_position_image + target_positions[batch_id].cpu().numpy()/r

And the trajectories are mirrored as you can see in the images above. I wonder which is the simplest way to apply the right rotation to the raw trajectories, I would like to do that on my transform dataset class. I can't just use transform_points because I need coordinates in their original values as the network needs them in that way. My attempt is the following:

batch['target_positions'][:,1] =  cfg['ego_center'][1] - batch['target_positions'][:,1] #cfg['ego_center'][1] = 0.5
batch['history_positions'][:,1] = cfg['ego_center'][1] - batch['history_positions'][:,1]

is it resonable in your opinion or I totally missing it? because now the orientation seems correct, the only thing is that sometimes the trajectory (ground truth and predicted are ouside the image and it is not perfectly centered.

Sometimes some frames are very weird:

[lucabergamini]

because now the orientation seems correct, the only thing is that sometimes the trajectory (ground truth and predicted are ouside the image and it is not perfectly centered.

In general the transform will only convert the coordinates to the destination space but it won't apply any clipping. So it's possible to have trajectories overspilling I would say.

Regarding the centre I would expect them to be centred on Y..

with an older version of l5kit

A lot of things have changed, so it's perfectly possible that you're getting weird behaviours if you're using a different version of L5Kit

[FrancescoMandru]

In general the transform will only convert the coordinates to the destination space but it won't apply any clipping. So it's possible to have trajectories overspilling I would say.

So is this an expected behaviour?

A lot of things have changed, so it's perfectly possible that you're getting weird behaviours if you're using a different version of L5Kit

Yes and I'm using the latest one, the only thing is that the model was built taking into consideration an older version so the input representation is a bit different and as they reported:

There were some troubles with rotation angle for history/target positions in l5kit dataset which required additional change of coordinates in code. This transformation should be included in later versions of l5kit, which leads to small changes in transform method of TransformDataset class.

So I deleted their adaptation as the new version of l5kit already done pre-processing. I'm trying to adapt the input with the one needed by the model. It's almost done, I have some troubles only with the rotation and I tried to fix it with the approach in my previous comment, is it enough in your opinion? Am I missing something?

[lucabergamini]

So is this an expected behaviour?

yeah, definitely. It's just a visual issue I suspect

[FrancescoMandru]

@lucabergamini The version used in the network I'm trying is l5kit==1.0.6.

and this is my adaptation for the angle.

batch['target_positions'][:,1] =  cfg['ego_center'][1] - batch['target_positions'][:,1] #cfg['ego_center'][1] = 0.5
batch['history_positions'][:,1] = cfg['ego_center'][1] - batch['history_positions'][:,1]

[lucabergamini]

This does not seem to change the angles but the positions.

What you should do is:

• use an RT matrix to transform positions;
• correct the yaw (trajectories only) by adding/subtracting the reference angle

Don't subtract/add positions in general, as you're not taking into account the rotation angle by doing it. It may work for some spaces, but will fail eventually.

If what you're after is a flip on X, there is a flag which will correctly flip the RT matrix for you

[FrancescoMandru]

@lucabergamini Ok I removed that part of code and it seems that setting the flag to False has flipped the trajectories accordingly. I will notify new bugs if something happen. Sometimes the plotted trajectories are outside the rasterized image. How can I change the code to solve it?

gen_batch = predictor(image, actor_state, noise)

xy = gen_batch[batch_id].cpu().detach().numpy()  # trajectory

W = cfg['raster_params']['raster_size'][0]
# agent's position (in pixels)
agent_position_image = np.array([cfg['raster_params']['ego_center'][0] * W,
                                 cfg['raster_params']['ego_center'][1] * W])

# transform to pixels by division on r and then translate
r = cfg['raster_params']['pixel_size'][0]
target_positions_pixels_pred = agent_position_image + xy/r  # (target_size, 2)
target_positions_pixels_history = agent_position_image + history_positions[batch_id].cpu().numpy()/r
target_positions_pixels_future = agent_position_image + target_positions[batch_id].cpu().numpy()/r

I tried to copy the functions used in the agent motion notebook to plot the trajectories but they don't fit good into my plot function

[lucabergamini]

is there any reason why you're not using raster_from_agent or raster_from_world here to get the pixel coordinates?

Anyway, if you want to clip the coordinates in raster space you just need to apply something like np.clip, e.g:

coords = np.arange(100).reshape(50,2) # assuming XY
W, H = 25, 45
coords  = np.clip(coords, a_min=(0,0), a_max=(W, H))

[FrancescoMandru]

@lucabergamini I tried to use raster_from_agent or raster_from_world but probably I misunderstood their meaning . The clip is smart and I don't know why I didn't think about that, m fault. Really Thanks Luca

[lucabergamini]

np @FrancescoMandru, can we close this then?