Understand and Visualize the Waypoint Predictions

[animikhaich]

I am trying to visualize and understand the waypoint predictions from the TF++ model.

I added the following piece of code to team_code/sensor_agent.py to try and visualize the Waypoints and the corresponding control translations via PID:

# Print Controls
print(f"[Model] Throttle: {throttle} | Steer: {steer} | Brake: {brake}")

# Inits
fig = plt.figure()
canvas = fig.canvas
ax = fig.gca()

# Plotting
pts = np.squeeze(self.pred_wp.detach().cpu().numpy())
ax.scatter(pts[:, 0], pts[:, 1], cmap='viridis', c=range(pts.shape[0]), s=100)

# Visualization
ax.axis('off')
canvas.draw()
image_flat = np.frombuffer(canvas.tostring_rgb(), dtype='uint8')
image = image_flat.reshape(*reversed(canvas.get_width_height()), 3)
raw_rgb = tick_data['rgb'].squeeze().permute(1, 2, 0).cpu().numpy()
cv2.imshow('Image', raw_rgb.astype(np.uint8)[:,:,::-1]) # The RGB Input
cv2.imshow('waypoints', image) # The Waypoints Plot
cv2.waitKey(1)

I am trying to visualize the live waypoints either in a BEV space similar to Figure 2(b) in the paper, or in the RGB camera similar to this Video by Wayve.

However, when I plot the raw waypoint values, they don't correspond to the path ahead (as shown in the two images below, where the road is straight, yet the waypoint plots differ completely).

Please explain the raw waypoint predictions of the model and how to interpret/visualize them. Also, if there is any transformation that needs to be done, please share a code snippet for the same.

Thanks!

Screenshot 1:

Screenshot 2:

[Kait0]

I would recommend you use the existing visualization code by setting the DEBUG_CHALLENGE environment flag to 1. As well as a SAVE_PATH which tell the code where to save the generated images.

[Kait0]

If you want to see the GT map instead of predicted semantic segmentation you need to uncomment it here and add comments to the bev semantic segmentation.