Aloception transforms may create invalid pixels

[Dee61298]

Some transforms in Aloception such as Rotate and Spatial Shift may create invalid pixels in the frame, that is, pixels that don't carry any meaningful data (example attached). These invalid pixels are currently not flagged in any way, so it's possible that someone might use a frame with invalid pixels by error. One possible fix would be that these transforms first add a .mask to the frame if there isn't one already, and when the transform is applied, the mask takes the value 1 where a pixel is invalid. Idea of @jsalotti

[thibo73800]

Yes, agree, it could work. I think you can do it @Dee61298

[Dee61298]

Update : just changing the implementations of the functions _rotate and _spatial_shift for the Mask class is not enough. If I use the batch_list function after my transform, the current implementation of batch_list recreates a new .mask, and therefore all my changes related to the transform are lost (ref here)

I changed the implementation so that the new .mask take into account the existing .mask (ref here). However when I use that, my network (Deformable2DPoints) behaves badly.

Frame and points without the new batch_list implem:

Values contained in the .mask :

print("Unique",torch.unique(n_mask))

Unique tensor([0.])

Points with the new batch_list implem :

Values contained in the .mask :

print("Unique",torch.unique(n_mask))
Unique tensor([0., 1.])

Minimal reproducible example : run this script without arguments. Requirements :

• You need to be on my branch of aloception (deformable2slam) to have the new implementations of _rotate and _spatial_shift in mask.py,
• You need to be able to comment and uncomment lines 366 to 369 in batch_list
• You need to have a ForwardX11 connection to see the displays
• You’ll need roughly 4GiB of free GPU

The bug is marked as « hard » because it might have potential implications on the Mask class (maybe the mask looks OK visually but the display is actually wrong?) and on all the DETR-based networks (maybe something is wrong with them and this is why they are reacting so strangely to changes in the .mask?)