search

NiccoloCavagnero / PEM

PEM: Prototype-based Efficient MaskFormer for Image Segmentation
https://niccolocavagnero.github.io/PEM/
71 stars 5 forks source link

dtype mismatch in Deform conv #3

Closed abhishekaich27 closed 6 months ago

abhishekaich27 commented 6 months ago

Hi @Gabrysse!

Why do you have this line here: https://github.com/NiccoloCavagnero/PEM/blob/42c3f652720edacc15e61d51e8c679879c5265ad/pem/modeling/pixel_decoder/pem_pixel_decoder.py#L127

It's causing an error in https://github.com/NiccoloCavagnero/PEM/blob/42c3f652720edacc15e61d51e8c679879c5265ad/pem/modeling/pixel_decoder/pem_pixel_decoder.py#L248 : image

If I disable this line, I get a device-side trigger in the hungarian matching. Any inputs?

Gabrysse commented 6 months ago

Hi @abhishekaich27, this bug is caused by detectron itself. For this reason we asked in the readme to install detectron from our repository as we have made a change to fix the above issue.

abhishekaich27 commented 6 months ago

Understood! Can you point to the change?

Gabrysse commented 6 months ago

Yes, of course.

https://github.com/NiccoloCavagnero/PEM/blob/main/detectron2/detectron2/layers/deform_conv.py#L221 https://github.com/NiccoloCavagnero/PEM/blob/main/detectron2/detectron2/layers/deform_conv.py#L251

abhishekaich27 commented 6 months ago

Anyone trying to use their default detectron2 with PEM can add the following line after PEM/pem/modeling/pixel_decoder/pem_pixel_decoder.py

Line 127 in 42c3f65

x = self.dcn(out.to(self.dcn.weight.dtype), offset.to(self.dcn.weight.dtype), mask.to(self.dcn.weight.dtype))

Gabrysse commented 5 months ago

Thanks a lot @abhishekaich27 for your solution, appreciate :)