Can I retrain the same algorithm for a different resolution ToF sensor?

[darshit-desai]

Hi Authors, Awesome paper, I have pretty much the question said in the subject. I have a ToF sensor which has a depth image resolution of 224x172? Can I retrain the algorithm for this sensor and use it for RGB-D mapping while this algorithm gives super resolved depth image in real time?

[eugenelyj]

@darshit-desai Hello, sure you can try it. I think DELTAR will achieve amazing performance, especially when the resolution ratio of RGB to ToF is large because it matches the design of the cross-domain transformer.

[darshit-desai]

Edit: After reading the paper once again, I found out that the calibration process was done to basically find out the rotation and translation of the depth camera with respect to the RGB camera, and similarly the 8x8 depth zones. In my case I have the ToF sensor's Rotation and Translation with respect to a fixed point on a robot and with the same fixed point I have rotation and translation of the RGB camera.

If my understanding of your method of calibration is correct, it also finds the relative Rotation and Translation with respect to each of the cameras, and if I already have that, all I need to do is make the histogram of depth image and pass it to the pretrained model along with the RGB image.

Old content: Hi @eugenelyj thank you for replying, after reading your paper I felt that the sensor combination I have is not going to be relevant enough. Especially in the Related work section of the paper the authors mentioned that the algorithm is made by taking in consideration the number. of depth pixels ranging from 100-1000 pixels. The depth ToF I have has 38000 (224x172) depth pixels. In this resolution I want to change to a resolution of 640x480 (or more like 4208x3120). Now I understand that this might not be possible with the current version of Delta- AR. Also Delta AR takes into consideration the ToF's way of capturing depth data i.e., zone wise calibration, but that is not the case with the ToF sensor I have, the maker hasn't specified any zone wise data acquisition specifications in the data sheet.

In your opinion is there any other method apart from Delta AR I can follow or is there a way to modify Delta AR or use it as it is in my use case.

For further information I am attaching sensor data sheets for your reference:

ToF sensor: Vendor website datasheet:https://docs.modalai.com/M0040/ Chip maker website datasheet: https://www.infineon.com/dgdl/Infineon-REAL3+Image+Sensor+Family-PB-v01_00-EN.PDF?fileId=5546d462518ffd850151a0afc2302a58

The ToF sensor mentioned above is used in a drone whose vendor website datasheet is mentioned above. The chip maker for this ToF sensor is info neon whose data brief available on the website is also mentioned above.

RGB Image sensor: https://docs.modalai.com/M0024/

Although the RGB sensor provides an image at 4K resolution, it is still bulky to process in a neural network for detection, so it will be resized to 640x480.

Thank you for your help,

Best, Darshit

[eugenelyj]

@darshit-desai Hi, according to your description, I would suggest try some detph super-resoltuion method, which can be found in the section of related work. Moreover, can you obtain the histogram data from your ToF sensor? If not, I think DELTAR is not suitable for that sensor.

[darshit-desai]

I dont think my sensor gives the histogram data from the sensor, Thank you for your help I will have a look, Do you have any knowledge of algorithms that work on classical interpolation or mentioned any such papers with code on the related work section?