Problems with TFLite model quality

CDDelta commented 4 years ago

Hi @FilippoAleotti,

I am a high school student trying to use you and your team's PyDNet model for a project I am working on. Specifically, I'm trying to run your .tflite model on a Raspberry Pi but I am having trouble getting a similar quality depth map from it as shown in the demos.

I notice #1 has a mention about this issue but I also noticed that #3, which introduced "optimized_pydnet++.tflite", comes a few months after the last comment on #1 which suggests that you did not face this issue with model quality.

AFAICT I am performing the same procedures that are done in the Android sample on the master branch (scaling the input and dividing the pixel values by 255) but the output still looks wrong. This Colab I have here demonstrates the issues I am having.

For some reason, the model is consistently outputting a ridiculously high depth value for that particular spot in all images. index This is possibly a TensorFlow bug so I worked around it by simply cropping the output depth to a region where that point is not in the image but then I stumble across another problem.

The cropped depth map looks to be of a rather low quality unlike the demos shown. Here is an example: index The input I'm providing is 640x448 pixels and from what I can tell there is no option to configure the resolution on the tflite model.

I have been trying to get this working for a few days now and am at a lost on what to do. I realise that you're busy with your research but could you offer me some guidance on how to move forward? I notice that you have v2 of PyDNet mobile in the works for Android, could you provide an update on the status for a v2 tflite model?

Thank you for the work you've put into this project, it is quite incredible!

FilippoAleotti commented 4 years ago

Hi, have you tried with the new model? The tflite model you are using seems the one trained on MatterPort

CDDelta commented 4 years ago

Hi, thanks for replying.

I have tried using that model before along with the script provided and the results I get are really good but I'm looking for a .tflite version of this model.

Right now I am trying to convert that model into the .tflite format but I am doing so. Have you done some work converting this model to tflite? I notice you have a tf.js version of the model working.

CDDelta commented 4 years ago

Hi, I have an additional unrelated question. Does the model predict the straight line distance from the camera to the point in the image or the distance perpendicular to the camera plane?

ie. does the model predict "di" as in the diagram below or the length of the hypotenuse of that triangle?

FilippoAleotti commented 4 years ago

Hi, I’ve included the code to export the model as tflite. About the second question, actually the model tries to estimate the inverse depth, so you need to apply a scale factor to obtain the depth

CDDelta commented 4 years ago

Hey, thank you so much for taking the time out of your day to help me out!

The script worked and I managed to export the model for tflite with really nice looking results. I will have to look deeper into inverse depth.

I really appreciate your help and work here, thank you! ❤️