search

FilippoAleotti / mobilePydnet

Pydnet on mobile devices
Apache License 2.0
255 stars 40 forks source link

Incorrect output dimension on ios #9

Closed l0stpenguin closed 4 years ago

l0stpenguin commented 4 years ago

I tried reconverting the latest pretrained model from the Pydnet repository to ios via tfcoreml. The conversion is successful but the output shape has wrong dimensions: (1, 512, 256, 2) I expected it to be 1 instead of 2. I know there is already a provided ios coreml file here but i plan to retrain the Pydnet model on my own dataset later. That's why i am attempting to do the conversation myself.

mlmodel = tfcoreml.convert(
            tf_model_path = './checkpoint/IROS18/frozen_model.pb',
            mlmodel_path = './checkpoint/IROS18/pydnet.mlmodel',
            output_feature_names = ['model/L0/ResizeBilinear'],
            image_input_names = ['input'],
            input_name_shape_dict = {'input':[1,512,256,3]},
            minimum_ios_deployment_target = '13'
 )

@GZaccaroni did you encounter such issue when you did the conversion for the ios part? Due to the incorrect dimension, i am not able to transform it to a valid image. Here is the full conversion log from tfcoreml:

[SSAConverter] Converting function main ...
[SSAConverter] [1/143] Converting op type: 'Placeholder', name: 'input', output_shape: (1, 512, 256, 3).
[SSAConverter] [2/143] Converting op type: 'Const', name: 'model/pyramid/conv1a/mul/x'.
[SSAConverter] [3/143] Converting op type: 'Const', name: 'model/pyramid/conv1b/mul/x'.
[SSAConverter] [4/143] Converting op type: 'Const', name: 'model/pyramid/conv2a/mul/x'.
[SSAConverter] [5/143] Converting op type: 'Const', name: 'model/pyramid/conv2b/mul/x'.
[SSAConverter] [6/143] Converting op type: 'Const', name: 'model/pyramid/conv3a/mul/x'.
[SSAConverter] [7/143] Converting op type: 'Const', name: 'model/pyramid/conv3b/mul/x'.
[SSAConverter] [8/143] Converting op type: 'Const', name: 'model/pyramid/conv4a/mul/x'.
[SSAConverter] [9/143] Converting op type: 'Const', name: 'model/pyramid/conv4b/mul/x'.
[SSAConverter] [10/143] Converting op type: 'Const', name: 'model/pyramid/conv5a/mul/x'.
[SSAConverter] [11/143] Converting op type: 'Const', name: 'model/pyramid/conv5b/mul/x'.
[SSAConverter] [12/143] Converting op type: 'Const', name: 'model/pyramid/conv6a/mul/x'.
[SSAConverter] [13/143] Converting op type: 'Const', name: 'model/pyramid/conv6b/mul/x'.
[SSAConverter] [14/143] Converting op type: 'Const', name: 'model/L6/estimator/disp-3/mul/x'.
[SSAConverter] [15/143] Converting op type: 'Const', name: 'model/L6/estimator/disp-4/mul/x'.
[SSAConverter] [16/143] Converting op type: 'Const', name: 'model/L6/estimator/disp-5/mul/x'.
[SSAConverter] [17/143] Converting op type: 'Const', name: 'model/L6/upsampler/weights/read', output_shape: (2, 2, 8, 8).
[SSAConverter] [18/143] Converting op type: 'Const', name: 'model/L6/upsampler/biases/read', output_shape: (1, 8, 1, 1).
[SSAConverter] [19/143] Converting op type: 'Const', name: 'model/L6/upsampler/conv2d_transpose/output_shape', output_shape: (4,).
[SSAConverter] [20/143] Converting op type: 'Const', name: 'model/L6/upsampler/mul/x'.
[SSAConverter] [21/143] Converting op type: 'Const', name: 'model/L5/estimator/concat/axis'.
[SSAConverter] [22/143] Converting op type: 'Const', name: 'model/L5/estimator/disp-3/mul/x'.
[SSAConverter] [23/143] Converting op type: 'Const', name: 'model/L5/estimator/disp-4/mul/x'.
[SSAConverter] [24/143] Converting op type: 'Const', name: 'model/L5/estimator/disp-5/mul/x'.
[SSAConverter] [25/143] Converting op type: 'Const', name: 'model/L5/upsampler/weights/read', output_shape: (2, 2, 8, 8).
[SSAConverter] [26/143] Converting op type: 'Const', name: 'model/L5/upsampler/biases/read', output_shape: (1, 8, 1, 1).
[SSAConverter] [27/143] Converting op type: 'Const', name: 'model/L5/upsampler/conv2d_transpose/output_shape', output_shape: (4,).
[SSAConverter] [28/143] Converting op type: 'Const', name: 'model/L5/upsampler/mul/x'.
[SSAConverter] [29/143] Converting op type: 'Const', name: 'model/L4/estimator/concat/axis'.
[SSAConverter] [30/143] Converting op type: 'Const', name: 'model/L4/estimator/disp-3/mul/x'.
[SSAConverter] [31/143] Converting op type: 'Const', name: 'model/L4/estimator/disp-4/mul/x'.
[SSAConverter] [32/143] Converting op type: 'Const', name: 'model/L4/estimator/disp-5/mul/x'.
[SSAConverter] [33/143] Converting op type: 'Const', name: 'model/L4/upsampler/weights/read', output_shape: (2, 2, 8, 8).
[SSAConverter] [34/143] Converting op type: 'Const', name: 'model/L4/upsampler/biases/read', output_shape: (1, 8, 1, 1).
[SSAConverter] [35/143] Converting op type: 'Const', name: 'model/L4/upsampler/conv2d_transpose/output_shape', output_shape: (4,).
[SSAConverter] [36/143] Converting op type: 'Const', name: 'model/L4/upsampler/mul/x'.
[SSAConverter] [37/143] Converting op type: 'Const', name: 'model/L3/estimator/concat/axis'.
[SSAConverter] [38/143] Converting op type: 'Const', name: 'model/L3/estimator/disp-3/mul/x'.
[SSAConverter] [39/143] Converting op type: 'Const', name: 'model/L3/estimator/disp-4/mul/x'.
[SSAConverter] [40/143] Converting op type: 'Const', name: 'model/L3/estimator/disp-5/mul/x'.
[SSAConverter] [41/143] Converting op type: 'Const', name: 'model/L3/estimator/Slice/begin', output_shape: (4,).
[SSAConverter] [42/143] Converting op type: 'Const', name: 'model/L3/estimator/Slice/size', output_shape: (4,).
[SSAConverter] [43/143] Converting op type: 'Const', name: 'model/L3/estimator/mul/x'.
[SSAConverter] [44/143] Converting op type: 'Const', name: 'model/L0/size', output_shape: (2,).
[SSAConverter] [45/143] Converting op type: 'Transpose', name: 'input_to_nchw', output_shape: (1, 3, 512, 256).
[SSAConverter] [46/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv1a/Conv2D', output_shape: (1, 16, 256, 128).
[SSAConverter] [47/143] Converting op type: 'Mul', name: 'model/pyramid/conv1a/mul', output_shape: (1, 16, 256, 128).
[SSAConverter] [48/143] Converting op type: 'Maximum', name: 'model/pyramid/conv1a/Maximum', output_shape: (1, 16, 256, 128).
[SSAConverter] [49/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv1b/Conv2D', output_shape: (1, 16, 256, 128).
[SSAConverter] [50/143] Converting op type: 'Mul', name: 'model/pyramid/conv1b/mul', output_shape: (1, 16, 256, 128).
[SSAConverter] [51/143] Converting op type: 'Maximum', name: 'model/pyramid/conv1b/Maximum', output_shape: (1, 16, 256, 128).
[SSAConverter] [52/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv2a/Conv2D', output_shape: (1, 32, 128, 64).
[SSAConverter] [53/143] Converting op type: 'Mul', name: 'model/pyramid/conv2a/mul', output_shape: (1, 32, 128, 64).
[SSAConverter] [54/143] Converting op type: 'Maximum', name: 'model/pyramid/conv2a/Maximum', output_shape: (1, 32, 128, 64).
[SSAConverter] [55/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv2b/Conv2D', output_shape: (1, 32, 128, 64).
[SSAConverter] [56/143] Converting op type: 'Mul', name: 'model/pyramid/conv2b/mul', output_shape: (1, 32, 128, 64).
[SSAConverter] [57/143] Converting op type: 'Maximum', name: 'model/pyramid/conv2b/Maximum', output_shape: (1, 32, 128, 64).
[SSAConverter] [58/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv3a/Conv2D', output_shape: (1, 64, 64, 32).
[SSAConverter] [59/143] Converting op type: 'Mul', name: 'model/pyramid/conv3a/mul', output_shape: (1, 64, 64, 32).
[SSAConverter] [60/143] Converting op type: 'Maximum', name: 'model/pyramid/conv3a/Maximum', output_shape: (1, 64, 64, 32).
[SSAConverter] [61/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv3b/Conv2D', output_shape: (1, 64, 64, 32).
[SSAConverter] [62/143] Converting op type: 'Mul', name: 'model/pyramid/conv3b/mul', output_shape: (1, 64, 64, 32).
[SSAConverter] [63/143] Converting op type: 'Maximum', name: 'model/pyramid/conv3b/Maximum', output_shape: (1, 64, 64, 32).
[SSAConverter] [64/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv4a/Conv2D', output_shape: (1, 96, 32, 16).
[SSAConverter] [65/143] Converting op type: 'Mul', name: 'model/pyramid/conv4a/mul', output_shape: (1, 96, 32, 16).
[SSAConverter] [66/143] Converting op type: 'Maximum', name: 'model/pyramid/conv4a/Maximum', output_shape: (1, 96, 32, 16).
[SSAConverter] [67/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv4b/Conv2D', output_shape: (1, 96, 32, 16).
[SSAConverter] [68/143] Converting op type: 'Mul', name: 'model/pyramid/conv4b/mul', output_shape: (1, 96, 32, 16).
[SSAConverter] [69/143] Converting op type: 'Maximum', name: 'model/pyramid/conv4b/Maximum', output_shape: (1, 96, 32, 16).
[SSAConverter] [70/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv5a/Conv2D', output_shape: (1, 128, 16, 8).
[SSAConverter] [71/143] Converting op type: 'Mul', name: 'model/pyramid/conv5a/mul', output_shape: (1, 128, 16, 8).
[SSAConverter] [72/143] Converting op type: 'Maximum', name: 'model/pyramid/conv5a/Maximum', output_shape: (1, 128, 16, 8).
[SSAConverter] [73/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv5b/Conv2D', output_shape: (1, 128, 16, 8).
[SSAConverter] [74/143] Converting op type: 'Mul', name: 'model/pyramid/conv5b/mul', output_shape: (1, 128, 16, 8).
[SSAConverter] [75/143] Converting op type: 'Maximum', name: 'model/pyramid/conv5b/Maximum', output_shape: (1, 128, 16, 8).
[SSAConverter] [76/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv6a/Conv2D', output_shape: (1, 192, 8, 4).
[SSAConverter] [77/143] Converting op type: 'Mul', name: 'model/pyramid/conv6a/mul', output_shape: (1, 192, 8, 4).
[SSAConverter] [78/143] Converting op type: 'Maximum', name: 'model/pyramid/conv6a/Maximum', output_shape: (1, 192, 8, 4).
[SSAConverter] [79/143] Converting op type: 'Conv2D', name: 'model/pyramid/conv6b/Conv2D', output_shape: (1, 192, 8, 4).
[SSAConverter] [80/143] Converting op type: 'Mul', name: 'model/pyramid/conv6b/mul', output_shape: (1, 192, 8, 4).
[SSAConverter] [81/143] Converting op type: 'Maximum', name: 'model/pyramid/conv6b/Maximum', output_shape: (1, 192, 8, 4).
[SSAConverter] [82/143] Converting op type: 'Conv2D', name: 'model/L6/estimator/disp-3/Conv2D', output_shape: (1, 96, 8, 4).
[SSAConverter] [83/143] Converting op type: 'Mul', name: 'model/L6/estimator/disp-3/mul', output_shape: (1, 96, 8, 4).
[SSAConverter] [84/143] Converting op type: 'Maximum', name: 'model/L6/estimator/disp-3/Maximum', output_shape: (1, 96, 8, 4).
[SSAConverter] [85/143] Converting op type: 'Conv2D', name: 'model/L6/estimator/disp-4/Conv2D', output_shape: (1, 64, 8, 4).
[SSAConverter] [86/143] Converting op type: 'Mul', name: 'model/L6/estimator/disp-4/mul', output_shape: (1, 64, 8, 4).
[SSAConverter] [87/143] Converting op type: 'Maximum', name: 'model/L6/estimator/disp-4/Maximum', output_shape: (1, 64, 8, 4).
[SSAConverter] [88/143] Converting op type: 'Conv2D', name: 'model/L6/estimator/disp-5/Conv2D', output_shape: (1, 32, 8, 4).
[SSAConverter] [89/143] Converting op type: 'Mul', name: 'model/L6/estimator/disp-5/mul', output_shape: (1, 32, 8, 4).
[SSAConverter] [90/143] Converting op type: 'Maximum', name: 'model/L6/estimator/disp-5/Maximum', output_shape: (1, 32, 8, 4).
[SSAConverter] [91/143] Converting op type: 'Conv2D', name: 'model/L6/estimator/disp-6/Conv2D', output_shape: (1, 8, 8, 4).
[SSAConverter] [92/143] Converting op type: 'Conv2DBackpropInput', name: 'model/L6/upsampler/conv2d_transpose', output_shape: (1, 8, 16, 8).
[SSAConverter] [93/143] Converting op type: 'BiasAdd', name: 'model/L6/upsampler/BiasAdd', output_shape: (1, 8, 16, 8).
[SSAConverter] [94/143] Converting op type: 'Mul', name: 'model/L6/upsampler/mul', output_shape: (1, 8, 16, 8).
[SSAConverter] [95/143] Converting op type: 'Maximum', name: 'model/L6/upsampler/Maximum', output_shape: (1, 8, 16, 8).
[SSAConverter] [96/143] Converting op type: 'ConcatV2', name: 'model/L5/estimator/concat', output_shape: (1, 136, 16, 8).
[SSAConverter] [97/143] Converting op type: 'Conv2D', name: 'model/L5/estimator/disp-3/Conv2D', output_shape: (1, 96, 16, 8).
[SSAConverter] [98/143] Converting op type: 'Mul', name: 'model/L5/estimator/disp-3/mul', output_shape: (1, 96, 16, 8).
[SSAConverter] [99/143] Converting op type: 'Maximum', name: 'model/L5/estimator/disp-3/Maximum', output_shape: (1, 96, 16, 8).
[SSAConverter] [100/143] Converting op type: 'Conv2D', name: 'model/L5/estimator/disp-4/Conv2D', output_shape: (1, 64, 16, 8).
[SSAConverter] [101/143] Converting op type: 'Mul', name: 'model/L5/estimator/disp-4/mul', output_shape: (1, 64, 16, 8).
[SSAConverter] [102/143] Converting op type: 'Maximum', name: 'model/L5/estimator/disp-4/Maximum', output_shape: (1, 64, 16, 8).
[SSAConverter] [103/143] Converting op type: 'Conv2D', name: 'model/L5/estimator/disp-5/Conv2D', output_shape: (1, 32, 16, 8).
[SSAConverter] [104/143] Converting op type: 'Mul', name: 'model/L5/estimator/disp-5/mul', output_shape: (1, 32, 16, 8).
[SSAConverter] [105/143] Converting op type: 'Maximum', name: 'model/L5/estimator/disp-5/Maximum', output_shape: (1, 32, 16, 8).
[SSAConverter] [106/143] Converting op type: 'Conv2D', name: 'model/L5/estimator/disp-6/Conv2D', output_shape: (1, 8, 16, 8).
[SSAConverter] [107/143] Converting op type: 'Conv2DBackpropInput', name: 'model/L5/upsampler/conv2d_transpose', output_shape: (1, 8, 32, 16).
[SSAConverter] [108/143] Converting op type: 'BiasAdd', name: 'model/L5/upsampler/BiasAdd', output_shape: (1, 8, 32, 16).
[SSAConverter] [109/143] Converting op type: 'Mul', name: 'model/L5/upsampler/mul', output_shape: (1, 8, 32, 16).
[SSAConverter] [110/143] Converting op type: 'Maximum', name: 'model/L5/upsampler/Maximum', output_shape: (1, 8, 32, 16).
[SSAConverter] [111/143] Converting op type: 'ConcatV2', name: 'model/L4/estimator/concat', output_shape: (1, 104, 32, 16).
[SSAConverter] [112/143] Converting op type: 'Conv2D', name: 'model/L4/estimator/disp-3/Conv2D', output_shape: (1, 96, 32, 16).
[SSAConverter] [113/143] Converting op type: 'Mul', name: 'model/L4/estimator/disp-3/mul', output_shape: (1, 96, 32, 16).
[SSAConverter] [114/143] Converting op type: 'Maximum', name: 'model/L4/estimator/disp-3/Maximum', output_shape: (1, 96, 32, 16).
[SSAConverter] [115/143] Converting op type: 'Conv2D', name: 'model/L4/estimator/disp-4/Conv2D', output_shape: (1, 64, 32, 16).
[SSAConverter] [116/143] Converting op type: 'Mul', name: 'model/L4/estimator/disp-4/mul', output_shape: (1, 64, 32, 16).
[SSAConverter] [117/143] Converting op type: 'Maximum', name: 'model/L4/estimator/disp-4/Maximum', output_shape: (1, 64, 32, 16).
[SSAConverter] [118/143] Converting op type: 'Conv2D', name: 'model/L4/estimator/disp-5/Conv2D', output_shape: (1, 32, 32, 16).
[SSAConverter] [119/143] Converting op type: 'Mul', name: 'model/L4/estimator/disp-5/mul', output_shape: (1, 32, 32, 16).
[SSAConverter] [120/143] Converting op type: 'Maximum', name: 'model/L4/estimator/disp-5/Maximum', output_shape: (1, 32, 32, 16).
[SSAConverter] [121/143] Converting op type: 'Conv2D', name: 'model/L4/estimator/disp-6/Conv2D', output_shape: (1, 8, 32, 16).
[SSAConverter] [122/143] Converting op type: 'Conv2DBackpropInput', name: 'model/L4/upsampler/conv2d_transpose', output_shape: (1, 8, 64, 32).
[SSAConverter] [123/143] Converting op type: 'BiasAdd', name: 'model/L4/upsampler/BiasAdd', output_shape: (1, 8, 64, 32).
[SSAConverter] [124/143] Converting op type: 'Mul', name: 'model/L4/upsampler/mul', output_shape: (1, 8, 64, 32).
[SSAConverter] [125/143] Converting op type: 'Maximum', name: 'model/L4/upsampler/Maximum', output_shape: (1, 8, 64, 32).
[SSAConverter] [126/143] Converting op type: 'ConcatV2', name: 'model/L3/estimator/concat', output_shape: (1, 72, 64, 32).
[SSAConverter] [127/143] Converting op type: 'Conv2D', name: 'model/L3/estimator/disp-3/Conv2D', output_shape: (1, 96, 64, 32).
[SSAConverter] [128/143] Converting op type: 'Mul', name: 'model/L3/estimator/disp-3/mul', output_shape: (1, 96, 64, 32).
[SSAConverter] [129/143] Converting op type: 'Maximum', name: 'model/L3/estimator/disp-3/Maximum', output_shape: (1, 96, 64, 32).
[SSAConverter] [130/143] Converting op type: 'Conv2D', name: 'model/L3/estimator/disp-4/Conv2D', output_shape: (1, 64, 64, 32).
[SSAConverter] [131/143] Converting op type: 'Mul', name: 'model/L3/estimator/disp-4/mul', output_shape: (1, 64, 64, 32).
[SSAConverter] [132/143] Converting op type: 'Maximum', name: 'model/L3/estimator/disp-4/Maximum', output_shape: (1, 64, 64, 32).
[SSAConverter] [133/143] Converting op type: 'Conv2D', name: 'model/L3/estimator/disp-5/Conv2D', output_shape: (1, 32, 64, 32).
[SSAConverter] [134/143] Converting op type: 'Mul', name: 'model/L3/estimator/disp-5/mul', output_shape: (1, 32, 64, 32).
[SSAConverter] [135/143] Converting op type: 'Maximum', name: 'model/L3/estimator/disp-5/Maximum', output_shape: (1, 32, 64, 32).
[SSAConverter] [136/143] Converting op type: 'Conv2D', name: 'model/L3/estimator/disp-6/Conv2D', output_shape: (1, 8, 64, 32).
[SSAConverter] [137/143] Converting op type: 'Transpose', name: 'model/L3/estimator/disp-6/Conv2D_to_nhwc', output_shape: (1, 64, 32, 8).
[SSAConverter] [138/143] Converting op type: 'Slice', name: 'model/L3/estimator/Slice', output_shape: (1, 64, 32, 2).
[SSAConverter] [139/143] Converting op type: 'Sigmoid', name: 'model/L3/estimator/Sigmoid', output_shape: (1, 64, 32, 2).
[SSAConverter] [140/143] Converting op type: 'Mul', name: 'model/L3/estimator/mul', output_shape: (1, 64, 32, 2).
[SSAConverter] [141/143] Converting op type: 'Transpose', name: 'model/L3/estimator/mul_to_nchw', output_shape: (1, 2, 64, 32).
[SSAConverter] [142/143] Converting op type: 'ResizeBilinear', name: 'model/L0/ResizeBilinear_orig0', output_shape: (1, 2, 512, 256).
[SSAConverter] [143/143] Converting op type: 'Transpose', name: 'model/L0/ResizeBilinear', output_shape: (1, 512, 256, 2).
FilippoAleotti commented 4 years ago

Hi, In the IROS2018 model the network predicts 2 disparities (ie, a left-aligned and a "virtual" right-aligned maps). If you are using this checkpoint, you have to slice the tensor

l0stpenguin commented 4 years ago

Yes i am using that checkpoint. I am not sure about how exactly i should proceed with slicing. I found only 1 output model/L0/ResizeBilinear and used that to freeze the graph. Are you suggesting that i should be modifying the output before freezing the graph?

FilippoAleotti commented 4 years ago

Yes, something like: tf.image.resize_images(self.disp2[:,:,:,0], size)

l0stpenguin commented 4 years ago

I am not experienced in neural networks, so i am quite confused. I don't manipulate any outputs nor do i run any inference in python. I just used the script in this post to export the pb file:

https://blog.metaflow.fr/tensorflow-how-to-freeze-a-model-and-serve-it-with-a-python-api-d4f3596b3adc

Since it's has only 1 output with 2 channels, i don't know how i am supposed to slice after freezing the graph.

FilippoAleotti commented 4 years ago

You have two options: