search

Unity-Technologies / barracuda-release

Other
561 stars 76 forks source link

OnnxImportException, converting .pb model to .onnx fails (tensorflow_to_barracuda.py and tf2onnx) #171

Closed DaphiFluffi closed 3 years ago

DaphiFluffi commented 3 years ago

Hi everyone,

I have a quite specific situation and any help is apprechiated.

Versions:

My Problem: I have successfully trained a YOLOv3 Tiny model with Darknet. Then I converted it to a saved_model with the save_model.py script from this https://github.com/hunglc007/tensorflow-yolov4-tflite repository. And that worked as far as I can see. It created an assets and variables folder and the saved_model.pb. Now, I want to use my model in Unity and need to convert it to .onnx or .nn. I have tried converting it with this command and it did produce an .onnx file successfully, but it won't import into Unity. !python -m tf2onnx.convert \ --saved-model /content/MyModel \ --output ./output/pillbox.onnx \ --opset 10 In Unity, I get this error:

OnnxImportException: Unexpected error while parsing layer StatefulPartitionedCall/functional_1/tf_op_layer_Tile_1/multiples/Tile_1/multiples_Concat__125:0 of type Concat.
An item with the same key has already been added. Key: const_fold_opt__373

Unfortunately, I couldn't find a solution that worked for me. That is why I decided to dig up tensorflow_to_barracuda.py from version 1.2.1 of this repository and tried running this command: !python tensorflow_to_barracuda.py /content/TM/saved_model.pb /content/Dest/pillbox-again.nn

But I got the following error and after looking around forums for days I still have not found a way to solve it:

Traceback (most recent call last):
  File "tensorflow_to_barracuda.py", line 20, in <module>
    tf2bc.convert(args.source_file, args.target_file, args.trim_unused_by_output, args)
  File "/content/barracuda-release/Tools/tensorflow_to_barracuda.py", line 1350, in convert
    i_model.ParseFromString(f.read())
google.protobuf.message.DecodeError: Error parsing message

I am assuming that the script cannot propertly read the model, but I have tried it with other saved models as well and still got this error. Maybe it is a versioning issue? I know that tensorflow_to_barracuda.py is depricated and I would have liked to use ft2onnx, but both haven't worked for me yet.

Any tipps on how to solve the Unity or tensorflow_to_barracuda.py error would be very apprechiated. If you need any further information, I am happy to provide it.

Thank you in advance. Sorry if this is the wrong place for posting this question. I am quite new to this.

FlorentGuinier commented 3 years ago

Hi @DaphiFluffi , As you have noticed we don't maintain tensorflow_to_barracuda.py anymore. The onnx import path is thus were we need to focus here. Could you share the converted .onnx on this thread (random weights are fine) or via barracuda-support @ unity3d.com please?

OnnxImportException: Unexpected error while parsing layer StatefulPartitionedCall/functional_1/tf_op_layer_Tile_1/multiples/Tile_1/multiples_Concat125:0 of type Concat. An item with the same key has already been added. Key: const_fold_opt373

This look like a bug on our side.

Thanks Florent

DaphiFluffi commented 3 years ago

Hi @FlorentGuinier,

I am uploading the model to my Google Drive and will send you the link via email shortly.

Thank you for helping Daphna

FlorentGuinier commented 3 years ago

Thanks @DaphiFluffi , just to confirm we got the model, we will look into it.

AlexRibard commented 3 years ago

@DaphiFluffi I have checked your model. Barracuda does not handle dynamic shape logic that much unfortunately. We do our best if all shapes are know. If not, we do try to import correctly but this requires all ranks to be inferable. In this case the end of the model contains a NonZero followed closely by a Gather image The problem with this is that we do not know the rank of that Gather node (since it is dynamic) and then are unable to import the rest of the model.

Would you be able to move that last part of your model in C#?

DaphiFluffi commented 3 years ago

Hi Alex,

Thank you very much for your answer. Unfortunately, I am quite new to Barracuda and I wouldn't know how to move that part into C#. Do you think there is a way to convert a Tensorflow saved model to Barracuda in a way that this problem does not occur?

Kind Regards Daphna

Am Fr., 7. Mai 2021 um 18:52 Uhr schrieb AlexRibard < @.***>:

@DaphiFluffi https://github.com/DaphiFluffi I have checked your model. Barracuda does not handle dynamic shape logic that much unfortunately. We do our best if all shapes are know. If not, we do try to import correctly but this requires all ranks to be inferable. In this case the end of the model contains a NonZero followed closely by a Gather [image: image] https://user-images.githubusercontent.com/12990800/117481618-3ad18500-af31-11eb-9fb9-0e071b072421.png The problem with this is that we do not know the rank of that Gather node (since it is dynamic) and then are unable to import the rest of the model.

Would you be able to move that last part of your model in C#?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Unity-Technologies/barracuda-release/issues/171#issuecomment-834618004, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKGDJW7E7KHAW5ZIBV2DBFLTMQLEFANCNFSM44C2KGVA .

AlexRibard commented 3 years ago

What does the code in TF looks like for the bottom part of the model?

DaphiFluffi commented 3 years ago

I didn't create the Tensorflow Model from Scratch unfortunately. I converted the .weights file I got after training YOLOv3 Tiny with Darknet to a Tensorflow Saved Model. When I ran the script I got the following output. I hope this can answer your question. 

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 416, 416, 3) 0                                            
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, 416, 416, 16) 432         input_1[0][0]                    
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 416, 416, 16) 64          conv2d[0][0]                     
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu (TensorFl [(None, 416, 416, 16 0           batch_normalization[0][0]        
__________________________________________________________________________________________________
max_pooling2d (MaxPooling2D)    (None, 208, 208, 16) 0           tf_op_layer_LeakyRelu[0][0]      
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 208, 208, 32) 4608        max_pooling2d[0][0]              
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 208, 208, 32) 128         conv2d_1[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_1 (Tensor [(None, 208, 208, 32 0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)  (None, 104, 104, 32) 0           tf_op_layer_LeakyRelu_1[0][0]    
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 104, 104, 64) 18432       max_pooling2d_1[0][0]            
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 104, 104, 64) 256         conv2d_2[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_2 (Tensor [(None, 104, 104, 64 0           batch_normalization_2[0][0]      
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)  (None, 52, 52, 64)   0           tf_op_layer_LeakyRelu_2[0][0]    
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 52, 52, 128)  73728       max_pooling2d_2[0][0]            
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 52, 52, 128)  512         conv2d_3[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_3 (Tensor [(None, 52, 52, 128) 0           batch_normalization_3[0][0]      
__________________________________________________________________________________________________
max_pooling2d_3 (MaxPooling2D)  (None, 26, 26, 128)  0           tf_op_layer_LeakyRelu_3[0][0]    
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 26, 26, 256)  294912      max_pooling2d_3[0][0]            
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 26, 26, 256)  1024        conv2d_4[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_4 (Tensor [(None, 26, 26, 256) 0           batch_normalization_4[0][0]      
__________________________________________________________________________________________________
max_pooling2d_4 (MaxPooling2D)  (None, 13, 13, 256)  0           tf_op_layer_LeakyRelu_4[0][0]    
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 13, 13, 512)  1179648     max_pooling2d_4[0][0]            
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 13, 13, 512)  2048        conv2d_5[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_5 (Tensor [(None, 13, 13, 512) 0           batch_normalization_5[0][0]      
__________________________________________________________________________________________________
max_pooling2d_5 (MaxPooling2D)  (None, 13, 13, 512)  0           tf_op_layer_LeakyRelu_5[0][0]    
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, 13, 13, 1024) 4718592     max_pooling2d_5[0][0]            
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, 13, 13, 1024) 4096        conv2d_6[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_6 (Tensor [(None, 13, 13, 1024 0           batch_normalization_6[0][0]      
__________________________________________________________________________________________________
conv2d_7 (Conv2D)               (None, 13, 13, 256)  262144      tf_op_layer_LeakyRelu_6[0][0]    
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, 13, 13, 256)  1024        conv2d_7[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_7 (Tensor [(None, 13, 13, 256) 0           batch_normalization_7[0][0]      
__________________________________________________________________________________________________
conv2d_10 (Conv2D)              (None, 13, 13, 128)  32768       tf_op_layer_LeakyRelu_7[0][0]    
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, 13, 13, 128)  512         conv2d_10[0][0]                  
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_9 (Tensor [(None, 13, 13, 128) 0           batch_normalization_9[0][0]      
__________________________________________________________________________________________________
tf_op_layer_ResizeBilinear (Ten [(None, 26, 26, 128) 0           tf_op_layer_LeakyRelu_9[0][0]    
__________________________________________________________________________________________________
tf_op_layer_concat (TensorFlowO [(None, 26, 26, 384) 0           tf_op_layer_ResizeBilinear[0][0] 
                                                                 tf_op_layer_LeakyRelu_4[0][0]    
__________________________________________________________________________________________________
conv2d_11 (Conv2D)              (None, 26, 26, 256)  884736      tf_op_layer_concat[0][0]         
__________________________________________________________________________________________________
conv2d_8 (Conv2D)               (None, 13, 13, 512)  1179648     tf_op_layer_LeakyRelu_7[0][0]    
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, 26, 26, 256)  1024        conv2d_11[0][0]                  
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, 13, 13, 512)  2048        conv2d_8[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_10 (Tenso [(None, 26, 26, 256) 0           batch_normalization_10[0][0]     
__________________________________________________________________________________________________
tf_op_layer_LeakyRelu_8 (Tensor [(None, 13, 13, 512) 0           batch_normalization_8[0][0]      
__________________________________________________________________________________________________
conv2d_12 (Conv2D)              (None, 26, 26, 18)   4626        tf_op_layer_LeakyRelu_10[0][0]   
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 13, 13, 18)   9234        tf_op_layer_LeakyRelu_8[0][0]    
__________________________________________________________________________________________________
tf_op_layer_Shape (TensorFlowOp [(4,)]               0           conv2d_12[0][0]                  
__________________________________________________________________________________________________
tf_op_layer_Shape_1 (TensorFlow [(4,)]               0           conv2d_9[0][0]                   
__________________________________________________________________________________________________
tf_op_layer_strided_slice (Tens [()]                 0           tf_op_layer_Shape[0][0]          
__________________________________________________________________________________________________
tf_op_layer_strided_slice_1 (Te [()]                 0           tf_op_layer_Shape_1[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Reshape/shape (Tens [(5,)]               0           tf_op_layer_strided_slice[0][0]  
__________________________________________________________________________________________________
tf_op_layer_Reshape_3/shape (Te [(5,)]               0           tf_op_layer_strided_slice_1[0][0]
__________________________________________________________________________________________________
tf_op_layer_Reshape (TensorFlow [(None, 26, 26, 3, 6 0           conv2d_12[0][0]                  
                                                                 tf_op_layer_Reshape/shape[0][0]  
__________________________________________________________________________________________________
tf_op_layer_Reshape_3 (TensorFl [(None, 13, 13, 3, 6 0           conv2d_9[0][0]                   
                                                                 tf_op_layer_Reshape_3/shape[0][0]
__________________________________________________________________________________________________
tf_op_layer_split (TensorFlowOp [(None, 26, 26, 3, 2 0           tf_op_layer_Reshape[0][0]        
__________________________________________________________________________________________________
tf_op_layer_split_1 (TensorFlow [(None, 13, 13, 3, 2 0           tf_op_layer_Reshape_3[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Sigmoid (TensorFlow [(None, 26, 26, 3, 2 0           tf_op_layer_split[0][0]          
__________________________________________________________________________________________________
tf_op_layer_Tile/multiples (Ten [(5,)]               0           tf_op_layer_strided_slice[0][0]  
__________________________________________________________________________________________________
tf_op_layer_Sigmoid_3 (TensorFl [(None, 13, 13, 3, 2 0           tf_op_layer_split_1[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Tile_1/multiples (T [(5,)]               0           tf_op_layer_strided_slice_1[0][0]
__________________________________________________________________________________________________
tf_op_layer_Mul (TensorFlowOpLa [(None, 26, 26, 3, 2 0           tf_op_layer_Sigmoid[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Tile (TensorFlowOpL [(None, 26, 26, 3, 2 0           tf_op_layer_Tile/multiples[0][0] 
__________________________________________________________________________________________________
tf_op_layer_Mul_4 (TensorFlowOp [(None, 13, 13, 3, 2 0           tf_op_layer_Sigmoid_3[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Tile_1 (TensorFlowO [(None, 13, 13, 3, 2 0           tf_op_layer_Tile_1/multiples[0][0
__________________________________________________________________________________________________
tf_op_layer_Sub (TensorFlowOpLa [(None, 26, 26, 3, 2 0           tf_op_layer_Mul[0][0]            
__________________________________________________________________________________________________
tf_op_layer_Cast (TensorFlowOpL [(None, 26, 26, 3, 2 0           tf_op_layer_Tile[0][0]           
__________________________________________________________________________________________________
tf_op_layer_Sub_1 (TensorFlowOp [(None, 13, 13, 3, 2 0           tf_op_layer_Mul_4[0][0]          
__________________________________________________________________________________________________
tf_op_layer_Cast_1 (TensorFlowO [(None, 13, 13, 3, 2 0           tf_op_layer_Tile_1[0][0]         
__________________________________________________________________________________________________
tf_op_layer_AddV2 (TensorFlowOp [(None, 26, 26, 3, 2 0           tf_op_layer_Sub[0][0]            
                                                                 tf_op_layer_Cast[0][0]           
__________________________________________________________________________________________________
tf_op_layer_Exp (TensorFlowOpLa [(None, 26, 26, 3, 2 0           tf_op_layer_split[0][1]          
__________________________________________________________________________________________________
tf_op_layer_AddV2_1 (TensorFlow [(None, 13, 13, 3, 2 0           tf_op_layer_Sub_1[0][0]          
                                                                 tf_op_layer_Cast_1[0][0]         
__________________________________________________________________________________________________
tf_op_layer_Exp_1 (TensorFlowOp [(None, 13, 13, 3, 2 0           tf_op_layer_split_1[0][1]        
__________________________________________________________________________________________________
tf_op_layer_Mul_1 (TensorFlowOp [(None, 26, 26, 3, 2 0           tf_op_layer_AddV2[0][0]          
__________________________________________________________________________________________________
tf_op_layer_Mul_2 (TensorFlowOp [(None, 26, 26, 3, 2 0           tf_op_layer_Exp[0][0]            
__________________________________________________________________________________________________
tf_op_layer_Mul_5 (TensorFlowOp [(None, 13, 13, 3, 2 0           tf_op_layer_AddV2_1[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Mul_6 (TensorFlowOp [(None, 13, 13, 3, 2 0           tf_op_layer_Exp_1[0][0]          
__________________________________________________________________________________________________
tf_op_layer_concat_1 (TensorFlo [(None, 26, 26, 3, 4 0           tf_op_layer_Mul_1[0][0]          
                                                                 tf_op_layer_Mul_2[0][0]          
__________________________________________________________________________________________________
tf_op_layer_Reshape_2/shape (Te [(3,)]               0           tf_op_layer_strided_slice[0][0]  
__________________________________________________________________________________________________
tf_op_layer_concat_2 (TensorFlo [(None, 13, 13, 3, 4 0           tf_op_layer_Mul_5[0][0]          
                                                                 tf_op_layer_Mul_6[0][0]          
__________________________________________________________________________________________________
tf_op_layer_Reshape_5/shape (Te [(3,)]               0           tf_op_layer_strided_slice_1[0][0]
__________________________________________________________________________________________________
tf_op_layer_Sigmoid_1 (TensorFl [(None, 26, 26, 3, 1 0           tf_op_layer_split[0][2]          
__________________________________________________________________________________________________
tf_op_layer_Sigmoid_2 (TensorFl [(None, 26, 26, 3, 1 0           tf_op_layer_split[0][3]          
__________________________________________________________________________________________________
tf_op_layer_Sigmoid_4 (TensorFl [(None, 13, 13, 3, 1 0           tf_op_layer_split_1[0][2]        
__________________________________________________________________________________________________
tf_op_layer_Sigmoid_5 (TensorFl [(None, 13, 13, 3, 1 0           tf_op_layer_split_1[0][3]        
__________________________________________________________________________________________________
tf_op_layer_Reshape_2 (TensorFl [(None, None, 4)]    0           tf_op_layer_concat_1[0][0]       
                                                                 tf_op_layer_Reshape_2/shape[0][0]
__________________________________________________________________________________________________
tf_op_layer_Reshape_5 (TensorFl [(None, None, 4)]    0           tf_op_layer_concat_2[0][0]       
                                                                 tf_op_layer_Reshape_5/shape[0][0]
__________________________________________________________________________________________________
tf_op_layer_Mul_3 (TensorFlowOp [(None, 26, 26, 3, 1 0           tf_op_layer_Sigmoid_1[0][0]      
                                                                 tf_op_layer_Sigmoid_2[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Reshape_1/shape (Te [(3,)]               0           tf_op_layer_strided_slice[0][0]  
__________________________________________________________________________________________________
tf_op_layer_Mul_7 (TensorFlowOp [(None, 13, 13, 3, 1 0           tf_op_layer_Sigmoid_4[0][0]      
                                                                 tf_op_layer_Sigmoid_5[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Reshape_4/shape (Te [(3,)]               0           tf_op_layer_strided_slice_1[0][0]
__________________________________________________________________________________________________
tf_op_layer_concat_3 (TensorFlo [(None, None, 4)]    0           tf_op_layer_Reshape_2[0][0]      
                                                                 tf_op_layer_Reshape_5[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Reshape_1 (TensorFl [(None, None, 1)]    0           tf_op_layer_Mul_3[0][0]          
                                                                 tf_op_layer_Reshape_1/shape[0][0]
__________________________________________________________________________________________________
tf_op_layer_Reshape_4 (TensorFl [(None, None, 1)]    0           tf_op_layer_Mul_7[0][0]          
                                                                 tf_op_layer_Reshape_4/shape[0][0]
__________________________________________________________________________________________________
tf_op_layer_Shape_2 (TensorFlow [(3,)]               0           tf_op_layer_concat_3[0][0]       
__________________________________________________________________________________________________
tf_op_layer_concat_4 (TensorFlo [(None, None, 1)]    0           tf_op_layer_Reshape_1[0][0]      
                                                                 tf_op_layer_Reshape_4[0][0]      
__________________________________________________________________________________________________
tf_op_layer_strided_slice_2 (Te [(2,)]               0           tf_op_layer_Shape_2[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Max (TensorFlowOpLa [(None, None)]       0           tf_op_layer_concat_4[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Shape_3 (TensorFlow [(3,)]               0           tf_op_layer_concat_3[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Prod (TensorFlowOpL [()]                 0           tf_op_layer_strided_slice_2[0][0]
__________________________________________________________________________________________________
tf_op_layer_Shape_4 (TensorFlow [(3,)]               0           tf_op_layer_concat_3[0][0]       
__________________________________________________________________________________________________
tf_op_layer_GreaterEqual (Tenso [(None, None)]       0           tf_op_layer_Max[0][0]            
__________________________________________________________________________________________________
tf_op_layer_strided_slice_3 (Te [(0,)]               0           tf_op_layer_Shape_3[0][0]        
__________________________________________________________________________________________________
tf_op_layer_concat_5/values_1 ( [(1,)]               0           tf_op_layer_Prod[0][0]           
__________________________________________________________________________________________________
tf_op_layer_strided_slice_4 (Te [(1,)]               0           tf_op_layer_Shape_4[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Reshape_7 (TensorFl [(None,)]            0           tf_op_layer_GreaterEqual[0][0]   
__________________________________________________________________________________________________
tf_op_layer_concat_5 (TensorFlo [(2,)]               0           tf_op_layer_strided_slice_3[0][0]
                                                                 tf_op_layer_concat_5/values_1[0][
                                                                 tf_op_layer_strided_slice_4[0][0]
__________________________________________________________________________________________________
tf_op_layer_Where (TensorFlowOp [(None, 1)]          0           tf_op_layer_Reshape_7[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Reshape_6 (TensorFl [(None, 4)]          0           tf_op_layer_concat_3[0][0]       
                                                                 tf_op_layer_concat_5[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Squeeze (TensorFlow [(None,)]            0           tf_op_layer_Where[0][0]          
__________________________________________________________________________________________________
tf_op_layer_GatherV2 (TensorFlo [(None, 4)]          0           tf_op_layer_Reshape_6[0][0]      
                                                                 tf_op_layer_Squeeze[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Shape_8 (TensorFlow [(3,)]               0           tf_op_layer_concat_4[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Shape_9 (TensorFlow [(2,)]               0           tf_op_layer_GatherV2[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Shape_5 (TensorFlow [(3,)]               0           tf_op_layer_concat_4[0][0]       
__________________________________________________________________________________________________
tf_op_layer_strided_slice_8 (Te [()]                 0           tf_op_layer_Shape_8[0][0]        
__________________________________________________________________________________________________
tf_op_layer_strided_slice_9 (Te [()]                 0           tf_op_layer_Shape_9[0][0]        
__________________________________________________________________________________________________
tf_op_layer_strided_slice_5 (Te [(2,)]               0           tf_op_layer_Shape_5[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Reshape_10/shape (T [(3,)]               0           tf_op_layer_strided_slice_8[0][0]
                                                                 tf_op_layer_strided_slice_9[0][0]
__________________________________________________________________________________________________
tf_op_layer_Shape_6 (TensorFlow [(3,)]               0           tf_op_layer_concat_4[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Prod_1 (TensorFlowO [()]                 0           tf_op_layer_strided_slice_5[0][0]
__________________________________________________________________________________________________
tf_op_layer_Shape_7 (TensorFlow [(3,)]               0           tf_op_layer_concat_4[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Reshape_10 (TensorF [(None, None, None)] 0           tf_op_layer_GatherV2[0][0]       
                                                                 tf_op_layer_Reshape_10/shape[0][0
__________________________________________________________________________________________________
tf_op_layer_strided_slice_6 (Te [(0,)]               0           tf_op_layer_Shape_6[0][0]        
__________________________________________________________________________________________________
tf_op_layer_concat_6/values_1 ( [(1,)]               0           tf_op_layer_Prod_1[0][0]         
__________________________________________________________________________________________________
tf_op_layer_strided_slice_7 (Te [(1,)]               0           tf_op_layer_Shape_7[0][0]        
__________________________________________________________________________________________________
tf_op_layer_Reshape_9 (TensorFl [(None,)]            0           tf_op_layer_GreaterEqual[0][0]   
__________________________________________________________________________________________________
tf_op_layer_split_2 (TensorFlow [(None, None, 2), (N 0           tf_op_layer_Reshape_10[0][0]     
__________________________________________________________________________________________________
tf_op_layer_concat_6 (TensorFlo [(2,)]               0           tf_op_layer_strided_slice_6[0][0]
                                                                 tf_op_layer_concat_6/values_1[0][
                                                                 tf_op_layer_strided_slice_7[0][0]
__________________________________________________________________________________________________
tf_op_layer_Where_1 (TensorFlow [(None, 1)]          0           tf_op_layer_Reshape_9[0][0]      
__________________________________________________________________________________________________
tf_op_layer_strided_slice_13 (T [(None, None, 2)]    0           tf_op_layer_split_2[0][1]        
__________________________________________________________________________________________________
tf_op_layer_Reshape_8 (TensorFl [(None, 1)]          0           tf_op_layer_concat_4[0][0]       
                                                                 tf_op_layer_concat_6[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Squeeze_1 (TensorFl [(None,)]            0           tf_op_layer_Where_1[0][0]        
__________________________________________________________________________________________________
tf_op_layer_strided_slice_12 (T [(None, None, 2)]    0           tf_op_layer_split_2[0][0]        
__________________________________________________________________________________________________
tf_op_layer_RealDiv (TensorFlow [(None, None, 2)]    0           tf_op_layer_strided_slice_13[0][0
__________________________________________________________________________________________________
tf_op_layer_RealDiv_2 (TensorFl [(None, None, 2)]    0           tf_op_layer_strided_slice_13[0][0
__________________________________________________________________________________________________
tf_op_layer_GatherV2_1 (TensorF [(None, 1)]          0           tf_op_layer_Reshape_8[0][0]      
                                                                 tf_op_layer_Squeeze_1[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Sub_2 (TensorFlowOp [(None, None, 2)]    0           tf_op_layer_strided_slice_12[0][0
                                                                 tf_op_layer_RealDiv[0][0]        
__________________________________________________________________________________________________
tf_op_layer_AddV2_2 (TensorFlow [(None, None, 2)]    0           tf_op_layer_strided_slice_12[0][0
                                                                 tf_op_layer_RealDiv_2[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Shape_10 (TensorFlo [(3,)]               0           tf_op_layer_concat_4[0][0]       
__________________________________________________________________________________________________
tf_op_layer_Shape_11 (TensorFlo [(2,)]               0           tf_op_layer_GatherV2_1[0][0]     
__________________________________________________________________________________________________
tf_op_layer_RealDiv_1 (TensorFl [(None, None, 2)]    0           tf_op_layer_Sub_2[0][0]          
__________________________________________________________________________________________________
tf_op_layer_RealDiv_3 (TensorFl [(None, None, 2)]    0           tf_op_layer_AddV2_2[0][0]        
__________________________________________________________________________________________________
tf_op_layer_strided_slice_10 (T [()]                 0           tf_op_layer_Shape_10[0][0]       
__________________________________________________________________________________________________
tf_op_layer_strided_slice_11 (T [()]                 0           tf_op_layer_Shape_11[0][0]       
__________________________________________________________________________________________________
tf_op_layer_strided_slice_14 (T [(None, None, 1)]    0           tf_op_layer_RealDiv_1[0][0]      
__________________________________________________________________________________________________
tf_op_layer_strided_slice_15 (T [(None, None, 1)]    0           tf_op_layer_RealDiv_1[0][0]      
__________________________________________________________________________________________________
tf_op_layer_strided_slice_16 (T [(None, None, 1)]    0           tf_op_layer_RealDiv_3[0][0]      
__________________________________________________________________________________________________
tf_op_layer_strided_slice_17 (T [(None, None, 1)]    0           tf_op_layer_RealDiv_3[0][0]      
__________________________________________________________________________________________________
tf_op_layer_Reshape_11/shape (T [(3,)]               0           tf_op_layer_strided_slice_10[0][0
                                                                 tf_op_layer_strided_slice_11[0][0
__________________________________________________________________________________________________
tf_op_layer_concat_7 (TensorFlo [(None, None, 4)]    0           tf_op_layer_strided_slice_14[0][0
                                                                 tf_op_layer_strided_slice_15[0][0
                                                                 tf_op_layer_strided_slice_16[0][0
                                                                 tf_op_layer_strided_slice_17[0][0
__________________________________________________________________________________________________
tf_op_layer_Reshape_11 (TensorF [(None, None, None)] 0           tf_op_layer_GatherV2_1[0][0]     
                                                                 tf_op_layer_Reshape_11/shape[0][0
__________________________________________________________________________________________________
tf_op_layer_concat_8 (TensorFlo [(None, None, None)] 0           tf_op_layer_concat_7[0][0]       
                                                                 tf_op_layer_Reshape_11[0][0]     
==================================================================================================
Total params: 8,676,244
Trainable params: 8,669,876
Non-trainable params: 6,368
AlexRibard commented 3 years ago

I see, in that case there isn't much we can work around unfortunately :/ But on the bright side if what you are looking for is a YOLOv3 Tiny that imports in Barracuda, I'd suggest you look on this thread: https://github.com/Unity-Technologies/barracuda-release/issues/132 We do support a number of YOLOv3 Tiny variants.

Let me know if that works for you

DaphiFluffi commented 3 years ago

Hi Alex,

I tried out the Onnx simplifier mentioned in #132. However, I got the same error in Unity as before.

AlexRibard commented 3 years ago

yeah that will not make your model import. Sorry for the confusion, I was more referring to using the models they shared as those import

DaphiFluffi commented 3 years ago

Yeah I tried those out, but I am trying to import my own model. I am now trying out the techniques mentioned here https://github.com/derenlei/Unity_Detection2AR/issues/16

AlexRibard commented 3 years ago

I see ok. Unfortunately the main resides in our importer not supporting fully dynamic Tensors. One other option (apart from changing your architecture) is to strip that bottom part I mentioned and do it in C#. That would be a good workaround. Would that work for you?

DaphiFluffi commented 3 years ago

It could. But I don't have any idea how to do that.

AlexRibard commented 3 years ago

Ok, let me give it a try

DaphiFluffi commented 3 years ago

Hi @AlexRibard,

I actually found a way of converting the darknet model to onnx that worked and was importable into Unity. I will try and document it as good as i can, but I basically followed the steps specified here: https://github.com/derenlei/Unity_Detection2AR/issues/16#issuecomment-786339303 Thank you for your help!

AlexRibard commented 3 years ago

Hey that's great to hear! I'm going to close the issue, feel free to reopen if you have other issues :)

DaphiFluffi commented 3 years ago

Hi @AlexRibard,

thank you for your help so far. Now that I got the barracuda model imported successfully into Unity, when trying to read the model as specified here https://docs.unity3d.com/Packages/com.unity.barracuda@1.0/manual/ModelOutput.html I get the following error after building it to my Android device: JobTempAlloc has allocations that are more than 4 frames old I have read through all the threads adressing this issue but there seems to be no cohesive solution.

AlexRibard commented 3 years ago

On which Barracuda version? Can you try the latest? If it doesn't work that means you are not properly de-allocating Tensors used with the CSharpBurst worker

DaphiFluffi commented 3 years ago

I used barracuda 1.3.0. I tried 1.4.0 now but the exact same thing is happening. This is the code I am using: https://github.com/derenlei/Unity_Detection2AR/blob/main/Assets/Scripts/DetectorYolo3.cs. What could I change?

amirebrahimi commented 3 years ago

@DaphiFluffi - before I try to reproduce can you verify for me which model you are using? Is it a custom one or the one in the project you linked to? If custom, then can you email it to us at barracuda-support at unity3d dot com?

DaphiFluffi commented 3 years ago

@amirebrahimi Yes, it is a custom model. The model from the example project exports fine. Mine doesn't for some reason. I sent it to you via Email just now.

amirebrahimi commented 3 years ago

Thanks, @DaphiFluffi. I'll take a look.

amirebrahimi commented 3 years ago

Hi @DaphiFluffi, so I finally re-created the situation you are experiencing, which is actually something you can ignore. Oddly enough, in the editor when using Burst the messages don't show up, but on Android they do. However, it's just a warning and not an error. The warning is due to memory being released longer than 4 frames since it was first allocated. This occurs because of the use of StartManualSchedule, which only executes a single layer at a time in any given frame. We use a memory allocator that keeps previously allocated buffers around for re-use, so when that memory finally gets released from a previous job it may warn about it.

If you really want to get rid of the errors at the possible expense of performance, then you can do one of the following: Change this line from:

yield return StartCoroutine(worker.StartManualSchedule(inputs));

to

worker.Execute(inputs);

(you'll see that the original author has that commented out below)

OR if you would still like to execute over many frames, you can purge the held JobTempAlloc memory by replacing:

yield return StartCoroutine(worker.StartManualSchedule(inputs));

with

            var workerExecution = worker.StartManualSchedule(inputs);
            int frameCount = 0;
            while (workerExecution.MoveNext())
            {
                frameCount++;
                if (frameCount == 4)
                {
                    frameCount = 0;
                    worker.FlushSchedule(true);
                    yield return null;
                }
            }
            worker.FlushSchedule(true);

If you'd like to communicate more on this issue, then I'd suggest either creating a new issue or re-opening this one since it makes it more difficult for us to track with the issue already being closed.

DaphiFluffi commented 3 years ago

Hi @amirebrahimi,

thank you so much for our help. I actually did not realize it was just a warning and thought it was the reason why the model was not working. I will try out your suggestions on how to remove the warning. However, I think that's it on this issue. Thank you again. 🙏😊