search

Samsung / ONE

On-device Neural Engine
Other
435 stars 157 forks source link

Compiler FE: Compile open models #1179

Closed seanshpark closed 2 years ago

seanshpark commented 4 years ago

Let's compile open models. Through this we can:

Our CI http://npuci.mooo.com:8080/job/nnfw/job/master/job/daily-publish-package/ does this

seanshpark commented 4 years ago

Step 1; from https://www.tensorflow.org/lite/guide/hosted_models

seanshpark commented 4 years ago

Installed to ~/bin

python ~/bin/tf2tfliteV2.py \
--v1 \
--input_path inception_v3.pb \
--output_path inception_v3_luci.tflite \
--input_arrays input \
--output_arrays InceptionV3/Predictions/Reshape_1
~/bin/tflite2circle \
inception_v3_luci.tflite inception_v3_luci.circle
~/bin/circle2circle \
inception_v3_luci.circle inception_v3_luci_2.circle

No problems in running programs. But the size has changed

-rw-rw-r--  1 95321952 May 20 16:30 inception_v3_luci_2.circle
-rw-rw-r--  1 95324796 May 20 16:30 inception_v3_luci.circle
-rw-rw-r--  1 95326656 May 20 16:28 inception_v3_luci.tflite
-rw-r--r--  1 95660966 Apr 26  2018 inception_v3.pb
-rw-r--r--  1 95325648 Apr 26  2018 inception_v3.tflite
-rw-r-----  1    10484 Mar 27  2018 labels.txt
seanshpark commented 4 years ago

Also no problems running with resnet_v2_101_299.tflite but size changed

-rw-rw-r--  1 178325240 May 20 16:42 resnet_v2_101_299_luci_2.circle
-rw-rw-r--  1 178325240 May 20 16:49 resnet_v2_101_299_luci_3.circle
-rw-rw-r--  1 178328968 May 20 16:41 resnet_v2_101_299_luci.circle
-rw-rw-r--  1 178332096 May 20 16:40 resnet_v2_101_299_luci.tflite

ran again resnet_v2_101_299_luci_2.circle with circle2circle and got same size resnet_v2_101_299_luci_3.circle

seanshpark commented 4 years ago

Also OK with nasnet

-rw-rw-r--  1 354874120 May 20 17:58 nasnet_large_luci_2.circle
-rw-rw-r--  1 354888380 May 20 17:57 nasnet_large_luci.circle
-rw-rw-r--  1 354901388 May 20 17:55 nasnet_large_luci.tflite
-rw-r--r--  1 356602421 Apr 26  2018 nasnet_large.pb

nasnet has these operators

    114 ADD
     70 AVERAGE_POOL_2D
     26 CONCATENATION
    268 CONV_2D
    220 DEPTHWISE_CONV_2D
      1 FULLY_CONNECTED
      8 MAX_POOL_2D
      1 MEAN
      4 MUL
      4 PAD
    154 RELU
      1 SOFTMAX
      4 STRIDED_SLICE

from

one/build/debug/compiler/tfl-inspect/tfl-inspect \
--operators nasnet_large_luci.tflite | sort | uniq -c
seanshpark commented 4 years ago

Step 2; https://www.tensorflow.org/lite/guide/hosted_models#question_and_answer

seanshpark commented 4 years ago 
one/build/debug/compiler/tfl-inspect/tfl-inspect \
--operators mobilebert_1_default_1.tflite | sort | uniq -c
    315 ADD
      1 CAST
      1 CONCATENATION
    578 FULLY_CONNECTED
      2 GATHER
    194 MUL
     48 PACK
      2 PAD
    653 RESHAPE
    384 SLICE
     24 SOFTMAX
      2 STRIDED_SLICE
     24 SUB
    313 TRANSPOSE
      1 UNPACK
~/bin/tflite2circle \
mobilebert_1_default_1.tflite mobilebert_1_default_1.circle

This is OK

~/bin/circle2circle \
mobilebert_1_default_1.circle mobilebert_1_default_1_2.circle

This is not OK

terminate called after throwing an instance of 'oops::InternalExn' what(): Internal Exception. Only support rank 1 CircleConst: 2 [/.../one/compiler/luci/service/src/CircleShapeInferenceRule.cpp:954] Aborted (core dumped)

from 

  loco::NodeShape visit(const luci::CircleReshape *node) final
  {
    const loco::DataType S32 = loco::DataType::S32;

    loco::TensorShape shape_by_input;
    {
      LUCI_ASSERT(node->shape(), "2nd input shape() should not be nullptr");

      // Only support node's shape() is CircleConst with S32
      // TODO support other node with other types
      auto const_shape_node = loco::must_cast<luci::CircleConst *>(node->shape());
      LUCI_ASSERT(const_shape_node, "Only support CircleConst for shape of CircleReshape");
      LUCI_ASSERT(const_shape_node->dtype() == S32, "Only support int32 CircleConst");

      if (const_shape_node->rank() != 1)
        INTERNAL_EXN_V("Only support rank 1 CircleConst",
                       oops::to_uint32(const_shape_node->rank()));

With 0478cb4b385e34fe3841a99fa3accfaad0ad50eb , it is now OK

seanshpark commented 4 years ago

Name of the Reshape node: bert/embeddings/ExpandDims

seanshpark commented 4 years ago 
O(0:1) RESHAPE 
    NewShape(1, 384, 1)
    I T(5422) input_ids
    I T(6) bert/embeddings/ExpandDims/dim_0
    O T(5) bert/embeddings/ExpandDims
T(0:6) INT32 (1, 3) B(2165) bert/embeddings/ExpandDims/dim_0
seanshpark commented 4 years ago

Step 3; https://www.tensorflow.org/lite/guide/hosted_models#image_segmentation

seanshpark commented 4 years ago 
one/build/debug/compiler/tfl-inspect/tfl-inspect \
--operators deeplabv3_1_default_1.tflite | sort | uniq -c
     10 ADD
      1 AVERAGE_POOL_2D
      1 CONCATENATION
     38 CONV_2D
     17 DEPTHWISE_CONV_2D
      3 RESIZE_BILINEAR

RESIZE_BILINEAR is not ready yet.

seanshpark commented 4 years ago

(Reserved this comment for deeplabv3_1_default_1.tflite after RESIZE_BILINEAR lands)

~/bin/tflite2circle \
deeplabv3_1_default_1.tflite deeplabv3_1_default_1.circle

OK

~/bin/circle2circle -all \
deeplabv3_1_default_1.circle deeplabv3_1_default_1.2.circle

OK

seanshpark commented 4 years ago

Step 4; https://www.tensorflow.org/lite/guide/hosted_models#smart_reply

seanshpark commented 4 years ago 
one/build/debug/compiler/tfl-inspect/tfl-inspect \
--operators smartreply_1_default_1.tflite | sort | uniq -c
      1 CUSTOM(ExtractFeatures)
      1 CUSTOM(Normalize)
      1 CUSTOM(Predict)
      1 HASHTABLE_LOOKUP
      1 LSH_PROJECTION
      1 SKIP_GRAM

Unknown custom ops and unsupported yet Ops...

Let's comeback with this laaaaater...

seanshpark commented 4 years ago

With one-cmds for 

one-import tf  \
--input_path inception_v3.pb \
--output_path inception_v3.circle \
--input_arrays input --input_shapes "1,299,299,3" \
--output_arrays InceptionV3/Predictions/Reshape_1
one-pack -i inception_v3.circle -o inception_v3
BACKENDS="cpu" nnpackage_run --num_runs=10 --warmup_runs=1 \
inception_v3/inception_v3

invalid

one-import tf  \
--input_path inception_v3.pb \
--output_path inception_v3.circle \
--input_arrays input --input_shapes "1,299,299,3" \
--output_arrays 
one-import tf  \
--output_path inception_v3.circle \
--input_arrays input --input_shapes "1,299,299,3" \
--output_arrays InceptionV3/Predictions/Reshape_1 \
--input_path
one-import tf  \
--input_path inception_v3.pb \
--input_arrays input --input_shapes "1,299,299,3" \
--output_arrays InceptionV3/Predictions/Reshape_1 \
--output_path
one-import tf  \
--input_path inception_v3.pb \
--output_path inception_v3.circle \
--output_arrays InceptionV3/Predictions/Reshape_1 \
--input_arrays
seanshpark commented 4 years ago

with mobilenet_v2_1.4_224

one-import tf  \
--input_path mobilenet_v2_1.4_224.pb \
--output_path mobilenet_v2_1.4_224.circle \
--input_arrays input --input_shapes "1,224,224,3" \
--output_arrays MobilenetV2/Predictions/Reshape_1
one-pack -i mobilenet_v2_1.4_224.circle -o mobilenet_v2_1.4_224
BACKENDS="cpu" nnpackage_run --num_runs=10 --warmup_runs=1 \
mobilenet_v2_1.4_224/mobilenet_v2_1.4_224
seanshpark commented 4 years ago 
one-import tf  \
--input_path resnet_v2_101_299_frozen.pb \
--output_path resnet_v2_101_299_frozen.circle \
--input_arrays input --input_shapes "1,299,299,3" \
--output_arrays output
one-optimize --all --input_path resnet_v2_101_299_frozen.circle \
--output_path resnet_v2_101_299_frozen.o.circle
one-pack -i resnet_v2_101_299_frozen.o.circle -o resnet_v2_101_299_frozen
BACKENDS="cpu" nnpackage_run --num_runs=10 --warmup_runs=1 \
resnet_v2_101_299_frozen/resnet_v2_101_299_frozen.o
seanshpark commented 4 years ago 
one-import tf  \
--input_path nasnet_large.pb \
--output_path nasnet_large.circle \
--input_arrays input --input_shapes "1,331,331,3" \
--output_arrays final_layer/predictions
one-optimize --all --input_path  nasnet_large.circle \
--output_path nasnet_large.o.circle
one-pack -i nasnet_large.o.circle -o nasnet_large
BACKENDS="cpu" nnpackage_run --num_runs=10 --warmup_runs=1 \
nasnet_large/nasnet_large.o
seanshpark commented 4 years ago 
one-import tflite  \
--input_path mobilebert_1_default_1.tflite \
--output_path mobilebert_1_default_1.circle
one-pack -i mobilebert_1_default_1.circle -o mobilebert_1_default_1
BACKENDS="cpu" nnpackage_run --num_runs=10 --warmup_runs=1 \
mobilebert_1_default_1/mobilebert_1_default_1
one-optimize --all --input_path mobilebert_1_default_1.circle --output_path mobilebert_1_default_1.o.circle
one-pack -i mobilebert_1_default_1.o.circle -o mobilebert_1_default_1_o
BACKENDS="cpu" nnpackage_run --num_runs=10 --warmup_runs=1 \
mobilebert_1_default_1_o/mobilebert_1_default_1.o

invalid

one-import tflite  \
--input_path mobilebert_1_default_1.tflite \
--output_path

one-import tflite  \
--output_path mobilebert_1_default_1.circle \
--input_path

one-import tflite  \
--input_path --output_path
seanshpark commented 4 years ago

Test with invalid file

one-import tf  \
--input_path nasnet_large.circle \
--output_path nasnet_large.2.circle \
--input_arrays input --input_shapes "1,331,331,3" \
--output_arrays final_layer/predictions
one-optimize --all --input_path  nasnet_large.pb \
--output_path nasnet_large.2.circle
seanshpark commented 2 years ago

Closing.