search

tensorflow / tensorflow

An Open Source Machine Learning Framework for Everyone
https://tensorflow.org
Apache License 2.0
185.53k stars 74.18k forks source link

Parse toco generated file (.tflite) in python? #16561

Closed santoshchilkunda closed 6 years ago

santoshchilkunda commented 6 years ago

System information

Describe the problem

I am using toco to optimize a frozen model (.pb). How do I read the .tflite file in python - something similar to tf.gfile.GFile('frozen.pb', 'rb')?

reedwm commented 6 years ago

/CC @bjacob

santoshchilkunda commented 6 years ago

bump

tensorflowbutler commented 6 years ago

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

aselle commented 6 years ago

@santoshchilkunda, what exactly are you trying to do once you read it? Do you want to run inference on it. Do you want to analyze it?

santoshchilkunda commented 6 years ago

@aselle Thanks for looking into this.

I am trying to read/parse the model, convert it to our internal format, and then run inference on it.

As a workaround (to not being able to parse tflite), I set the output_format while running toco tool to TENSORFLOW_GRAPHDEF (i.e. both input and output formats are TENSORFLOW_GRAPHDEF). And then parse the generated protobuf.

However, I see that protobuf is "less" optimized compared to tflite

Following is the log when output_format is TENSORFLOW_GRAPHDEF: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 773 operators, 1072 arrays (0 quantized) I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 141 operators, 260 arrays (0 quantized) I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 141 operators, 260 arrays (0 quantized) I tensorflow/contrib/lite/toco/toco_tooling.cc:273] Estimated count of arithmetic ops: 3.01265 billion (note that a multiply-add is counted as 2 ops).

Following is the log when output_format is set to TFLITE: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 773 operators, 1072 arrays (0 quantized) I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 84 operators, 203 arrays (0 quantized) I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 84 operators, 203 arrays (0 quantized) I tensorflow/contrib/lite/toco/allocate_transient_arrays.cc:313] Total transient array allocated size: 4014080 bytes, theoretical optimal value: 4014080 bytes. I tensorflow/contrib/lite/toco/toco_tooling.cc:273] Estimated count of arithmetic ops: 3.01265 billion (note that a multiply-add is counted as 2 ops).

The number of operators after graph transformation is different (141 v/s 84) while the estimated count of arithmetic ops is same (3.01265 billion). From a performance perspective, I am not completely sure if this means both the output formats are same or not.

aselle commented 6 years ago

well you can use flatc to generate a python api that can read the tflite format. https://google.github.io/flatbuffers/flatbuffers_guide_use_python.html Does that answer your question?

santoshchilkunda commented 6 years ago

I will try it and update. Thanks!

Regarding the second part of my question, is there a performance difference between the two output formats?

bjacob commented 6 years ago

@santoshchilkunda, it is an inherent characteristic of the TensorFlow Lite flatbuffer format that it allows to represent the same neural network in fewer nodes than are needed in the TensorFlow GraphDef format, as you found from this logging. To find out more about what the difference is in your graph, use --dump_graphviz as explained there, https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/toco/g3doc/cmdline_examples.md#using---dump_graphviz

santoshchilkunda commented 6 years ago

dump_graphviz worked, thanks! Still working on flatc method that was suggested

aselle commented 6 years ago

Another approach is to generate json from the flatbuffer using flatc. This is used by the tflite visualizer: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/tools/visualize.py Please close the issue if it is resolve. Thanks!

santoshchilkunda commented 6 years ago

Thanks for all the suggestions. dump_graphviz served my purpose. Will explore flat buffer method soon.

cumberb1tch commented 5 years ago

well you can use flatc to generate a python api that can read the tflite format. https://google.github.io/flatbuffers/flatbuffers_guide_use_python.html Does that answer your question?

@aselle could you please make it a little bit clear for me

I am using pyhon generated code to read and get node attributes from the model graph.

I generated it this way (output is tflite/ folder with autogenerated *.py files): flatc -python tensorflow/tensorflow/lite/schema/schema.fbs

Than I read the model: 

    from tflite.Model import Model
    def read_tflite_model(file):
        buf = open(file, "rb").read()
        buf = bytearray(buf)
        model = Model.GetRootAsModel(buf, 0)
        return model

Getting model parameters:

    def print_model_info(model):
        version = model.Version()
        print("Model version:", version)
        description = model.Description().decode('utf-8')
        print("Description:", description)
        subgraph_len = model.SubgraphsLength()
        print("Subgraph length:", subgraph_len)

Than I realized that graph nodes could be interpreted as Tensor object or Operator object.

Tensor object stores quantization params, shape, tensor type (I don't understand the meaning of it yet)

Operator object has BuiltinOptions and CustomOptions methods that seems to me should give me access to node parameters (such as paddings, dilations and all layer specific info)

I tried to iterate over them since there are Inputs and Outputs methods. But failed. I don't understand w

This is my scratch:

def print_nodes_info(model):
    # what does this 0 mean? should it always be zero?
    subgraph = model.Subgraphs(0)
    operators_len = subgraph.OperatorsLength()
    print('Operators length:', operators_len)

    from collections import deque
    nodes = deque(subgraph.InputsAsNumpy())

    STEP_N = 0
    MAX_STEPS = operators_len
    print("Nodes info:")
    while len(nodes) != 0 and STEP_N <= MAX_STEPS:
        print("MAX_STEPS={} STEP_N={}".format(MAX_STEPS, STEP_N))
        print("-" * 60)

        node_id = nodes.pop()
        print("Node id:", node_id)

        tensor = subgraph.Tensors(node_id)
        print("Node name:", tensor.Name().decode('utf-8'))
        print("Node shape:", tensor.ShapeAsNumpy())

        # which type is it? what does it mean?
        type_of_tensor = tensor.Type()
        print("Tensor type:", type_of_tensor)

        quantization = tensor.Quantization()
        min = quantization.MinAsNumpy()
        max = quantization.MaxAsNumpy()
        scale = quantization.ScaleAsNumpy()
        zero_point = quantization.ZeroPointAsNumpy()
        print("Quantization: ({}, {}), s={}, z={}".format(min, max, scale, zero_point))

        # I do not understand it again. what is j, that I set to 0 here?
        operator = subgraph.Operators(0)
        for i in operator.OutputsAsNumpy():
            nodes.appendleft(i)

        STEP_N += 1

    print("-"*60)

Please help me to get access the node attributes. Thank you in advance for your help.

Nagamani732 commented 3 years ago

Hi all It's too late but it might be useful for someone who needs it. Here, you will find the TFLite Parser Python Package: https://github.com/jackwish/tflite

Thanks jackwish for the wonderful package.