`tflite` newbie ask for help on the `Interpreter` lifetime issue

[wisonye]

@boncheolgu @tylerhawkes Could u have a look and give me help if possible:)

1. Interpreter lifetime issue

Long story in short, I want to create an Interpreter instance which can live as long as with my struct instance, then outside the current lib (API), interpreter.invoke() can get called many times for doing frame-by-frame video objection detection. As tflite doc is broken, only got 2 example cases on the crate home page, it's very difficult for some newbies (like me) to figure out how to use it. I'm getting stuck into the lifetime issue below:

But I jump into the InterpreterBuilder::new source code, It seems it doesn't ask for a reference (should be just any instance that implemented the OpResolver trait?):

I've already tried the different solution, still can't get around....:(

The reason why I'm doing this way is that read the TensorFlow lite C++ API sample, it seems the workflow looks like below:

• Load FlatBufferModel from file (or from another buffer)
• Create Interpreter from the FlatBufferModel
• Use OpenCV to read single image frame and fill the data into the input tensor
• Call Interpreter.invoke()
• Get the output tensor and draw on image frame

And I thought I can keep the same Interpreter instance and reuse it for any incomingInterprepter.invoke() call (I'm not sure that's correct or not), that's why I want to save the Interpreter instance there, plz let me know if I'm making a mistake on this :)

As I saw this, that's why I think @tylerhawkes should know something about that:

Also, I saw this as well, is that means: I can (or should) create a new Interpreter instance for every Interpreter.invoke()? (create an Interpreter instance is cheap?):

---

2. Question about multithreading

In my original plan, I think I should use the separated thread instance to handle theOpenCV reading image frame from webcam. And I saw the tflite issue below which means tflite should support multithreading:

In another hand, I saw the warning in the official tutorial like below:

Any real-life suggestion for using tflite in multithreading solution, plz :)

[tylerhawkes]

You can create a new interpreter with an owned or referenced OpResolver. Here's some of the code I use

    let flat_buffer_model = FlatBufferModel::build_from_buffer(model.bytes().to_vec())?;
    let op_resolver = BuiltinOpResolver::default();
    let builder = InterpreterBuilder::new(flat_buffer_model, op_resolver)?;
    let interpreter = builder.build()?;

If you do it that way then the interpreter will have a static lifetime.

[wisonye]

You can create a new interpreter with an owned or referenced OpResolver. Here's some of the code I use

    let flat_buffer_model = FlatBufferModel::build_from_buffer(model.bytes().to_vec())?;
    let op_resolver = BuiltinOpResolver::default();
    let builder = InterpreterBuilder::new(flat_buffer_model, op_resolver)?;
    let interpreter = builder.build()?;

If you do it that way then the interpreter will have a static lifetime.

I end up with this:

pub struct ObjectDetectionEngine {
    interpreter: Interpreter<'static, BuiltinOpResolver>, // This works
    already_init: bool,
}

///
impl ObjectDetectionEngine {
    /// Load model from file and deal with the init stuff.
    pub fn new() -> Result<Self> {
        let model_load_result = ModelUtil::load_model_from_file()?;
        let object_detection_model = model_load_result.0;
        let object_detection_model_type = model_load_result.1;
        let resolver = BuiltinOpResolver::default();
        let builder = InterpreterBuilder::new(object_detection_model, resolver)?; // This works
        let mut interpreter = builder.build()?;

        interpreter.allocate_tensors()?;

        let inputs = interpreter.inputs().to_vec();
        let input_index = inputs[0];
        let input_tensor = interpreter.tensor_info(input_index).unwrap();
        println!("inputs: {:#?}", &inputs[0]);
        println!("input_tensor: {:#?}", &input_tensor);

        match object_detection_model_type {
            ObjectDetectionModelType::SsdMobileNetByTensorFlow => {
                assert_eq!(
                    input_tensor.dims,
                    vec![
                        ssd_mobilenet::INPUT_IMAGE_TENSOR_BATCH,
                        ssd_mobilenet::INPUT_IMAGE_TENSOR_IMAGE_WIDTH,
                        ssd_mobilenet::INPUT_IMAGE_TENSOR_IMAGE_HEIGHT,
                        ssd_mobilenet::INPUT_IMAGE_TENSOR_COLOR_CHANNELS
                    ]
                );
            }
            ObjectDetectionModelType::MobileObjectByGoogle => {
                assert_eq!(
                    input_tensor.dims,
                    vec![
                        mobile_object::INPUT_IMAGE_TENSOR_BATCH,
                        mobile_object::INPUT_IMAGE_TENSOR_IMAGE_WIDTH,
                        mobile_object::INPUT_IMAGE_TENSOR_IMAGE_HEIGHT,
                        mobile_object::INPUT_IMAGE_TENSOR_COLOR_CHANNELS
                    ]
                );
            }
            ObjectDetectionModelType::Unknown => {}
        }

        // let outputs = interpreter.outputs().to_vec();

        let already_init = true;

        Ok(Self {
            interpreter,
            already_init,
        })
    }

[wisonye]

@tylerhawkes BTW, any suggestion about the multithreading if possible: ) I know all those are not related to a bug of this flite and I suppose NOT to ask a question by firing an issue, but the flite doc is broken and no much detail tutorial about this crate (even a lot of download), that's why I ask a help like this, plz accept my apology.

[tylerhawkes]

calling builder.build_with_threads(threads) instead of builder.build() will allow it to use the number of threads you give it. Remember that using threads won't give a linear speed up, so test out what works well for you. I run our models single threaded mostly so that we have better control over resources and higher throughput at the cost of a little latency.