Object detection for dummies

[bartbutenaers]

Dear @yhwang,

Thanks for sharing this interesting node!

I would like to detect objects (e.g. human body or a car or ...) in images from my IP cameras. I assume that is possible somehow with your node?

But unfortunately I have no experience with machine learning... And I have no clue how to implement object detection on images in Node-RED. Where should I find the model (in which format), do I have to train anything, how do I supply the image, and so on ...

Would be very nice if you could add somewhere (e.g. on the wiki of this repository) some kind of step-by-step tutorial, for dummies and idiots like me :-)

Thanks a lot for your time!! Bart

[yhwang]

Sure! Actually, I just finished my speech to text flow and object detection would be my next goal. I will work on it and share it here soon.

For object dection, Coco SSD would be a good one. And I would use it in my flow. If you are good at JavaScript programming, you can check the documentation in that Coco SSD link. If you prefer the Node-RED and would like to use drag and drop style for composing the data flow, I would provide a detailed step-by-step tutorial later.

[bartbutenaers]

Hi @yhwang, Thanks for the pointer. Will have a look at it.
Kind regards, Bart

[yhwang]

I added an example flow here: https://github.com/yhwang/node-red-contrib-tf-model/tree/master/examples/object-detection and it's for object detection. Hope this may help you.

[johnwalicki]

@ bartbutenaers I've successfully implemented a few flows using the node-red-contrib-tf-model - If you want some assistance, I'd be happy to help.

[bartbutenaers]

Hi @yhwang , @johnwalicki, Thanks a lot for the great support!!
But I had a surgery this week. Will contact you guys as soon as I'm back into business :-) Kind regards, Bart

[viacheslav-dobrynin]

Hello @johnwalicki ! I would like to get a simple linear regression in my flow. But when I load my model in JSON format, the node throws an error: "Cannot read property 'producer' of undefined". I searched the internet. As I understand it, the problem is loading the model, but here the node controls it. I am not a JavaScript developer. Could you tell me what the problem is? Thanks in advance.

Best Regards! Dobrynin Vyacheslav.

[bartbutenaers]

@yhwang , @johnwalicki, I had completely lost this issue out of sight. My sincere apologies for that!!! I'm too busy with other Node-RED stuff at the moment, but I will surely follow this discussion! Thanks! Bart

[Viggen-l]

Hi @yhwang and @bartbutenaers I have followed the node-red object detection example and everything is running with the cocco ssd model. But i need to use an other special model, So i trained one at https://teachablemachine.withgoogle.com with the 4 new classes needed. When i try to load this i get the same error as @johnwalicki: "Cannot read property 'producer' of undefined" Same error both with link to download page and from file.

Can any one of you explaine what's wrong?

[yhwang]

@Viggen-l when loading different model, you need to know what inputs are needed for the model you are using. The main purpose of tf-model node is to load tf.js models. You have to handle the data pre-processing and post-processing separately. After you specify the URL/file path in the tf-model node and deploy the flow, you should be able to see a log message from the console of Node-RED process telling you what inputs the model needs. Then you have to provide the corresponding named object for each input your model needs in the pre-processing node and pass the data to tf-model node.

Could you share detailed message you see from the console and also the model? Then maybe I can help you to figure out what the problem is.

[Viggen-l]

Hi Yes all help is appreciated, I stumbled into AI for picture handling a few weeks ago and made some test with python (and it's working with the trained model). But realised that Node-red would be a better solution for maintainace and monitoring in a live system. So know i'am lerning javascript as well :), A lot of new things for the last weeks.

I made a quick sampel model that can be found here https://teachablemachine.withgoogle.com/models/KhW7t95Hm/model.json There are 4 classes (Class 1, Class 2, Class 3, Class 4) And it's trained on loaded pallets, 1 and 2 are bad pallets, 3 are empty woodpallets, and 4 are correct load (also dogs and other stuff ends up here if you try).

Setup running on a Pi 4 8gb ram. so first run with the model in the example flow to verify setup and it's working, then changed model.

From debug window in Node-Red 2020-08-13 09:20:43node: COCO SSD lite msg : string[44] "Cannot read property 'producer' of undefined"

In console 13 Aug 09:20:15 - [info] Starting flows 13 Aug 09:20:16 - [info] Started flows 13 Aug 09:20:17 - [info] [tf-model:COCO SSD lite] loading model from: https://teachablemachine.withgoogle.com/models/KhW7t95Hm/model.json 13 Aug 09:20:17 - [error] [tf-model:COCO SSD lite] Cannot read property 'producer' of undefined

I think it's beacuse the base models are different. tf-model (coco) are used for objectdetection and teachablemachine produce an object cassification model.

i looked into the example node-red-contrib-teachable-machine (only working on windows) what was setup, but as i can't find the real node-red parts for tf-model code i cant compare.

I hope you can find something. Teachablemachin offers a very easy way of training a model so to be able to run it in Node-Red would be great.

[Viggen-l]

So i actually made some progress today, I found where the node-red code was hiding (index.js dep down). And the problem "Cannot read property 'producer' of undefined" looks like it's solved, The T.machine model say it's a 'layers-model' type and that's not handel so it was loaded with loadGraphModel. Added check for that so it's loaded with loadLayersModel and it's happy so far.

Know there is a new problem this model gives an empty input array: [info] [tf-model:COCO SSD lite] input(s) for the model: [] and when and prediction are made it's crashing. I have no clue where the code for the prediction are made so my journey ends here.

Log from console. 13 Aug 14:15:38 - [info] Starting flows 13 Aug 14:15:38 - [info] Started flows 13 Aug 14:15:38 - [info] [tf-model:COCO SSD lite] loading model from: https://teachablemachine.withgoogle.com/models/KhW7t95Hm/model.json 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] loaded model type: layers-model 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] Loading as: layers-model 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] model loaded 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] input(s) for the model: [] 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] model name: "sequential_4"

Here a prediction are started 13 Aug 14:16:03 - [info] [tf-model:COCO SSD lite] Run as LayerModel: 13 Aug 14:16:03 - [error] [tf-model:COCO SSD lite] Error: Error when checking model : the Array of Tensors that you are passing to your model is not the size the the model expected. Expected to see 1 Tensor(s), but instead got 0 Tensors(s).

[yhwang]

@Viggen-l the information you provided above are awesome. And you're right about the LayerGraph. The checking criteria is the value of format property needs to be 'layers' and apparently the value of the format property in your model is 'layers-model'. I can change the comparison from "strict equal" to "startsWith"

And the input for your model should be an array of tensor and size should be 1 and the image size needs to be [224, 224]. I tried the following pre-processing code in a tf-function node and linked it to the tf-model node:

const image = tf.tidy(() => {
  return tf.image.resizeBilinear(tf.node.decodeImage(msg.payload, 3), [224,224]).expandDims(0);
});

msg.image = msg.payload;
msg.payload = [ image ]; // tensor array with size 1
return msg;

I used bilinear sampling to resize the image to [224, 224]. You can use tf.image.resizeNearestNeighbor if you prefer nearest neighbor sampling.

[yhwang]

BTW, the inference result is the probabilities of your 4 classes, i.e.:

Tensor
     [[0.000408, 0.0019049, 0.0000467, 0.9976404]]

[Viggen-l]

Thank you very mutch for all your the help here. I will be of for a week vacation so i can't try it ot right away.

If you ever think of dooing an other tutorial about AI and tensorflow, can it be about how to use preetrained models. How to find out what the input/output parameters are like you showed above. That would be greate because most of the information focus on training a model.

again thank's so far. i will notify when it's tested

[Viggen-l]

So back in business, The preprocess are working and are receiving predictions as Tensor [[0.005894, 0.0066852, 0.0068979, 0.9805231],], So far so good, But the result are always sorted, what ever class picture is sent in the predictions are returned with lowest in [0] and highest in [3]. what do I misunderstand about this result?

[yhwang]

it shouldn't be sorted data. it should be indexed by your classes. for example, you result above

[[0.005894, 0.0066852, 0.0068979, 0.9805231],],

means probability for class 1 is 0.005894%, probability for class 2 is 0.0066852% and so on. I don't know why your 4th class always got the highest probability.

Can you retrain your model and try it again?

[Viggen-l]

God evening, I will try to retrain the model tomorrow. To convert the tensor data into a usable data array i used the function dataSync() (squesed it into a standard array of float32). Just found some notes that this could destroy the order of the array (https://github.com/tensorflow/tfjs/issues/939). Are there any other way of converting the tensor array into an array that could be used in a for loop?

[yhwang]

you can try arraySync() instead like the last comment in the issue you pasted above.

[Viggen-l]

So, I retrained the model (https://teachablemachine.withgoogle.com/models/xGyfSjOpI/) 4 classes as before example for each class added. Tested it online and it's working on webpage. But the result from tf_mode are that highest prediction always ends up at [3] (what ever picture sent in even a cat :)), before it was at [4].

So I got a filling that the model was not using the correct data. A Teachable machine model contains 3 files. model.json, metadata.json and weights.bin. the only file loaded by tf_model are model.json. i can remove the other 2 without any error. So I think that the model are running "untrained" bacause it's not loading the weights correctly,

If i remove a file from the exampel node-red-contrib-teachable-machine node, it's rasing an error for missing file.

Class 1

Class2

Class3

Class4

[Viggen-l]

debug from consol running the pictures above class 1-4 25 Aug 10:29:56 - [info] [tf-model:COCO SSD lite] Result : Tensor [[0, 0, 1, 0],] 25 Aug 10:30:07 - [info] [tf-model:COCO SSD lite] Result : Tensor [[0.0000017, 0.0000013, 0.99998, 0.0000169],] 25 Aug 10:30:20 - [info] [tf-model:COCO SSD lite] Result : Tensor [[0, 1e-7, 0.999999, 9e-7],] 25 Aug 10:30:29 - [info] [tf-model:COCO SSD lite] Result : Tensor [[0, 0, 1, 0],]

in the index.js file just print result straight out.

        result.then(function (result) {
            var msg = origMsg;
            _this.log("Result : " + result);
            //_this.log("Result dataSync: " + result.dataSync());
            //_this.log("Result arraySync: " + result.arraySync());                                
            msg.payload = result;

            _this.send(msg);
            _this.cleanUp(inputs);
        })

[yhwang]

@Viggen-l can you modify your pre-processing node to have the following code:

const image = tf.tidy(() => {
  return tf.image.resizeBilinear(tf.node.decodeImage(msg.payload, 3), [224,224]).expandDims(0)
      .toFloat().div(tf.scalar(127)).sub(tf.scalar(1));
});
msg.image = msg.payload;
msg.payload = [image ]
return msg;

I forgot to normalize the image before sending the tensor for inference...

[Viggen-l]

Hi. yes that fixed the problem. Realized that i did something simular in the Python script i tested before moving to Node-Red. But (there is always a but here it feels like) when i try some pictures i get different result beteween prediction on webpage and tf-model.

if i send any of these pictures below into the webpage results are 100% class 1 and thats correct. Same pictures in tf-model say highest score class4 and thats wrong. How can the results be so different? Can it be that the teachable machine preprocess crop the pictures different?

Class1 (17).jgp tested.

Tensor: [[0.0001826, 0.0000011, 0.0000268, 0.9997897],]

Class1 (27) tested.

Tensor: [[0.4584374, 0.000126, 0.0163986, 0.5250379],]

[yhwang]

I checked the exported model and model loading procedure. They should be correct. You should be right. I guess the possible reason is pre-processing logic. The logic in teachablemachine website is here: https://github.com/googlecreativelab/teachablemachine-community/blob/master/libraries/image/src/utils/canvas.ts#L42-L72

The only difference is it try not to change the width/height ratio when doing the resize. One thing you can try is to use a image with size 224x244 and feed it to teachablemachine and the node-red. And see if the result is the same. If yes, then resize could be problem.

[Viggen-l]

Thank you for the information. I have talked to my colleagues and we decided that we want this up and running so we are going with windows. Maby we come back and continue with a pi/linux setup or an Ai enabled camera we don't know, For know we are just aming for a "Prof of concept" in live production.

Thanks for all help with this. There are alot of detailes hidden behinde that all this tutorials misses to mention.

[Viggen-l]

i hade some time left over today, So the code you point at are as i understand dynamically croping the picture depending size and scale before rescaling it. Just to try this out i made a static crop by 100pixcel when loading a picture before the preprocess and with that the prediction are givning correct result. So to make this work permenently maby croping 100 pixcels are ok (always the same camera in our solution) or try to recreate the dynamic croping with a function.

Thanks agine for all help.

[yhwang]

I guess the performance of running your image classification model on PI should be acceptable. If you want to have GPU acceleration, you can try Jetson Nano.

Agree with you that the data preparation is important. It impacts the inference result a lot. When people publish a pre-trained model, sometimes they forget to explain the data pre-processing procedures. And different model has different data pre-processing. There is no one approach that could fit for all. Like you did for your model, I have tried to figure out the pre-processing for several models. I have the same feelings as you. Congratulation to you, you figure it out!

[Viggen-l]

Again thank's a lot for the help here, i have learned many new things both AI and Javascript (have not worked with it before) and also discovered what a big task it is to realy understand what to do with AI. Training and predition is not the big thing. It's all preparations that's hard to setup and understand.

If you ever do an other clip/tutorail i think a god topic are about how to document and figure out models.

[Vickpire]

So i actually made some progress today, I found where the node-red code was hiding (index.js dep down). And the problem "Cannot read property 'producer' of undefined" looks like it's solved, The T.machine model say it's a 'layers-model' type and that's not handel so it was loaded with loadGraphModel. Added check for that so it's loaded with loadLayersModel and it's happy so far.

Know there is a new problem this model gives an empty input array: [info] [tf-model:COCO SSD lite] input(s) for the model: [] and when and prediction are made it's crashing. I have no clue where the code for the prediction are made so my journey ends here.

Log from console. 13 Aug 14:15:38 - [info] Starting flows 13 Aug 14:15:38 - [info] Started flows 13 Aug 14:15:38 - [info] [tf-model:COCO SSD lite] loading model from: https://teachablemachine.withgoogle.com/models/KhW7t95Hm/model.json 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] loaded model type: layers-model 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] Loading as: layers-model 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] model loaded 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] input(s) for the model: [] 13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] model name: "sequential_4"

Here a prediction are started 13 Aug 14:16:03 - [info] [tf-model:COCO SSD lite] Run as LayerModel: 13 Aug 14:16:03 - [error] [tf-model:COCO SSD lite] Error: Error when checking model : the Array of Tensors that you are passing to your model is not the size the the model expected. Expected to see 1 Tensor(s), but instead got 0 Tensors(s).

Hi Viggen-I,

I am currently facing the same error when trying to load my converted from .h5 to .json model: "Cannot read property 'producer' of undefined". You said you were able to fix by accessing index.js of node-red. Could you please be more specific to as where this file is located and what exactly needs to be changed so I can modify it and make my model work? I have to specify, my model is also a layers-model.

Thank you in advance for your time and answer.

Best regards, Victoria

[Viggen-l]

I will look into that, i have not touched that code for 1 year so I will see if i can remember what i changed. But with the latest version of node-red teachable machine and models from googles pages there is no problem, How have you trained you model?

[Vickpire]

I will look into that, i have not touched that code for 1 year so I will see if i can remember what i changed. But with the latest version of node-red teachable machine and models from googles pages there is no problem, How have you trained you model?

I had trained the model using keras and tensorflow in python and then converted it to the .json format using tfjs-converter.

[Viggen-l]

Looked into the code today and first of all the version i downloaded today was only 5 days old so i guess much have changed since i was working on it a year ago i did not even finde the index.js file for some reason. But what i do remeber are that i never hade it working on Linux i run into more problems after the fixex mentioned here, so we decided to move to windows insted where all was working fine.

We have 5 systems up and running withe Node-red and the teachablemachine block and it's in production 24/7. We train our models on googles page https://teachablemachine.withgoogle.com/train/image and so far just for image classification it's working greate.

i will take a look for the index.js file when i have more time. But try the google trainingpages,

[Vickpire]

Looked into the code today and first of all the version i downloaded today was only 5 days old so i guess much have changed since i was working on it a year ago i did not even finde the index.js file for some reason. But what i do remeber are that i never hade it working on Linux i run into more problems after the fixex mentioned here, so we decided to move to windows insted where all was working fine.

We have 5 systems up and running withe Node-red and the teachablemachine block and it's in production 24/7. We train our models on googles page https://teachablemachine.withgoogle.com/train/image and so far just for image classification it's working greate.

i will take a look for the index.js file when i have more time. But try the google trainingpages,

Hi Viggen-I,

We had tried a series of possible solutions but nothing worked so far... Were you able to find more about what was done for your model to work?

Unfortunately our model is an object detection model which was trained in python with keras and tensorflow. I had also opened another Issue but the author hasn't responded yet.

Thank you in advance.

Best regards, Victoria

[Viggen-l]

Hi Sorry to say that i have been busy with other things. Googles training page are only available for classification so then it's not an option. Have you tried this node block? https://flows.nodered.org/node/node-red-contrib-tensorflow the example show object detection.

[GTOVA12]

@Viggen-l the information you provided above are awesome. And you're right about the LayerGraph. The checking criteria is the value of format property needs to be 'layers' and apparently the value of the format property in your model is 'layers-model'. I can change the comparison from "strict equal" to "startsWith"

And the input for your model should be an array of tensor and size should be 1 and the image size needs to be [224, 224]. I tried the following pre-processing code in a tf-function node and linked it to the tf-model node:

const image = tf.tidy(() => {
  return tf.image.resizeBilinear(tf.node.decodeImage(msg.payload, 3), [224,224]).expandDims(0);
});

msg.image = msg.payload;
msg.payload = [ image ]; // tensor array with size 1
return msg;

I used bilinear sampling to resize the image to [224, 224]. You can use tf.image.resizeNearestNeighbor if you prefer nearest neighbor sampling. Hi sorry to bother you guys but i face a similar problem when i pass the image to the tf-model @yhwang ""TypeError: Cannot read properties of undefined (reading 'backend')" i don't know why i got this error i think its because of the preprocessing node i got another error before this one which is " Cannot read property 'length' of undefined" all help are apreciated