problem using automl node "No data found in msg.payload.payload"

[barney2074]

Hi Kolban

I'm trying to use the 'automl' node with a custom object detection model.

The 'vision' nodes work fine- but for the 'automl' node, I get an error No data found in msg.payload.payload

For the 'vision' node, it works ok with either a local image or the url of a public image For the 'automl' node- I've tried adding in a 'change' node to get the image back into msg.payload, with no success & I've been through the Google api documentation-

Hopefully I'm close- but would be very grateful if you could assist with this

thanks

Andrew

[kolban-google]

Howdy Andrew, First of all thanks for raising the post and sorry for any issues you are having based on this contribution. I'm here to help. I noticed that I failed to add any documentation on this node and have fixed that now here:

https://github.com/GoogleCloudPlatform/node-red-contrib-google-cloud/blob/master/docs/automl.md

I had to re-read the docs myself ... what the node does is execute an AutoML prediction. We need to provide a "payload" for the prediction. What is a payload? That is what is documented here. The location of that payload in NodeRED should be at:

msg.payload.payload

Let's look and see see what you are passing in as a payload to automl. My guess is that you are setting the data at "msg.payload" and not at "msg.payload.payload". If you want to share a what the input to the automl node looks like we can debug together.

[barney2074]

Hi Kolban

I was passing in an image as a buffer directly into the automl node (in the same way as the vision node) so that it obviously where I've gone wrong.

thanks for the doc- it is useful

So I guess my (optional) params would be something like: params = {"score_threshold": "0.6", "max_bounding_box_count": "2"}

So the question is how to get the image into a child payload I would certainly appreciate your assistance- my node-red skills are quite low, but trying my best to learn it...!

What is the best way to share this with you?

• the flow is not exactly top-secret, but I'm not sure if it includes API key/secret etc. For now, I've just included a screenshot

• 

many thanks

Andrew

[kolban-google]

Let's imagine that Node-RED wasn't in the picture. How are your skills on using Google's AutoML technology without Node-RED? Can you explain some about how you went about training your model?

My immediate guess is that you will populate the Node-RED data structure called msg.payload with:

msg.payload = {
  "payload": {
    "image": {
      "imageBytes": <String .. a base64 encoding of your image>
    }
  },
  "params": {
    "score_threshold": "0.6",
     "max_bounding_box_count": "2"
  }
}

Maybe try calling your prediction service using the GCP API Explorer low level mechanism to validate that all is working for you at the API level before going to Node-RED. The automl Node-RED node is using this NodeJS package:

https://cloud.google.com/nodejs/docs/reference/automl/latest/automl/v1.predictionserviceclient

[barney2074]

Hi Neil

I trained the object detection model (just a proof-of-concept at this stage) by loading several thousand images, with corresponding labels/annotations, using the web GUI. Although laborious to label, I found the process very easy The model seems to work o.k, although is overfitting- but that is ok for a POC

My coding skills are pretty low- I do my best to blunder my way through. I tried the node-red option, because it seemed to work well with other ML nodes, as long as I plugged in the right information I've been trying to get the Python version to work- https://cloud.google.com/vision/automl/docs/tutorial- the roadblock at this stage is authentication. The documentation seems a bit different from the current system- but I'll persevere

I'll take a look at the GCP API Explorer as you've suggested & play around with node-red as well

many thanks

Andrew

[barney2074]

Hi Neil

Just to give you some context on what I am trying to achieve:

• a proof of concept of an ML-based reporting/monitoring system of construction vehicles and machinery
• POC uses photos of toy models- as above, this is what I've labelled
• To mimimise work- bolting together off-the shelf applications wherever possible

My current thinking of overall architecture is:

• images captured at a specific interval- say every 10 seconds for each location
• images analysed by automl, and detections written to a database- I'll probably use InfluxDB as I'm familiar with this and it deals with time-series data easily. So data would be something like 'concrete truck at gate D from X:Y timestamps
• Node-red could probably do this- including the automl queries and writing automl response to DB
• Reporting dashboard from the database- I'll probably use Grafana as it works well with Influx. So this could report that 10 concrete trucks have passed gate D in the last hour, and 3 excavators are in area B

Andrew

[kolban-google]

Howdy Andrew, This looks like a pretty good example for performing predictions. If we look at the JavaScript code, we see the core of what we are trying to do in Node-RED

https://cloud.google.com/vision/automl/docs/samples/automl-vision-classification-predict#automl_vision_classification_predict-nodejs

The core to make a prediction is to have a model deployed, convert the image into base64 data and then pass that in through the msg.payload.payload.image.imageBytes.

I'd suggest spiltting the project into multiple parts including preparing the data, training the model, deploying the model and then making predictions. I have the feeling that you are already at the making predictions stage ... the core (to my mind) is to realize that you are making an API call to GCP AutoML. You supplying input data (a base64 encoded image) and the identity of the model you want to predict against. You then make the API call and you get a result.

[barney2074]

Hey Neil,

I've got it working with Python, but despite my best efforts, just can't get it working with the node-red nodes- this would be my first preference. The other GCP nodes are easy to use.

For now, I'm just calling the Python from node-red which also works. When my patience/spirit recovers- I might revisit node-red GCP- otherwise, if you get a chance, a sample would be great

Just a question on the response payload. The structure of the data (at least using the Python option) is a little strange in that the 'Predicted class name' doesn't have a score, and seems to be a repeat of data in that section. Is there any documentation on this, or any options to format differently. I think for my purposes, I just want the 'display_name' and 'score'

thanks again

Predicted class name: truck
Predicted class score: 0.0
[annotation_spec_id: "481228201931046912"
image_object_detection {
  bounding_box {
    normalized_vertices {
      x: 0.8229872584342957
      y: 0.5793604850769043
    }
    normalized_vertices {
      x: 0.9742650985717773
      y: 0.6732610464096069
    }
  }
  score: 0.9870228171348572
}
display_name: "truck"
, annotation_spec_id: "2041373629094035456"
image_object_detection {
  bounding_box {
    normalized_vertices {
      x: 0.005605409853160381
      y: 0.5414829254150391
    }
    normalized_vertices {
      x: 0.1891111582517624
      y: 0.7010807991027832
    }
  }
  score: 0.9359530210494995
}
display_name: "excavator"
]
Predicted class name: excavator
Predicted class score: 0.0
[annotation_spec_id: "481228201931046912"
image_object_detection {
  bounding_box {
    normalized_vertices {
      x: 0.8229872584342957
      y: 0.5793604850769043
    }
    normalized_vertices {
      x: 0.9742650985717773
      y: 0.6732610464096069
    }
  }
  score: 0.9870228171348572
}
display_name: "truck"
, annotation_spec_id: "2041373629094035456"
image_object_detection {
  bounding_box {
    normalized_vertices {
      x: 0.005605409853160381
      y: 0.5414829254150391
    }
    normalized_vertices {
      x: 0.1891111582517624
      y: 0.7010807991027832
    }
  }
  score: 0.9359530210494995
}
display_name: "excavator"
]

Andrew

[kolban-google]

Looking here https://cloud.google.com/vision/automl/docs/samples/automl-vision-classification-predict#automl_vision_classification_predict-python

We seem to see an example of making a prediction request and seeing the response from Python. The documentation for the return from the predict API call seems to be here:

https://googleapis.dev/python/automl/latest/automl_v1/types.html#google.cloud.automl_v1.types.PredictResponse