Object detection | Keypoints to object detection model

[ALEXKIRNAS]

Hi, @jch1 , @tombstone , @derekjchow , I would appreciate your help.

ADDING FUNCTIONALITY I recently start to use SSD with MobileNet feature extractor for solving face detection problem . In term of this task I also need to to detect landmarks (keypoints). I saw that repo contain some files that connected with keypoints ( link ), but I don`t see how to add them to model. So can you describe how to do that?

With the best regards, Alexander

[Sheepybloke2-0]

Any update on this? It seems that the visualizations there and the standard field is there, but I can't seem to get the keypoints from the graph (I'm using ssd_mobilenet_v1_coco). Currently failing when I'm calling keypoints = graph.get_tensor_by_name('detection_keypoints:0')

[wcastil]

@pkulzc I saw you had done most of the work on adding keypoints to the detection framework. Any chance you can give an update on if/when that functionality will be available?

[dougsouza]

As far as I've got in the code I saw that only MaskRCNNBoxPredictor supports keypoint prediction. How hard it would be to add keypoint prediction to a ConvolutionalBoxPredictor as well? Any thoughts?

[dustindorroh]

I had the same issue, but got reading and writing tfrecords to work. I have links to the source that helped me understand but, here is the tl,dr

Short Answer

1. When creating tfrecords write the keypoints to 'image/object/keypoint/x' and 'image/object/keypoint/y'
2. When reading tfrecords use 'groundtruth_keypoints' key
3. Trying to use higher level inputs.create_train_input_fn will not load keypoints because a few levels down when tf_example_decoder.TfExampleDecoder is called it isn't called with num_keypoints which defaults to 0 and thus never runs decoding.
4. Example of writing keypoints tfrecords . A little messy, but eh not sure if this topic is still alive.

More Detail:

1. We can see this happening in the decoding block https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/data_decoders/tf_example_decoder.py#L285-L293

2. The mapping specifically happens at: https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/data_decoders/tf_example_decoder.py#L290-L293

   Where fields.InputDataFields.groundtruth_keypoints is defined here: https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/core/standard_fields.py#L94

3. Trying to use the higher level dataset_builder.build doesn't pass a num_keypoints and this defaults to 0. https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/builders/dataset_builder.py#L124-L131

   https://github.com/tensorflow/models/blob/b9ef963d1e84da0bb9c0a6039457c39a699ea149/research/object_detection/data_decoders/tf_example_decoder.py#L136-L148

[pharrellyhy]

Hi, @dustindorroh. Very useful information and I'm curious that have you successfully trained keypoints regression with object detection API? Thanks.

[pharrellyhy]

@dustindorroh I can create tfrecord with keypoints as labels and set num_keypoints=1. During training, the keypoint labels are loaded but I don't think the computation graph contains the keypoint output nodes. I can't find where these nodes are added to the graph automatically, so maybe we need to add them manually?

[soldierofhell]

Hi, is anyone working on keypoint functionality now? IMO there's still ~50% of work to do, basicly adding keypoint head to faster rcnn meta architecture. I can do this, but don't want to waste time if the work is in progress or sheduled?

[bhack]

Keypoints API is useful also for new anchor free models: /cc @see-- https://github.com/see--/keras-centernet https://github.com/princeton-vl/CornerNet-Lite

[bhack]

A new anchor free model https://github.com/Star-Clouds/CenterFace We need to support bbox + keypoints API for anchorfree models.

[bhack]

See also https://github.com/tensorflow/hub/issues/424

[bhack]

I added a Keypoints/Heatmaps issue in Tensorflow addons at https://github.com/tensorflow/addons/issues/1366

[kensuke-ueda]

It seems that you should specify "KeypointBoxCoder" in pipeline.config to enable keypoint detection feature.

--------------------------------------- default pipeline.config box_coder { faster_rcnn_box_coder { y_scale: 10.0 x_scale: 10.0 height_scale: 5.0 width_scale: 5.0 } } --------------------------------------- you should change like this box_coder { keypoint_box_coder { y_scale: 10.0 x_scale: 10.0 height_scale: 5.0 width_scale: 5.0 } }

[vesor]

I have been extending the box_coder to add keypoints detection for some time. I have tested on both Faster RCNN and SSD with customized training data which has keypoints labeling. The problem is it is a bit confusing with the keypoints concept in Mask RCNN related code. In Mask RCNN, keypoints are separated from box coding.

[pharrellyhy]

Hi @vesor , nice work! I think what your are saying is totally reasonable. If you don't mind, can you give some sample code on how to achieve that? Thanks!

[hum16]

@pharrellyhy @vesor could you share some information on training key points detection using TF object detection API?

[pharrellyhy]

@gsadhasivam It's easy if you are using TF2 and CenterNet.

[salpert-humane]

@dustindorroh - Are the keypoints expected to be normalized as the bounding boxes? (normalization by image width and height).

[dustindorroh]

That sounds correct. It’s been a while, but I remember keeping the keypoints normed with respect to the bounding box. The keypoint bookkeeping is a pain otherwise.

From my own personal experience/mistakes I made, I had the bounding box coordinate transform backwards from the key point transform…. Or was it the other way… lol.

Best of luck! Dustin

On Tue, Jun 29, 2021 at 11:46 AM salpert-humane @.***> wrote:

@dustindorroh https://github.com/dustindorroh - Are the keypoints expected to be normalized as the bounding boxes? (normalization by image width and height).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tensorflow/models/issues/2366#issuecomment-870831331, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHWZ752D6XISX5I67BRJXTTVIIKBANCNFSM4D2NPFJA .

[aidansmyth95]

Hi all,

Any advice on what the best way is today to train an SSD model with keypoints? The boxcoder seems the most promising answer. Can't use CenterNet unfortunately for compute resource reasons.

[dustindorroh]

Hello,

The way I was able to accomplish this was to have two models. One SSD model for the bbox. The second model used the cropped part of the image to feed the landmarking model. I also built my landmarking model off of the SSD backbone.

Best of luck!

Dustin Dorroh

On Tue, Apr 18, 2023 at 6:29 PM aidansmyth95 @.***> wrote:

Hi all,

Any advice on what the best way is today to train an SSD model with keypoints? The boxcoder seems the most promising answer. Can't use CenterNet unfortunately for compute resource reasons.

— Reply to this email directly, view it on GitHub https://github.com/tensorflow/models/issues/2366#issuecomment-1514008190, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHWZ7322WQ3YVAXTMZY5Q3XB45YVANCNFSM4D2NPFJA . You are receiving this because you were mentioned.Message ID: @.***>