pose keypoints

[soans1994]

Hello author,

i would like to know how to save pose keypoints location of hand, like image shown below. What are the handpose angles in the output json file? can you please explain.

thank you

[kaiidams]

Hello, it stores the joint angles in degree instead of 2D locations, which directly come from Blender's bone pose modifier. 0 means unmodified, i.e. straight, positive bended inwards, negative bended outwards. To get 2D location from Blender, something similar to get_bounding_box() in render_batch.py will be needed.

[soans1994]

Thank you for your advice, I will try to implement function like get_bounding_box(). Can i access the keypoints location from the existing hand model? or should i annotate the points inisde blender. Please give me some suggestions.

[kaiidams]

I think you need to use Blender's Python API to get key point locations and write them out and regenerate images.

[soans1994]

thank you, please let me know if you update your project.

[soans1994]

I have some doubts regarding the annotation tools in Blender. Should i perform manual annotation on the 3d objects or can i extract the points from the 3d mesh. Sorry since i am new to blender. Can you give me some suggestions if i need to achieve the hand image as shown above. Please tell me the important functions i need for annotation in blender, I will study those. Thank you

[kaiidams]

Please take a look at get_bounding_box(image_width, image_height) in renader_batch.py. It calculates the joints positions in the image in pixels and get the bounding box of the hand.

You can get the local position of bone tail like this.

>>> ob = bpy.data.objects['Hand']
>>> bone = ob.pose.bones['finger1.R']
>>> bone.tail
Vector((-0.07351075857877731, 0.0024329321458935738, 0.05441868305206299))

then, you can convert it to the view space by applying the inverted transform matrix of camera.

ob = bpy.data.objects['Camera']
mat = ob.matrix_world.normalized().inverted()
get_render_pos(mat, bone.tail)

[soans1994]

@kaiidams Thank you for your detailed explanation. I have tried this new function. also added

Translate to the image coordinate.

min_x = round((min_vx + 0.5) * image_width)
min_y = round((min_vy + 0.5) * image_height)

but the coordinates doesnt seem to match with the saved image. I have some other questions:

1. the images generated in the data folder is pink color. why is that?
2. also seems like the bounding box values in the json file are not matching the image coordinates(coordinates doesnt seem to match with the saved image). Am i missing something?

{"file_name": "00-001.png", "pose": [52.74349791847058, 18.603710865657426, 9.349725461966795, 7.850194668891849, 12.699051836506584, 38.162997497736534, 40.0, 12.625212113675754, 0.0, 0.0], "bbox": [82, 45, 77, 143], "keypoint": [59, 126]}

[kaiidams]

@soans1994 Thank you for the report. I made a new issue #2. Let me check the coordinates issue here.

[kaiidams]

@soans1994 Can you try git pull from master? It should fix the purple image issue.

[kaiidams]

@soans1994 Let me show you a sample code to draw bbox on the image. The origin of the bbox is the left-bottom corner. Hope this helps you.

from PIL import ImageDraw
from PIL import Image
import json

block = 0
index = 3
image_width = 224
image_height = 224

image = Image.open('./data/images/%02d/%02d-%03d.png' % (block, block, index))
anno = []
with open('data/annotations/%02d.json' % block) as f:
    for line in f:
        anno.append(json.loads(line))
draw = ImageDraw.Draw(image)
bbox =  anno[index]['bbox']
shape = [(bbox[0], image_height - bbox[1] - bbox[3]), (bbox[0] + bbox[2], image_height - bbox[1])] 
draw.rectangle(shape, outline ="red") 
image 

[soans1994]

@kaiidams Thank you very much. New update solved the problems of pink texture.Also the bounding box visualization code works fine. edit: sorry for trouble, i had some confusion regarding the bbox. But, now it is cleared.

Regards