search

zayfod / pycozmo

A pure-Python communication library, alternative SDK, and application for the Cozmo robot.
MIT License
178 stars 59 forks source link

Procedural face rendering #7

Closed zayfod closed 5 years ago

zayfod commented 5 years ago

Cozmo .bin FlatBuffers animation files describe Cozmo face images procedurally. Each face image is described with a structure in the following format: https://github.com/zayfod/pycozmo/blob/1fbf2775c2147356b39a031be7d92d74bb223a16/pycozmo/CozmoAnim/cozmo_anim.fbs#L11

The left and the right eye are described with 19 coefficients each.

Code that can interpret procedural face descriptions is needed. It should take pycozmo.CozmoAnim.ProceduralFace object as input and should generate a 128x32 PIL image as output.

The following presentation from Anki provides some background information: https://www.gdcvault.com/play/1024488/Cozmo-Animation-Pipeline-for-a

ppedro74 commented 5 years ago

Hi zayfod I've upload a project to generate a csv with all the procedural faces found: https://github.com/ppedro74/cozmo-playground/blob/master/Output/AllProcFaces.csv

here https://github.com/ppedro74/cozmo-playground/blob/master/FlatBuffersTest/Cozmo/ProceduralFace.cs

you can find ProceduralEyeParameter enum that conresponds to float array index.

zayfod commented 5 years ago

Hey, this looks nice! How did you figure out the meaning of the 19 coefficients?

ppedro74 commented 5 years ago

I did some reverse engineering on the Mobile app and I disassembled the shared library libcozmoEngine.so it was long ago, for example I recall the cv::ellipse2Poly is used to draw the eyes, i run out of time and I put the project on hold, until I found yours :)

ppedro74 commented 5 years ago

btw there are other faces like sad, surprised they don't exist as anim files (fb) I guess they are built in the shared library and rendered as needed via commands. Can you confirm this ? If there is an action to render faces using codes (constants) ?

CozmoCyke commented 5 years ago

Hello ppedro74,

I know that Anki doesn't always use .bin animation file for some Cozmo animated faces like the metronom animation. I also found that two animations are stored like a png file sequence (face_bored_event_02 and face_bored_event_04) ! We found two .bin files storing at 0x97 offset the png file sequence foldername in the .bin animation file (anim_bored_event_02.bin and anim_bored_event_04.bin respectively) Could you share the dissasembled libCozmoEngine.so or describe the way you dissassemble this file ? Great to find another great Cozmo fan like you :-)

zayfod commented 5 years ago

With the coefficient interpretation from @ppedro74 , Pillow.ImageDraw, and the new Client.display_image() method, everything is in place to start working on procedural face rendering.

I am planning to add a new pycozmo.procedural_face module to dev shortly.

zayfod commented 5 years ago

The latest commit on dev (5a62b146d2230741cda51153e4fc8c660ade65bd) finalizes the support for all 5+19+19=43 procedural face parameters.

There are now two examples:

The next step is to integrate procedural face rendering with animation playback.