Chinese calligraphy using ManiSkill robotic arm

[Adaickalavan]

We would like to train a diffusion model to control a ManiSkill robotic arm to pick up a brush and draw Chinese characters.

To train the diffusion model, we would like to first capture some training data. The training data would be the robotic arm's trajectory while tracing given sample Chinese characters.

Any help on getting started with the simulation of robotic arm drawing Chinese characters in ManiSkill environment and collection of the training data (i.e., capture of robotic arm's trajectory while tracing Chinese characters) is much appreciated.

[StoneT2000]

Sounds pretty cool!. If you plan to train on simulated image data, this might be a bit complicated since you would need to draw out the actual stroke by the robot onto a surface, which probably wouldn't look good unless you implement an actual runtime photoshop-style like brush stroke as Chinese calligraphy is much more precise and intricate than english letters.

If you train on just robot proipioceptive / environment state then maybe this is fine. I might need more clarifications about what you want to do with the simulation. Do you plan on collecting teleop data / demos outside of simulation / inside sim?

[StoneT2000]

Ok I stand corrected, this is closer to possible than impossible, but I could only get something working for simple line drawings. I am considering making this some kind of an official environment for benchmarking.

https://github.com/user-attachments/assets/e3be7608-4421-4cf4-96dd-c978182d43a1

https://github.com/user-attachments/assets/2296d3a5-f77f-45ed-8fdf-f267f0ea64e0

[Adaickalavan]

Hi @StoneT2000,

For a start, we would like to merely train a robotic arm to write (i.e., Chinese characters) within a simulated environment. Initially, the robotic arm motion could simply be hard-coded, later we would use a model to control the arm for writing within a simulated environment.

In short, your demo video is exactly what we would like to achieve as a first step. It is a great piece of work. Could you share your code, which would be of great help to us?

Creating an official environment for writing would be a great addition for benchmarking. I believe there will be significant interest on such writing benchmark environment.

[StoneT2000]

Awesome to see this line drawing sim may suit your needs. I will work on creating a new set base env / tools around "drawing" in ManiSkill. It will likely be a base drawing environment class that you can inherit/copy and modify yourself (e.g. the max amount of ink your robot gets to use, ink color etc.)

[StoneT2000]

Merged the code in. The environment is in mani_skill/envs/tasks/drawing/draw.py

If you want to collect data / hardcode, you can use our teleoperation tool (requires a display) which just lets you click + drag / use keyboard to move robot up down left right etc.

python -m mani_skill.examples.teleoperation.interactive_panda -e TableTopFreeDraw-v1 -r panda_stick

(press h for keybindings and pass in -h to see help menu)