search

danfergo / yarok

Yet another robot framework
5 stars 4 forks source link

Adaption for digit sensor #1

Open Jcastanyo opened 1 year ago

Jcastanyo commented 1 year ago

Hi!

My name is Julio, first of all congrats for this amazing repository and your work with tactile sensors and RL!

Yesterday, I saw your last work with Digit sensors and I am really interested in the sim2real approach. I wonder if you will think about updating the repository with the adaption for the Digit sensor, I would love to try it!

Thanks in advance and keep the good work!

danfergo commented 1 year ago

Hi Julio, I contacted the remaining authors of the paper, and I'll try to get the DIGIT sensor here as well. I'll let you know how it goes.

Jcastanyo commented 1 year ago

Hi, Ok, thank you very much, please let me know.

danfergo commented 1 year ago

Hi Julio, I've asked the other authors of the paper you mentioned, and they have uploaded their DIGIT code to GitHub https://github.com/Rancho-zhao/Digit_PhongSim

Meanwhile I've ported DIGIT to this repo/yarok as well. If you clone the repo, you should be able to see the demo world using

python3 -m yarok.comm.worlds.digit_world

I'm still adjusting some of the motors for the UR5 & the robotiq gripper, that you may find in the other examples, I'll drop a message here once I have that sorted out.

Jcastanyo commented 1 year ago

Hi,

I can run the simulation without errors, thank you very much for your work. I am comparing the digit image from this repo: https://github.com/Rancho-zhao/Digit_PhongSim and the digit image produced by your repo. I don't know why but the digit image in yarok repo is darker than in the other repo. I attach the images so you can see the difference to fix it.

Image from Digit_PhongSIm repo:

digit_bg_julio

image from yarok repo:

camera_screenshot_13 12 2022_2

Thanks for your help, I'm looking forward to hearing from you.

danfergo commented 1 year ago

Hi Julio, I've noticed this issue when porting the digit package to yarok, and I tried to fix it by adjusting the parameterization of the simulation model, you can also experiment with it a little bit.

https://github.com/danfergo/yarok/blob/master/yarok/comm/components/digit/digit.py

The difference comes down to the depth map read from mujoco. By default, MuJoCo emits a depth map that is normalized in terms of the camera frustum i.e. 1.0 when max depth/max visibility of the camera, 0 min depth/min visibility of the camera. In yarok I've coded a wrapper class for interfacing with MuJoCo and simplifying a few things. Including, by default it converts the depth map from that normalized z-buffer range into meters.

I thinks Digit_PhongSIm uses the raw z-buffer format, and so they have parameterized the sim model with some values that make sense in that scale i.e. elastomer_thickness 1.0, min_depth: 0.0;

Anyway, I've implemented an option on the read_camera depth parameter ("raw") of the InterfaceMJC (the MuJoCo wrapper) to allow the return of the "raw"/z-buffer depth map, and reparameterized the simulation model with the values that Digit_PhongSIm uses. So, the simulations should now be identical now.

Best, Daniel