Relative position between hand/object during Grasping Generation

[dulucas]

Hi,

I used the code thermal_grasp/dataset_tools/collect_surface_grasps.py to generate grasps given an object and a hand model. And I found that if I use the HumanHand model, with the pre-defined variable pTa (I guess this is for initializing the hand position, right?), the HumanHand model is too far from the object and can not even manage to grasp the object. I changed the values in pTa (typically the values for translation) and manage to initialize the hand model near to the object model. Is this normal? Or it is because I forgot to set some parameters before?

Thanks!

[samarth-robo]

Hi @dulucas,

Each hand has a predefined "approach vector" which you can see if you load it in GraspIt!. pTa is the transformation matrix from the coordinate system of the palm mesh to the coordinate system of this approach vector. This makes me think it might be a problem with the hand meshes. I remember modifying GraspIt's default HumanHand model for this project, I will dig up the data and share it in a day or two.

You could also try changing the N_dists and dist_step parameters, they control how far back in the direction of the surface normal the hand starts from the object point.

[dulucas]

Great! Thx! Hope to hear from you :)

PS: In my code, I just changed the 'pTa' from [-0.13, 0.015, 0] to [0.13/100, 0.015, 0] and it works fine. Maybe its the unit problem? I'm not sure..

[samarth-robo]

@dulucas I have uploaded the data and and added instructions to README. See the third point in Setup: https://github.com/contactgrasp/thermal_grasp/blob/master/README.md#setup.

I think the issue is a mismatch between the HumanHand model in GraspIt! and dart. The dart model was included in the original zip file, but I had forgotten to upload the GraspIt! data. Can you give it a try and see if it fixes the issue?

[dulucas]

Hi, I tried with the model that you gave, but it still has the same problem.

[dulucas]

Besides, I built your code with ROS melodic + Ubuntu18.04 + python2, and it works just fine.

[samarth-robo]

@dulucas did you see any difference at all with the new hand model? In other words, are you sure the new model was actually being used?

[dulucas]

@samarth-robo Yes, with the new model, the hand becomes extremely small compared with the object model(like ~1000 times smaller I guess).

[dulucas]

I test by opening directly the world model using Graspit

[dulucas]

It becomes normal(visually correct) when I changed the 1000 of line 4 of wrl_convert.mlx to 1.

[samarth-robo]

@dulucas hmm this seems like a meter - millimeter scale issue.

What happens when you use the new models after fixing this scale issue? Do you still have to modify the pTa variable to get grasps?

[dulucas]

@samarth-robo Yes, I still have to modify the pTa, and now it becomes harder to find a valid one.

[dulucas]

@samarth-robo Instead of modifying the pTa, I divide the obj_pts by 1000 and then I re-run the code. By doing this, the hand model is much closer with the object model than before. I think it's indeed a scaling issue

[dulucas]

It seems like Graspit works fine(visually) by dividing the obj_pts by 1000 and with the pTa given by your code(at least with the banana model).

[dulucas]

@samarth-robo By the way, have you ever tried to run the Graspit without grasphical interface? Thx!

[samarth-robo]

GraspIt! cannot run without an X server. But, you can start an X server (separate from your main X server that serves the desktop) remotely via SSH using the following script:

#!/usr/bin/env bash

# trap logic from https://linuxconfig.org/how-to-propagate-a-signal-to-child-processes-from-a-bash-script
trap 'kill "${child_pid}"; wait "${child_pid}"' SIGINT SIGTERM

if [ $# -eq 2 ]; then
  echo "DISPLAY=$1 using GPU $2"
  Xorg -novtswitch -keeptty -noreset +extension GLX +extension RENDER -config unity_xorg$2.conf $1 &
elif [$# -eq 1 ]; then
  echo "DISPLAY=$1 using GPU 0"
  Xorg -novtswitch -keeptty -noreset +extension GLX +extension RENDER -config unity_xorg0.conf $1 &
elif [ $# -eq 0 ]; then
  echo "DISPLAY=:10 using GPU 0"
  Xorg -novtswitch -keeptty -noreset +extension GLX +extension RENDER -config unity_xorg0.conf :10 &
else
  echo "Usage: $0 [DISPLAY e.g. :11] [GPU index e.g. 1]"
  exit -1
fi

child_pid="$!"
wait "${child_pid}"

where unity_xorgN.conf is file placed in /etc/X11/:

Section "ServerFlags"
     Option "IgnoreABI" "True"
     Option "nolisten" "True"
     Option "AutoAddDevices" "False"
EndSection

Section "ServerLayout"
    Identifier "layout"
    Screen 0 "nvidiaN" 0 0
EndSection

Section "Device"
   Identifier "nvidiaN"
    Driver "nvidia"
    BusId "PCI:75:0:0"
    Option "ConstantFrameRateHint" "True"
EndSection

Section "Screen"
    Identifier "nvidiaN"
    Device "nvidiaN"
    Option "UseDisplayDevice" "none"
    SubSection "Display"
      Virtual 1920 1080
      Depth 24
    EndSubSection
EndSection

Note the BusID field, which is the PCI bus ID of GPU N. You can get that for your system by looking at the output of nvidia-smi and converting it from hex to decimal.

For example, $ ./start_new_x.sh :13 1 will start a new X server at port :13 which will use GPU 1.

Then in another terminal you can tell GraspIt! to use that X server: $ DISPLAY=:13 <whatever-program-calls-graspit>.

Note that your screen will go blank once you start a new X server, and keyboard and mouse will become unresponsive because that new X server is very basic.

To get back to your main X server which draws your screen and responds to keyboard and mouse, you should beforehand find out its virtual terminal number using sudo fgconsole (it will be a single integer e.g. 1). Then, when you want to switch from your new X server to your main X server, you can SSH and run sudo chvt <VT_number>. This will not kill the GraspIt! program running on the new X server.

[samarth-robo]

It seems like Graspit works fine(visually) by dividing the obj_pts by 1000 and with the pTa given by your code(at least with the banana model).

OK, thanks for the feedback! It seems like the only issue now is this 1000 scaling factor. GraspIt! uses the unit of meters (as expected), but the mesh read by Python in collect_surface_grasps.py use millimeters (unexpected). What do you use for the --models_dir command line argument to collect_surface_grasps.py? @dulucas

They are the same as the models used by DART. I updated the README.

[dulucas]

Hi @samarth-robo , thanks a lot for your help! The models I used are downloaded from here.

[dulucas]

Hi @samarth-robo , I switched to the models that you shared. But these models are not well aligned with the 3D contact prior heatmap(in particular, the 3D contact prior heatmap is always larger than the object model). Do you have some ideas? Thx!

[samarth-robo]

@dulucas was it another scaling issue? Did you find a solution? If yes, please let me know so that I can update the READMEs for future reference.

[samarth-robo]

Hi @samarth-robo , thanks a lot for your help! The models I used are downloaded from here.

Those ContactPose models are likely to have different scales / shifts. Please use the models mentioned in the README, link for reference.

[dulucas]

Hi, @samarth-robo I changed the models and modified something, then it works fine. Here is what I did:

1. Instead of using the .wrl files in your given .zip, I re-generated .wrl files using your script and the .ply models in .zip(I tried with the given .wrl, they didn't work well).
2. I changed the 'scale_factor' here to 1.15. Hope it helps. Thx!

[samarth-robo]

Hi, @samarth-robo I changed the models and modified something, then it works fine. Here is what I did:

1. Instead of using the .wrl files in your given .zip, I re-generated .wrl files using your script and the .ply models in .zip(I tried with the given .wrl, they didn't work well).
2. I changed the 'scale_factor' here to 1.15. Hope it helps. Thx!

Oh yes, that 1.15 scale factor! Thanks for pointing it out.