Manual Control of Robot Arm By Human/User

Vidharth commented 2 years ago

Is it possible or is there some way to control the robots manually in the panda gym environment in order to capture recordings of demonstrations.

By manually I mean, instead of an agent predicting actions at every step, is it possible or is there some way for a human/user to control the robot using keyboard mappings or something like that.

qgallouedec commented 2 years ago

Hi, @Vidharth

Yes, you can manually control the robot by passing your own action into the step() method. Here is a simple example:

import gym
import panda_gym

env = gym.make('PandaPickAndPlace-v1')
env.reset()

env.step([1, 0, 0, 0]) # Go FORWARD
env.step([-1, 0, 0, 0]) # Go BACKWARD
env.step([0, 1, 0, 0]) # Go LEFT
env.step([0, -1, 0, 0]) # Go RIGHT
env.step([0, 0, 1, 0]) # Go UP
env.step([0, 0, -1, 0]) # Go DOWN
env.step([0, 0, 0, 1]) # OPEN fingers
env.step([0, 0, 0, -1]) # CLOSE fingers

Another example, where action depends on the observation:

import gym
import panda_gym

env = gym.make('PandaPush-v1')
obs = env.reset()

# Get specific observations
ee_position = obs['observation'][0:3]
object_position = obs['observation'][7:10]

# Choose action according to object position and end-effector position
action = object_position - ee_position # move in the direction of the object

# Step
obs, reward, done, info = env.step(action)

Each envrionment has its own observation and action space. Read the following for more details.

You should be able to adapt this code to do keyboard-control, if that's what you want to do (using the keyboard module for example).

For `PandaReach-v1`:

Observation:

end-effector position (3 coordinates)
end-effector velocity (3 coordinates)

Action:

displacement (3 coordinates)

For `PandaPush-v1` and `PandaSlide-v1`:

Observation:

end-effector position (3 coordinates)
end-effector velocity (3 coordinates)
object position (3 coordinates)
object rotation (3 coordinates)
object linerar velocity (3 coordinates)
object angular velocity (3 coordinates)

Action:

displacement (3 coordinates)

For `PandaPickAndPlace-v1`:

Observation:

end-effector position (3 coordinates)
end-effector velocity (3 coordinates)
finger width (1 coordinate)
object position (3 coordinates)
object rotation (3 coordinates)
object linerar velocity (3 coordinates)
object angular velocity (3 coordinates)

Action:

displacement (3 coordinates)
displacement of fingers (1 coordinate)

For `PandaStack-v1`:

Observation:

end-effector position (3 coordinates)
end-effector velocity (3 coordinates)
finger width (1 coordinate)
object 1 position (3 coordinates)
object 1 rotation (3 coordinates)
object 1 linerar velocity (3 coordinates)
object 1 angular velocity (3 coordinates)
object 2 position (3 coordinates)
object 2 rotation (3 coordinates)
object 2 linerar velocity (3 coordinates)
object 2 angular velocity (3 coordinates)

Action:

displacement (3 coordinates)
displacement of fingers (1 coordinate)

EDIT: added finger width in PandaStack

cchristofi commented 1 year ago

Hey @qgallouedec,

If the position of the end-effector is positioned in the observation space at positions [0:3] and for the object at positions [7:10] for tasks PandaPush and PandaSlide, where are the ee-position, object position and finger width in PandaPickAndPlace and PandaStack tasks?

qgallouedec commented 1 year ago

You can infer it the same way from my previous comment:

	Task	Position
ee position	PandaPickAndPlace	[0:3]
	PandaStack	[0:3]
object position	PandaPickAndPlace	[0:3]
	PandaStack	[7:10] for 1, [19:22] for 2
finger width	PandaPickAndPlace	[6]
	PandaStack	[6]

qgallouedec / panda-gym