search

ARISE-Initiative / robosuite

robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
https://robosuite.ai
Other
1.19k stars 384 forks source link

Integrating UR5 + real dataset #4

Closed ahundt closed 5 years ago

ahundt commented 5 years ago

This project looks very interesting! I recently released a real robot dataset with a UR5 and a block stacking task.

Do you think it might be feasible & worthwhile to integrate the following?

  1. A simulated UR5 + the taller block stacking task
  2. loading the real costar block stacking dataset, including real train/test/val splits

Side note: Have you also seen the MIME dataset?

I'd very much like to hear your thoughts, so thanks for your time and consideration!

yukezhu commented 5 years ago

Hi @ahundt thanks for your interest! We are definitely planning to expand our benchmark with more robot models (including UR5) and more tasks. But our priority right now is to support the robots used in our lab (http://pair.stanford.edu). Both CoSTAR and MIME datasets look very interesting. Feel free to contribute to this repo through PRs if you'd like to create a simulated mirror of the real CoSTAR block stacking task in robosuite.

ahundt commented 5 years ago

We've got something similar in costar_plan, particularly the costar_task_plan part, but I'm expecting to move from keras to pytorch and many aspects of robosuite look quite nice:

  1. clean implementation of reinforcement learning algorithms
  2. integrated distributed training
  3. integrated human controller
  4. multi-modal observation spec and action spec input (images, depth, joint information, gripper information etc)

If it lines up well I would like to contribute, but for the moment I'm evaluating the feasibility of participating, so I've got a few questions. As someone who hopes to contribute, a few things in the current docs & contributing instructions aren't yet easy to find or follow.

Could instructions for the following questions be added in a simple step-by-step copy/paste command-line commands sort of way? I have a single workstation that has two GPUs and a robot attached. I don't have any kubernetes clouds available to me.

  1. I see I can run the existing surreal reinforcement learning algorithms on these environments, but how do I view progress and reproduce the plots from the surreal paper?
  2. How do I run a trained policy in a simulated environment?
  3. How do I archive & reload pretrained weights?
  4. Where can I find the real robot interfaces I see in the roboturk faq?
  5. What is the API for real data, how can I add an interface for my own real dataset?
  6. How do I run human collection with a real robot?
  7. How do I run a policy with a real robot?
  8. How do I add a new simulated robot (I'd eventually plan to add my ur5 + robotiq gripper)?
  9. How do I add a new real robot (I'd eventually plan to add my ur5 + robotiq gripper)?
  10. For a dataset would I create a wrapper?

With step-by-step instructions I can learn about the design & requirements and then follow the concrete step by step process to integrate my new components very deeply.

I think a few of these questions might already have answers since I've read most of the readmes here and in the surreal repository. However, the information is split amongst many repositories and readme files. It is thus hard to determine the correct "human repository contributor" start position and the order to follow along these tutorials so that I might tie the pieces together.

amandlek commented 5 years ago

Hi @ahundt, thanks for your questions and interest.

(1) I think this question would be better suited for the Surreal repository.

(2) As long as your policy has a mechanism for picking actions, you can use a loop similar to this example and replace the random action sampling line with your policy output.

(3) Same as (1).

(4) Currently, RoboTurk is not making data collection facilities public. We are still in the process of further platform development.

(5-7) We do not yet have an interface for real data, but this is an extension that is on our roadmap.

(8) We have instructions on creating an environment here. You can refer to how we implemented the Sawyer and Baxter interfaces, and add your own robot similarly.

(9) Same as (5-7).

(10) Not too sure what you mean by creating a wrapper for the dataset. The wrappers are meant to provide additional functionality that are not provided by default in the environments.