[Proposal] Checklist of environments to add to the framework

Mayankm96 commented 1 month ago

Proposal

This issue is a tracking page for our current plan to add environments to the Isaac Lab framework.

Motivation

The initial Orbit work mentioned a suite of rigid and deformable body simulation environments. However, many of these environments have been held back due to breaking changes in the framework APIs, poor simulation stability and scaling (particularly for deformable bodies and particle-based cloths), and a lack of resources to transfer all the implementations. Since this is a repeated topic of discussion on GitHub, I am creating this issue to keep track of the environments and provide a (very rough) outline/roadmap.

We are actively working on adding new features and improvements. It is hard to dedicate time to everything at once, but slowly, we are adding new environments as well. If you're interested in contributing, feel free to contact us!

Checklist

Completed ✅

For a complete list, please check here: https://isaac-sim.github.io/IsaacLab/source/features/environments.html

[x] Reach - Tracks a pose command for an end effector
[x] Lift - Lifts the object to the desired position
[x] In-Hand Repose - Re-orient a cube with a dexterous hand
[x] Cabinet - Open or close a cabinet (hinged object)
[x] Quadrupedal Locomotion - Track velocity command over a variety of terrain with a legged robot
[x] Quadrupedal Navigation - Track desired position command with a legged robot

In Progress 🏗️

[ ] Nut-bolt - Tighten the Nut on the specified Bolt
[ ] Soft Lifting - Lift a soft object to the target position: https://github.com/isaac-sim/IsaacLab/pull/842
[ ] Throwing and catching - Throw and catch a ball with two floating dexterous hands
[ ] Soft Stacking - Stacking soft objects together

Help Needed 🆘

[ ] Beat the Buzz - Move a key around a pole without touching it
[ ] Pyramid Stack - Build blocks into a pyramid
[ ] Hockey - Shoot the puck into the net with the stick
[ ] Peg In Hole - Insert the block into the hole
[ ] Jenga - Take out blocks and stack them into towers
[ ] Soft Placing - Placing soft objects on a shelf
[ ] Soft Dropping - Dropping soft objects into a container
[ ] Tower of Hanoi - Stacks of soft objects around pillars
[ ] Rope Reshaping - Reshape the rope on the table

Stand-by 🛑

[ ] Cloth Lifting - Lift the cloth to the desired position
[ ] Cloth Folding - Fold the cloth into the desired state
[ ] Cloth Spreading - Spread cloth across the table
[ ] Cloth Dropping - Place the cloth in the container
[ ] Flag Hoisting - Raise a flag on a table
[ ] Fluid Pouring - Pour the liquid into another container
[ ] Fluid Transport - Move a container full of liquid without causing any spills

hanschanhs commented 1 month ago

Thanks you for the great work - Wanted to ask about the cloth/tshirt prims in IsaacLab. What is the hold up/why is it listed as standby? Separately, If I just want to spawn a tshirt, what is the best way to do this currently in Isaac Lab? Novice question. Thanks for this!

Mayankm96 commented 3 weeks ago

Hi @hanschanhs

We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available.

If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

robegi commented 2 weeks ago

Good morning, and thank you for the update! If I may ask, I would like to know something more about the issues with the particle based liquid tasks. I am currently working on a liquid pouring task, and until now I managed to spawn and set up the fluid correctly, but I am a bit worried about the "poor simulation stability and scaling" you mentioned above. Is it possible to train an agent using particle based fluids? Thanks in advance!

allenzren commented 1 week ago

Hi @hanschanhs

We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available.

If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

Hi @Mayankm96, do you think then it is possible to simulate manipulating a cloth with a Franka arm right now? or the gripper and cloth interaction issue is not possible to solve with the current solver? Thanks!

I am happy to spend time setting this up and tuning the parameters, but not sure if cloth manipulation is fundamentally possible right now.

wzx16 commented 1 week ago

Hi @hanschanhs We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available. If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

Hi @Mayankm96, do you think then it is possible to simulate manipulating a cloth with a Franka arm right now? or the gripper and cloth interaction issue is not possible to solve with the current solver? Thanks!

I am happy to spend time setting this up and tuning the parameters, but not sure if cloth manipulation is fundamentally possible right now.

I have the same question. In addition, I'd like to know what would be the recommended way to spawn a rope prim. Thank you!

isaac-sim / IsaacLab