Open hesic73 opened 2 weeks ago
Modifying collision shapes at runtime (after loading the objects and starting the gpu simulation) is not possible at the moment. Therefore they cannot be changed during an environment reset. You can however with GPU simulation generate a ton of parallel environments each with different scales for the actors and maybe that will be sufficient randomization. If that is not enough, you can turn on the reconfiguration frequency (set reconfiguration_freq >= 1), which essentially destroys all objects and allows you to recreate them but with new actor scales
With GPU sim you will need to iterate over each the actor._objs
list and modify the elements inside there, no API for batch modifying shape sizes/topology at the moment.
Is it possible to extend the SAPIEN API to support batched properties? This would be useful for domain randomization.
For example, consider the current sapien.wrapper.actor_builder.CollisionShapeRecord
:
@dataclass
class CollisionShapeRecord:
type: Literal[
"convex_mesh",
"multiple_convex_meshes",
"nonconvex_mesh",
"plane",
"box",
"capsule",
"sphere",
"cylinder",
]
# for mesh type
filename: str = ""
# mesh & box
scale: Tuple = (1, 1, 1)
# circle & capsule
radius: float = 1
# capsule
length: float = 1
material: Union[sapien.physx.PhysxMaterial, None] = None
pose: sapien.Pose = sapien.Pose()
density: float = 1000
patch_radius: float = 0
min_patch_radius: float = 0
is_trigger: bool = False
decomposition: str = "none"
decomposition_params: Union[Dict[str, Any], None] = None
And the current sapien.physx.PhysxMaterial
:
class PhysxMaterial:
dynamic_friction: float
restitution: float
static_friction: float
Would it be possible to modify PhysxMaterial
and CollisionShapeRecord
to support batched inputs using torch.Tensor
?
class PhysxMaterial:
dynamic_friction: float | torch.Tensor
restitution: float | torch.Tensor
static_friction: float | torch.Tensor
@dataclass
class CollisionShapeRecord:
type: Literal[
"convex_mesh",
"multiple_convex_meshes",
"nonconvex_mesh",
"plane",
"box",
"capsule",
"sphere",
"cylinder",
]
# for mesh type
filename: str = ""
# mesh & box
scale: Tuple | torch.Tensor = (1, 1, 1)
# circle & capsule
radius: float | torch.Tensor = 1
# capsule
length: float | torch.Tensor = 1
material: Union[sapien.physx.PhysxMaterial, None] = None
pose: sapien.Pose = sapien.Pose()
density: float | torch.Tensor = 1000
patch_radius: float = 0
min_patch_radius: float = 0
is_trigger: bool = False
decomposition: str = "none"
decomposition_params: Union[Dict[str, Any], None] = None
Additionally, could mani_skill.utils.building.actor_builder.ActorBuilder
be updated so that build_entity
can index these batched properties and create entities with different parameters across subscenes?
I see the use case and it would be really useful to do more batch randomizations (textures can also be given the same treatment, currently the way it is done is not as convenient).
What would you prefer most as an API for this? When building a task (I imagine the randomization must occur during the _load_scene function).
We can change the API in maniskill to support more batched like convenience operations to avoid users writing for loops and simply supplying a batched array which may be easier to understand/read.
SAPIEN's API will very likely remain the same without batched APIs. Between maintainers this is what we decided was best (SAPIEN is the most low-level of the python APIs for very intricate control over the simulation setup and ManiSkill is prebuilt / simplified APIs for fast iteration and ML workflows)
Thank you! Below is my idea on how to change the API:
ActorBuilder
InheritanceLet mani_skill.utils.building.actor_builder.ActorBuilder
not inherit from sapien.wrapper.actor_builder.ActorBuilder
. Instead, make the SAPIEN builder a member variable within ManiSkill's ActorBuilder
:
class ActorBuilder:
def __init__(self, ...):
self._sapien_builder = sapien.ActorBuilder(...)
Avoid using sapien.physx.PhysxMaterial
and sapien.render.RenderMaterial
directly. Instead, create batched versions of these materials to handle varying properties across subscenes:
class PhysxMaterial:
dynamic_friction: torch.Tensor
restitution: torch.Tensor
static_friction: torch.Tensor
Implement methods like add_box_collision
to handle batched properties:
def add_box_collision(
self,
pose: Pose = Pose(), # ManiSkill pose
half_size: Union[Vec3, torch.Tensor] = (1, 1, 1),
material: Union[PhysxMaterial, sapien.physx.PhysxMaterial, None] = None,
density: float = 1000,
patch_radius: float = 0,
min_patch_radius: float = 0,
is_trigger: bool = False,
):
if material is None:
material = sapien.physx.get_default_material()
if isinstance(material, sapien.physx.PhysxMaterial):
# Convert to batched version if necessary
# Use placeholders for pose, material, or other parameters as needed
# Save the information somewhere for later use
self._sapien_builder.add_box_collision(...)
return self
During the build phase, inject the batched properties into the SAPIEN builder:
for i, scene_idx in enumerate(self.scene_idxs):
if self.scene.parallel_in_single_scene:
sub_scene = self.scene.sub_scenes[0]
else:
sub_scene = self.scene.sub_scenes[scene_idx]
# Inject the properties for the i-th subscene into the SAPIEN builder
# Modify the placeholders with the i-th batched property values
# This may involve setting the material, pose, scale, etc., for each collision shape
entity = self._sapien_builder.build_entity()
# Prepend scene index to entity name to indicate which subscene it belongs to
entity.name = f"scene-{scene_idx}_{self.name}"
# Set the pose before adding to the scene
entity.pose = to_sapien_pose(initial_pose_np[i])
sub_scene.add_entity(entity)
entities.append(entity)
This way we maintain compatibility with existing APIs.
I am looking to adjust the scales of actors when an environment is reset. I have identified where the properties for entities in subscenes are specified, which is in
sapien.wrapper.actor_builder.ActorBuilder.build_physx_component
:My question is: How can I access these
PhysxCollisionShape
instances and modify their properties, such as scale, during the simulation? Thank you very much!