ThreeDWorld (TDW)
ThreeDWorld (TDW) is a platform for interactive multi-modal physical simulation. With TDW, users can simulate high-fidelity sensory data and physical interactions between mobile agents and objects in a wide variety of rich 3D environments.
TDW was developed primarily by Jeremy Schwartz and Esther Alter. It is now maintained by Esther Alter.
TDW is now in long-term support (LTS). TDW will receive only minor updates and bug fixes.
![](https://github.com/threedworld-mit/tdw/raw/master/splash.jpg)
General guide to TDW
Setup
1.1 Installation (Read this first!)
- Install TDW
- Upgrade TDW
1.2 How to run TDW on a Linux server
- Overview
- Launch a TDW build on a remote server from a personal computer
- Remote rendering with xpra
- X11 forwarding
Core Concepts
- The controller
- Auto-launching the TDW build
- Commands
- Design philosophy of TDW
- Scenes
- Avatars and cameras
- Add-ons and the
ThirdPersonCamera
- Objects
- Output data
- Images
- Outdoor scenes
Troubleshooting
- How to report an issue
- Common errors
- Performance optimizations
- Good coding practices
Tutorials
1. Scene Setup
Overview
1.1 Scene Setup (High-Level APIs)
- Overview
- Procedural generation (the
ProcGenKitchen
add-on)
- Regions, interior regions, and rooms
- Procedural object arrangements
- Scripted object placement (floorplan layouts)
- Reset a scene
High-level APs: Floorplan and ProcGenKitchen
1.2 Scene Setup (Low-Level APIs)
- Overview
- Units and data formats
Bounds
output data
- Visual materials, textures, and colors
- ProcGenRoom
2. 3D Models
2.1 Other model libraries
- Overview
- Free models
- Non-free models
2.2 Custom models
- Add your own models to TDW
- Add ShapeNet models to TDW
2.3 Composite (articulated) objects
- Overview
- Composite objects in TDW
- Create a composite object from a prefab
- Create a composite object from a .urdf file
3. Read/Write to Disk
- Overview
- The
Logger
and LogPlayback
add-ons
- The
JsonWriter
add-on
- The
OutputDataWriter
add-on
- Create a custom data writer
- Import .sdf and .lisdf files
- Images (Cross-referenced with "Core Concepts")
- Audio (Cross-referenced with "Audio")
- Video (Cross-referenced with "Video Recording")
4. Semantic States
- Overview
- Line of sight (
Raycast
output data)
- Proximity to region (
Overlap
output data)
- Proximity to other objects (the
TriggerCollisionManager
add-on)
- Containment (the
ContainerManager
add-on)
- Open and closed states
- Grasped objects
5. Visual Perception
- Overview
- Instance ID segmentation colors (
_id
pass)
- Semantic category segmentation colors (
_category
pass)
- Depth maps (
_depth
and _depth_simple
passes)
- Motion perception (
_flow
pass)
- Other image passes (
_mask
, _normals
, and _albedo
passes)
Occlusion
output data
6. Camera Controls
- Overview
- Move a camera
- Rotate a camera
- Follow an object
- The
CinematicCamera
add-on
7. Photorealism
- Overview
- Lighting (HDRI skyboxes)
- Post-processing
- Interior lighting (the
InteriorSceneLighting
add-on)
- Depth of field
- V-Ray Rendering
High-level API: tdw_image_dataset
8. Physics
Overview
8.1 Physics (PhysX)
- PhysX
- Object physics parameters
Rigidbodies
output data
Collision
output data
- Apply forces to objects
- Skip physics frames
- Disable physics
High-level API: tdw_physics
8.2 Physics (Obi)
- Obi
- Fluids
- Wind
- Cloth
ObiParticles
output data
- Colliders and collision materials
- Solvers
- Obi and robots
- Obi and Flex
8.3 Physics (Flex)
- Flex
- Solid and soft actors
- Cloth actors
- Fluid and source actors
- Move, rotate, and scale Flex objects
FlexParticles
output data
- Apply forces to Flex objects
- Reset a Flex scene
- Other Flex commands
High-level API: tdw_physics
9. Audio
- Overview
- Initialize audio and play .wav files
- Resonance Audio
- Recording audio
- Audio perception
9.1 Clatter (Physically-derived audio)
- Overview
- Object audio data
- Recording Clatter audio with the
PhysicsAudioRecorder
add-on
- Clatter and Resonance Audio
- Reset Clatter
- Manually generate audio
- Troubleshooting Clatter
- How to contribute to Clatter
9.2 PyImpact (obsolete predecessor to Clatter)
- PyImpact
- PyImpact advanced
- PyImpact and Clatter
10. Video Recording
- Overview
- Image-only video
- Video with audio
- Video with audio (Linux)
- Video with audio (OS X)
- Video with audio (Windows)
11. Agents
Overview
11.1 Robots
- Overview
- The
Robot
add-on
- Robot arm add-ons
- Robot collision detection
- Select a robot
- Add your own robots to TDW
- Robotics API (low-level)
- Add a camera to a robot
11.2 Magnebots
- Magnebot API (external repo)
11.3 Replicants
- Overview
- Actions
- Output data
- Collision detection
- Movement
- Animations
- Arm articulation, pt. 1: Basics
- Arm articulation, pt. 2: Grasp and drop objects
- Arm articulation, pt. 3: Advanced topics
- Arm articulation, pt. 4: Stacking objects
- Head rotation
- Navigation
- Custom actions
- Multiple Replicants
- Reset
11.4 Wheelchair Replicants
- Overview
- Actions
- Output data
- Collision detection
- Movement
- Arm articulation, pt. 1: Basics
- Arm articulation, pt. 2: Grasp and drop objects
- Arm articulation, pt. 3: Advanced topics
- Head rotation
- Navigation
- Custom actions
- Multiple Agents
- Reset
11.5 Drones
- Drones
11.6 Vehicles
- Vehicles
11.7 Virtual Reality (VR)
- Overview
- Oculus Touch
- Oculus Leap Motion
- FOVE Leap Motion
11.8 Keyboard and Mouse
- Overview
- Mouse input
- The
FirstPersonAvatar
- Keyboard input
11.9 Embodied avatars
- The
EmbodiedAvatar
12. Multi-agent simulations
- Overview
- Custom agent classes
13. Navigation
- Overview
- NavMesh pathfinding
- Occupancy maps
14. User Interface (UI)
- Overview
- The
UI
add-on
- UI Widgets
15. Non-physics objects
15.1 Non-physics humanoids
- Overview
- SMPL humanoids
- Create custom non-physics humanoids
- Create custom humanoid animations
15.2 Misc. non-physics objects
- Overview
- Position markers
- Line renderers
- Textured quads
- Compass rose
- Visual Effects
- The
FloorplanFlood
add-on
- Empty objects
16. Misc. other topics
- C# source code
- Freezing your code
- Download asset bundles
API Documentation
Command API
tdw
module API
tdw
tdw.add_ons
tdw.add_ons.ui_widgets
tdw.agent_data
tdw.asset_bundle_creator
tdw.backend
tdw.collision_data
tdw.container_data
tdw.drone
tdw.flex_data
tdw.lerp
tdw.librarian
tdw.lisdf_data
tdw.model_tests
tdw.obi_data
tdw.obi_data.cloth
tdw.obi_data.collision_materials
tdw.obi_data.fluids
tdw.object_data
tdw.object_data.composite_object
tdw.object_data.composite_object.sub_object
tdw.physics_audio
tdw.proc_gen.arrangements
tdw.proc_gen.arrangements.cabinetry
tdw.release
tdw.replicant
tdw.replicant.actions
tdw.replicant.ik_plans
tdw.robot_data
tdw.scene_data
tdw.vehicle
tdw.vray_data
tdw.vr_data
tdw.vr_data.fove
tdw.wheelchair_replicant
tdw.wheelchair_replicant.actions
Performance benchmarks
- Performance benchmarks
- Image capture
- Object data
- Command deserialization