search

isl-org / spear

SPEAR: A Simulator for Photorealistic Embodied AI Research
MIT License
221 stars 15 forks source link

Running `generate_images.py` with `normal` render pass throws error #233

Closed RachithP closed 1 year ago

RachithP commented 1 year ago 
(spear) C:\github\spear\examples\generate_image_dataset>python generate_images.py
[SPEAR | env.py] Launching Unreal instance...
[SPEAR | env.py] Writing temp config file: C:\github\spear\examples\generate_image_dataset\tmp\config.yaml
[SPEAR | env.py] Launching executable with the following command-line arguments:
C:/github/spear/cpp/unreal_projects/SpearSim/Standalone-Shipping/Windows/SpearSim/Binaries/Win64/SpearSim-Win64-Shipping-Cmd.exe -windowed -resx=1024 -resy=1024 -graphicsadapter=0 -nosound -fileopenlog -stdout -fullstdoutlogoutput -nologtimes -renderoffscreen -config_file=C:\github\spear\examples\generate_image_dataset\tmp\config.yaml
[SPEAR | env.py] Launching executable with the following config values:
CORE_UTILS:
  WAIT_FOR_KEYBOARD_INPUT_DURING_INITIALIZATION: False
SIMULATION_CONTROLLER:
  AGENT: CameraAgent
  CAMERA_AGENT:
    ACTION_COMPONENTS: ['set_location', 'set_rotation']
    CAMERA:
      FOV: 90.0
      IMAGE_HEIGHT: 1024
      IMAGE_WIDTH: 1024
      RENDER_PASSES: ['normal']
    CAMERA_ACTOR_NAME: camera_actor
    OBSERVATION_COMPONENTS: ['camera']
    SPAWN_ACTOR_NAME:
    SPAWN_LOCATION_X: 0.0
    SPAWN_LOCATION_Y: 0.0
    SPAWN_LOCATION_Z: 200.0
    SPAWN_MODE: specify_pose
    SPAWN_ROTATION_PITCH: 0.0
    SPAWN_ROTATION_ROLL: 0.0
    SPAWN_ROTATION_YAW: 0.0
    STEP_INFO_COMPONENTS: []
  CAMERA_SENSOR:
    READ_SURFACE_DATA: True
    USE_SHARED_MEMORY: True
  CUSTOM_UNREAL_CONSOLE_COMMANDS: []
  IMITATION_LEARNING_TASK:
    AGENT_ACTOR_NAME:
    AGENT_SPAWN_OFFSET_LOCATION_X: 0.0
    AGENT_SPAWN_OFFSET_LOCATION_Y: 0.0
    AGENT_SPAWN_OFFSET_LOCATION_Z: 0.0
    EPISODES_FILE:
    GOAL_ACTOR_NAME:
    OBSTACLE_IGNORE_ACTOR_NAMES: []
  IMU_SENSOR:
    DEBUG_RENDER: False
  INTERACTION_MODE: programmatic
  IP: 127.0.0.1
  MAP_ID:
  NAVMESH:
    AGENT_HEIGHT: 200.0
    AGENT_MAX_SLOPE: 1.0
    AGENT_MAX_STEP_HEIGHT: 1.0
    AGENT_RADIUS: 50.0
    CELL_HEIGHT: 1.0
    CELL_SIZE: 1.0
    DEBUG_NAVIGATION_DATA_FILE:
    MAX_SIMPLIFICATION_ERROR: 1.3
    MERGE_REGION_SIZE: 400.0
    MIN_REGION_AREA: 0.0
    TILE_POOL_SIZE: 1024
    TILE_SIZE_UU: 1000.0
    TRAJECTORY_SAMPLING_DEBUG_RENDER: False
    TRAJECTORY_SAMPLING_SEARCH_RADIUS: 0.0
  PHYSICS:
    ENABLE_ENHANCED_DETERMINISM: True
    ENABLE_SUBSTEPPING: True
    MAX_SUBSTEPS: 100
    MAX_SUBSTEP_DELTA_TIME: 0.01
    SIMULATION_STEP_TIME: 0.05
  POINT_GOAL_NAV_TASK:
    AGENT_ACTOR_NAME:
    EPISODE_BEGIN:
      AGENT_LOCATION_X_MAX: nan
      AGENT_LOCATION_X_MIN: nan
      AGENT_LOCATION_Y_MAX: nan
      AGENT_LOCATION_Y_MIN: nan
      AGENT_LOCATION_Z: nan
      GOAL_LOCATION_X_MAX: nan
      GOAL_LOCATION_X_MIN: nan
      GOAL_LOCATION_Y_MAX: nan
      GOAL_LOCATION_Y_MIN: nan
      GOAL_LOCATION_Z: nan
      SPAWN_DISTANCE_THRESHOLD: 1.0
    GOAL_ACTOR_NAME:
    GOAL_MATERIAL: /Engine/BasicShapes/BasicShapeMaterial.BasicShapeMaterial
    GOAL_MESH: /Engine/BasicShapes/Cylinder.Cylinder
    OBSTACLE_IGNORE_ACTOR_NAMES: []
    RANDOM_SEED: 0
    REWARD:
      HIT_GOAL: 1.0
      HIT_OBSTACLE: -1.0
  PORT: 30000
  SCENE_ID: apartment_0000
  SPHERE_AGENT:
    ACTION_COMPONENTS: ['add_force', 'add_rotation']
    CAMERA:
      FOV: 90.0
      IMAGE_HEIGHT: 512
      IMAGE_WIDTH: 512
      RENDER_PASSES: ['final_color']
    CAMERA_ACTOR_NAME:
    OBSERVATION_COMPONENTS: ['camera', 'location', 'rotation']
    SPAWN_ACTOR_NAME:
    SPAWN_LOCATION_X: 0.0
    SPAWN_LOCATION_Y: 0.0
    SPAWN_LOCATION_Z: 0.0
    SPAWN_MODE: specify_pose
    SPAWN_ROTATION_PITCH: 0.0
    SPAWN_ROTATION_ROLL: 0.0
    SPAWN_ROTATION_YAW: 0.0
    SPHERE:
      ANGULAR_DAMPING: 0.0
      LINEAR_DAMPING: 0.0
      MATERIAL: /Engine/BasicShapes/BasicShapeMaterial.BasicShapeMaterial
      MAX_ANGULAR_VELOCITY: 1.0
      MESH_SCALE_X: 0.5
      MESH_SCALE_Y: 0.5
      MESH_SCALE_Z: 0.5
      STATIC_MESH: /Engine/BasicShapes/Sphere.Sphere
    SPHERE_ACTOR_NAME:
    STEP_INFO_COMPONENTS: ['debug']
  TASK: NullTask
  URDFBOT_AGENT:
    ACTION_COMPONENTS: ['control_joints']
    CAMERA:
      FOV: 90.0
      IMAGE_HEIGHT: 512
      IMAGE_WIDTH: 512
      POSITION_OFFSET_X: 0.0
      POSITION_OFFSET_Y: 0.0
      POSITION_OFFSET_Z: 0.0
      RENDER_PASSES: ['final_color']
    IS_READY_VELOCITY_THRESHOLD: 0.001
    OBSERVATION_COMPONENTS: ['link_state', 'camera']
    SPAWN_ACTOR_NAME:
    SPAWN_MODE: specify_pose
    SPAWN_PITCH: 0.0
    SPAWN_POSITION_X: 0.0
    SPAWN_POSITION_Y: 0.0
    SPAWN_POSITION_Z: 0.0
    SPAWN_ROLL: 0.0
    SPAWN_YAW: 0.0
    STEP_INFO_COMPONENTS: []
    URDFBOT_ACTOR_NAME:
  VEHICLE_AGENT:
    ACTION_COMPONENTS: ['set_brake_torques', 'set_drive_torques']
    CAMERA:
      FOV: 90.0
      IMAGE_HEIGHT: 512
      IMAGE_WIDTH: 512
      RENDER_PASSES: ['final_color']
    IS_READY_VELOCITY_THRESHOLD: 0.001
    OBSERVATION_COMPONENTS: ['camera', 'location', 'rotation', 'wheel_rotation_speeds']
    SPAWN_ACTOR_NAME:
    SPAWN_LOCATION_X: 0.0
    SPAWN_LOCATION_Y: 0.0
    SPAWN_LOCATION_Z: 0.0
    SPAWN_MODE: specify_pose
    SPAWN_ROTATION_PITCH: 0.0
    SPAWN_ROTATION_ROLL: 0.0
    SPAWN_ROTATION_YAW: 0.0
    STEP_INFO_COMPONENTS: ['']
    VEHICLE_ACTOR_NAME:
  VISUALIZER:
    ACTOR_NAME: camera_actor
    SPAWN_CAMERA: False
    SPAWN_LOCATION_X: 0.0
    SPAWN_LOCATION_Y: 0.0
    SPAWN_LOCATION_Z: 0.0
    SPAWN_ROTATION_PITCH: 0.0
    SPAWN_ROTATION_ROLL: 0.0
    SPAWN_ROTATION_YAW: 0.0
SPEAR:
  CUSTOM_COMMAND_LINE_ARGUMENTS: []
  GPU_ID: 0
  LAUNCH_MODE: standalone_executable
  MAX_NUM_TICKS_AFTER_RESET: 10
  NUM_EXTRA_WARMUP_TICKS: 20
  PAKS_DIR:
  RENDER_OFFSCREEN: True
  RPC_CLIENT_INITIALIZE_CONNECTION_MAX_TIME_SECONDS: 120.0
  RPC_CLIENT_INITIALIZE_CONNECTION_SLEEP_TIME_SECONDS: 1.0
  RPC_CLIENT_INTERNAL_RECONNECT_LIMIT: 1
  RPC_CLIENT_INTERNAL_TIMEOUT_SECONDS: 5.0
  STANDALONE_EXECUTABLE: C:/github/spear/cpp/unreal_projects/SpearSim/Standalone-Shipping/Windows/SpearSim/Binaries/Win64/SpearSim-Win64-Shipping-Cmd.exe
  TEMP_DIR: tmp
  UNREAL_EDITOR_EXECUTABLE:
  UNREAL_INTERNAL_LOG_FILE:
  UPROJECT:
  VULKAN_DEVICE_FILES:
  WINDOW_RESOLUTION_X: 1024
  WINDOW_RESOLUTION_Y: 1024
URDFBOT:
  URDFBOT_PAWN:
    CAMERA_COMPONENT:
      FOV: 90.0
      PITCH: 0.0
      POSITION_X: 0.0
      POSITION_Y: 0.0
      POSITION_Z: 50.0
      ROLL: 0.0
      YAW: 0.0
    KEYBOARD_ACTIONS:

    URDF_DIR: C:\github\spear\python\spear\urdf
    URDF_FILE: locobot.xml
  URDF_LINK_COMPONENT:
    POSITION_SOLVER_ITERATION_COUNT: 16
    VELOCITY_SOLVER_ITERATION_COUNT: 16
VEHICLE:
  VEHICLE_COMPONENT:
    CHASSIS_HEIGHT: 15.0
    CHASSIS_WIDTH: 15.0
    DRAG_COEFFICIENT: 1.0
    LEGACY_WHEEL_FRICTION_POSITION_ENABLED: False
    MASS: 11.2
    SUSPENSION_ENABLED: True
    WHEEL_FRICTION_ENABLED: True
  VEHICLE_PAWN:
    ANIM_INSTANCE: /Vehicle/OpenBot/Meshes/ABP_OpenBot.ABP_OpenBot_C
    CAMERA_COMPONENT:
      ASPECT_RATIO: 1.333333
      FOV: 70.0
      LOCATION_X: 9.0
      LOCATION_Y: 5.0
      LOCATION_Z: 10.0
      ROTATION_PITCH: 0.0
      ROTATION_ROLL: 0.0
      ROTATION_YAW: 0.0
    IMU_COMPONENT:
      LOCATION_X: 8.0
      LOCATION_Y: 0.0
      LOCATION_Z: 10.0
      ROTATION_PITCH: 0.0
      ROTATION_ROLL: 0.0
      ROTATION_YAW: 0.0
    SKELETAL_MESH: /Vehicle/OpenBot/Meshes/SK_OpenBot.SK_OpenBot
  VEHICLE_WHEEL:
    FRICTION_FORCE_MULTIPLIER: 3.0
    SPRING_PRELOAD: 0.033
    SPRING_RATE: 0.167
    SUSPENSION_DAMPING_RATIO: 0.5
    SUSPENSION_MAX_DROP: 0.001
    SUSPENSION_MAX_RAISE: 0.001
    WHEEL_MASS: 0.033
    WHEEL_RADIUS: 3.3
    WHEEL_WIDTH: 2.5
[SPEAR | env.py] Initializing RPC client...
[SPEAR | CoreUtils.cpp] CoreUtils::StartupModule
LogMemory: Platform Memory Stats for Windows
LogMemory: Process Physical Memory: 126.18 MB used, 130.93 MB peak
LogMemory: Process Virtual Memory: 113.34 MB used, 113.34 MB peak
LogMemory: Physical Memory: 17906.08 MB used,  14519.26 MB free, 32425.34 MB total
LogMemory: Virtual Memory: 51070.60 MB used,  6834.19 MB free, 57904.79 MB total
[SPEAR | ActorHitEventComponent.h] UActorHitEventComponent::UActorHitEventComponent
[SPEAR | TickEventComponent.h] UTickEventComponent::UTickEventComponent
[SPEAR | VehicleMovementComponent.cpp] UVehicleMovementComponent::UVehicleMovementComponent
[SPEAR | VehicleMovementComponent.cpp] UVehicleMovementComponent::UVehicleMovementComponent
[SPEAR | VehiclePawn.cpp] AVehiclePawn::AVehiclePawn
[SPEAR | VehicleWheel.cpp] UVehicleWheel::UVehicleWheel
[SPEAR | SpearSimGameEngine.cpp] USpearSimGameEngine::USpearSimGameEngine
[SPEAR | SpearSimGameMode.cpp] ASpearSimGameMode::ASpearSimGameMode
[SPEAR | SpearSimSpectatorPawn.cpp] ASpearSimSpectatorPawn::ASpearSimSpectatorPawn
[SPEAR | Vehicle.cpp] Vehicle::StartupModule
[SPEAR | SimulationController.cpp] SimulationController::StartupModule
[SPEAR | SpearSimGameEngine.cpp] USpearSimGameEngine::USpearSimGameEngine
[SPEAR | SimulationController.cpp] SimulationController::postWorldInitializationEventHandler
[SPEAR | SimulationController.cpp] SimulationController::worldCleanupEventHandler
[SPEAR | env.py] Finished initializing RPC client.
[SPEAR | env.py] Initializing Unreal instance, warming up for 21 ticks...
[SPEAR | SimulationController.cpp] SimulationController::postWorldInitializationEventHandler
[SPEAR | SimulationController.cpp] scene_id:                apartment_0000
[SPEAR | SimulationController.cpp] map_id:                  apartment_0000
[SPEAR | SimulationController.cpp] desired_world_path_name: /Game/Scenes/apartment_0000/Maps/apartment_0000.apartment_0000
[SPEAR | SimulationController.cpp] desired_level_name:      /Game/Scenes/apartment_0000/Maps/apartment_0000
[SPEAR | SimulationController.cpp] world->GetPathName():    /Game/Scenes/apartment_0000/Maps/apartment_0000.apartment_0000
[SPEAR | SimulationController.cpp] open_level:              0
[SPEAR | SpearSimGameMode.cpp] ASpearSimGameMode::ASpearSimGameMode
[SPEAR | SpearSimSpectatorPawn.cpp] ASpearSimSpectatorPawn::ASpearSimSpectatorPawn
[SPEAR | SimulationController.cpp] SimulationController::worldBeginPlayEventHandler
[SPEAR | env.py] Finished initializing Unreal instance.
Traceback (most recent call last):
  File "generate_images.py", line 146, in <module>
    plt.imsave(os.path.realpath(os.path.join(render_pass_dir, "%04d.png"%pose["index"])), obs_render_pass_vis)
  File "C:\Users\Rachith\miniconda3\envs\spear\lib\site-packages\matplotlib\pyplot.py", line 2128, in imsave
    return matplotlib.image.imsave(fname, arr, **kwargs)
  File "C:\Users\Rachith\miniconda3\envs\spear\lib\site-packages\matplotlib\image.py", line 1630, in imsave
    rgba = sm.to_rgba(arr, bytes=True)
  File "C:\Users\Rachith\miniconda3\envs\spear\lib\site-packages\matplotlib\cm.py", line 493, in to_rgba
    raise ValueError("Floating point image RGB values "
ValueError: Floating point image RGB values must be in the 0..1 range.
mikeroberts3000 commented 1 year ago

This seems like a problem in generate_images.py, because it is not guaranteed that the normal values returned by Unreal will be between 0.0 and 1.0, and spear.Env just forwards the Unreal values directly to the returned observation.

RachithP commented 1 year ago

Yup, looks like np.clip() was incorrectly used. Fixing it in #236.

RachithP commented 1 year ago

Fix in #236.