[Headless] Cannot get camera tensor

[dibbla]

Hi!

I was using Isaac on a 3090 server (it works fine), and I basically setup the cameras in this way:

camera_properties = gymapi.CameraProperties()

camera_properties.width = 300
camera_properties.height = 240
camera_properties.enable_tensors = True
for i in range(len(self.envs)):
    cam_handle = self.gym.create_camera_sensor(self.envs[i], camera_properties)
    camera_offset = gymapi.Vec3(*self.cfg.visual_input.cam_pos)
    camera_rotation = gymapi.Quat.from_axis_angle(gymapi.Vec3(0, 1, 0), np.deg2rad(self.CAMERA_ANGLE))
    actor_handle = self.actor_handles[i]
    body_handle = self.gym.get_actor_rigid_body_handle(self.envs[i], actor_handle, 0)

    self.gym.attach_camera_to_body(
        cam_handle, self.envs[i], body_handle, gymapi.Transform(camera_offset, camera_rotation),
        gymapi.FOLLOW_TRANSFORM
    )
    self.camera_handles.append(cam_handle)

But both servers with A5000 CUDA11.7 & CUDA12 report error:

[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
[Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
[Error] [carb.windowing-glfw.plugin] GLFW window creation failed!
[Error] [carb.gym.plugin] Failed to create Window in CreateGymViewerInternal
[Error] [carb.gym.plugin] Gym cuda error: invalid device ordinal: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 991

Given I'm working headlessly, it's kind of crucial to get the image and videos for debugging. Any ideas?

[dibbla]

On the CUDA12 machine, this is what I get:

[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 0
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 1
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 2

Seems like I am getting some -1 as camera handle.

[dibbla]

Given I managed to find a local machine, the issue will be closed. You may reopen if were there any question related to this issue.

[youwyu]

On the CUDA12 machine, this is what I get:

[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 0
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 1
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 2

Seems like I am getting some -1 as camera handle.

Still have the same issue with CUDA 12.2

[HomerRong]

On the CUDA12 machine, this is what I get:

[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 0
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 1
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 2

Seems like I am getting some -1 as camera handle.

        self.graphics_device_id = self.sim_device_id
        # if self.headless == True:
        #     self.graphics_device_id = -1

I fix this problem with commenting above two lines to let graphics_device_id do not be set -1 in headless mode.

[HomerRong]

and always need gym.fetch_results

# if self.device == 'cpu':
self.gym.fetch_results(self.sim, True)