search

google-deepmind / lab

A customisable 3D platform for agent-based AI research
Other
7.06k stars 1.36k forks source link

Access camera params (extrinsics, intrinsics, near/far plane) for pointcloud calculation #232

Open stepjam opened 2 years ago

stepjam commented 2 years ago

Hello, Is there a way to get either access to a point-cloud observation or access to the camera params (extrinsics, intrinsics, near/far plane) such that I can calculate the point-cloud from z-buffer observation? Thanks!

charlesbeattie commented 2 years ago

You can get information about the camera position with game:playerInfo. I believe the default vertical field of view is 90. You can calculate the horizontal FOV using the aspect ratio. You can use ray casts from the players eye position to generate a point cloud. See game:raycast.

Player look direction and posion:

local game = require 'dmlab.system.game'
local info = game:playerInfo()
local look = info.angles
local eye = info.eyePos

Raycast see: https://github.com/deepmind/lab/blob/2f21c07371e0f2a2c8e6a039b330c44b97d418ff/game_scripts/levels/tests/raycast_test.lua#L74