search

BYU-PCCL / holodeck

High Fidelity Simulator for Reinforcement Learning and Robotics Research.
https://holodeck.cs.byu.edu
MIT License
579 stars 42 forks source link

Question about camera parameters #420

Closed sethmnielsen closed 4 years ago

sethmnielsen commented 4 years ago

Hey guys,

I tried reaching out to you guys on the Discord, but I didn't get any response, so I thought I'd create an issue to ask my question.

Do you know what values the parameters of the RGBCamera sensor are set to? By "parameters", I mean specifically the focal lengths - fx (horizontal) and fy (vertical) - and the lens distortion matrix (if there is any lens distortion at all). I asked Nick Walton a while back this same question, and he said he was sure that the horizontal field of view is 90°. Do you know if that has been modified at all? I assume it hasn't. Knowing the field of view would also work, as it is only an equation to convert FOV to focal length using the pinhole camera model.

Is there a way to find out the vertical focal length fy? I assume the two focal lengths are probably equal. In that case, the vertical field of view would be dependent on the horizontal field of view setting and the user-defined image resolution CaptureWidth and CaptureHeight, i.e., vertical FOV < horizontal FOV if CaptureHeight < CaptureWidth, VFOV = HFOV if CaptureWidth = CaptureHeight (square image). Is there a way to find out if both focal lengths are set to the same value?

Last thing I need to know is if there is any simulated lens distortion for the RGBCamera sensor. My understanding is that Unreal Engine tries to simulate a real-world camera, so it would be good to know if that includes lens distortion in the image for extra realism. In a real-world scenario, knowing the distortion parameters (also called the lens correction parameters) is essential for robots/drones that use visual data for autonomous navigation. Usually, the parameters such as the focal lengths and the distortion parameters for the camera lens would be provided by the manufacturer.

It may be the case that the horizontal field of view is 90°, the focal lengths fx and fy are equal, and that there isn't any lens distortion applied to the images. If that is the case, great! I just need a solid confirmation. We are using Holodeck for a new class that Dr. Randy Beard is teaching for the first time this semester (EC EN 682R: "Vision Enabled Estimation and Control of Multirotor Aircraft"), so it's pretty important to know whether we have those camera parameters right or not.

Hope you can help us out!

sethmnielsen commented 4 years ago

Also, here is an image I found inside the UE4 documentation somewhere that shows some camera settings that would be good to know:

image

I think these settings were for a cinematic camera - possibly a "Cine Camera Actor". Is the RGBCamera a Cine Camera actor?

daniekpo commented 4 years ago

@sethmnielsen, we're glad you're using Holodeck. The RGBCamera sensor currently doesn't have depth. We haven't changed the horizontal field of view so it's still at 90 degrees. The CaptureWidth and CaptureHeight are both set at 256. You can find out the parameters in holodeck-engine/Source/Holodeck/Sensors/Public/HolodeckCamera.h in the holodeck-engine repo. I hope that was helpful. Let me know if you have any other questions.

MaxRobinsonTheGreat commented 4 years ago

As for lens distortion, we do not apply any and as far as I am aware Unreal does not add any lens distortion by default, so there should not be any.

sethmnielsen commented 4 years ago

@daniekpo Great, thanks! I hadn't seen the location where the FOV was set before. That works; I'll assume that the horizontal and vertical focal lengths are equal and go from there.

@MaxRobinsonTheGreat Perfect, that's what I needed to know. If there were distortion and we assumed none, it could have made debugging the computer vision stuff pretty difficult. Thanks!