Please support "rs2_set_region_of_interest" and "rs2_get_region_of_interest" API calls in the .NET wrapper

[MartinRobins]

Required Info
Camera Model	D415/D435
Firmware Version	05.09.13.00
Operating System & Version	Win 10
Platform	PC
SDK Version	2.14
Language	C#

Issue Description

I would like to be able to use the "rs2_set_region_of_interest" and "rs2_get_region_of_interest" API calls against a sensor from the .NET wrapper. I can see that the calls are defined in NativeMethods.cs, however they are not referenced from the Sensor class (or any others).

What I am trying to achieve is to be able to tell the camera the area of the picture that I am actually interested in so that where there are objects at different distances the area I define is the area with the best focus. I have not seen any other way to achieve this within the SDK and if I am approaching it incorrectly then please let me know.

Alternatively, if I were to try and update the wrapper myself (I think I can see what would be required), would this be acceptable - subject to standards etc. - as a pull request?

[MartinRobins]

Is it as simple as (within the class "Sensor" (Sensor.cs))...

    public void GetRegionOfInterest(out int minX, out int minY, out int maxX, out int maxY)
    {
        object error;
        NativeMethods.rs2_get_region_of_interest(m_instance, out minX, out minY, out maxX, out maxY, out error);
    }

    public void SetRegionOfInterest(int minX, int minY, int maxX, int maxY)
    {
        object error;
        NativeMethods.rs2_set_region_of_interest(m_instance, minX, minY, maxX, maxY, out error);
    }

[dorodnic]

Hi @MartinRobins Thanks for pointing that out. We did seem to simply miss these methods. Please consider creating a pull-request.

[MartinRobins]

@dorodnic As per my comment, is it as simple as I think? If so, I will create the pull request as suggested.

[dorodnic]

set_roi certainly is, get_roi I'm not 100% sure, but I hope P/Invoke will correctly manage out variables. Ideally, you want to put these two functions in a sub-class of Sensor (similar to how AdvancedDevice extends Device) because not all sensors support this operation. CONTRIBUTING.md offers more info on contributing.

[MartinRobins]

I have managed to identify (I think) that in order to use the region_of_interest API calls the sensor must support rs2_is_sensor_extendable_to(Extension.Roi), however I am encountering a problem that I was not expecting.

If you use realSense-viewer.exe, it offers the ability (via a button) to select the ROI on the RGB camera and does not error if that button is used and a region selected. However, when querying rs2_is_sensor_extendable_to(Extension.Roi) on the Sensor IntPtr returned from the .NET wrapper for the RGB camera it returns false and if I try to call rs2_set_region_of_interest I get an error

Object does not support "librealsense::roi_sensor_interface" interface!

I am retrieving the sensors through the Sensors property Device property of the PipelineProfile returned from Pipeline.Start(Config)…

PipelineProfile pipelineProfile = pipeline.Start(config);
foreach (Sensor sensor in pipelineProfile.Device.Sensors.Where(sensor => sensor.IsExtendableTo(Extension.Roi)))

Sensor.IsExtendable is defined as...

    public Boolean IsExtendableTo(Extension extension)
    {
        object error;
        return Convert.ToBoolean(NativeMethods.rs2_is_sensor_extendable_to(m_instance, extension, out error));
    }

Why would the API be available within the viewer but not the API?

[dorodnic]

It is a GUI bug in the Viewer. The hardware doesn't support RGB exposure-ROI, unfortunately.

[MartinRobins]

Not the answer I was hoping for; perhaps if I describe what I am trying to do you can advise me on an alternative approach?

I have a D435 mounted on its side with the sensor side at an angle about 2m above the floor and am using the depth sensor to trigger photographs when an object comes within a certain range of a specified point within the depth image. I need the camera to be on its side because I want a portrait image (due to the height of some of the objects I want to photograph combined with the proximity to where the camera is mounted).

Currently, I am getting the pictures, but the main region of interest is what would normally be the left half of the image if the camera was mounted normally (landscape) however I often find (especially for smaller objects) that the object of interest is slightly out of focus but the background is in perfect focus. This is of course further complicated by the fact that the object itself is often moving.

Is there a way that I can tell the camera that I want the focus within the captured RGB image to be concentrated on a particular part of the overall image?

Thanks.

[MartinRobins]

Sorry, pressed the wrong button and closed the issue by accident.

[dorodnic]

Hi @MartinRobins Unfortunately, there is very limited flexibility offered by the RGB camera. As far as I know, the sensor has fixed focus. The auto-exposure is configurable on the stereo-module (depth and infrared) but not the RGB module, however, it doesn't seem like your problem is with exposure.
You can manually control exposure and gain in software, but not focus.

[mpcuter]

Is it possible to provide a software auto exposure algorithm through the realsense SDK?

This would be very helpful for my companies use case.

Thank you

[craigpears]

Hi @dorodnic

I'm a colleague of Martin's that's working on this as well.

Is there any information regarding the focus of the camera and what region is expected to be in focus? Is it designed for the middle to be in focus, with less focus on the periphery for example?

Probably really hoping here but given that motion is also involved, any guidelines on what the camera is designed for in terms of motion would be very helpful as well. Am I correct in my understanding is that the camera is designed to be used on things like drones and robots as a guidance mechanism?

Thanks, Craig

[dorodnic]

Hi @mpcuter For some D400 models that do not have a dedicated ISP we do run the auto-exposure in software. You can see the source code here. Adapting it to run on the D435/D415 can be done, but would require some work. That said, it is not recommended. You would probably need to decimate the color image to lower VGA / QVGA resolution to run the histogram analysis at reasonable rate, and even then the convergence time would be quite long since it take couple of frames for the firmware to register exposure / gain change and additional 60-120 ms for the frames to get back to algo.

@craigpears - I don't have this information right now but we will do our best to find and share it.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] @craigpears The RGB camera in D435 is OV2740, you can go to their website to check the information you need. please check the 3.6.3 section in https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-D400-Series-Datasheet.pdf

@mpcuter Per drodric said, the HW doesn't support exposure-ROI, so we are not able to add it into the RealSense SDK.

[dorodnic]

@MartinRobins Correction to my previous reply - we were able to enable RGB AE-ROI in the ISP firmware. The firmware supporting this should be released soon

[dorodnic]

With the new firmware released you can now use Auto-Exposure ROI on the color sensor. Also, the APIs mentioned in this issue were added to the C# wrapper.