search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.59k stars 4.82k forks source link

Converting depth filter opencv example in C++ to python #4842

Closed alter-sachin closed 4 years ago

alter-sachin commented 5 years ago
Required Info
Camera Model D400 }
Firmware Version (Open RealSense Viewer --> Click info)
Operating System & Version Linux (Ubuntu16)
Kernel Version (Linux Only) (e.g. 4.15.13)
Platform PC.
Language python }
Segment {Drones }

Issue Description

<Describe your issue / question / feature request / etc..>

Hi, I hope this finds you well . I am using the realsense D435 to do some obstacle avoidance. I am referring to the depth filter code and read here: https://github.com/IntelRealSense/librealsense/blob/7a3ffaf412401849894a4d69791b98559f624d47/wrappers/opencv/depth-filter/readme.md

I have managed to create equivalent main filters using python opencv:

`def main_filter(self, frame): source = frame

self.filter_edge(source)

    #self.filter_corner(source)
    #source = frame
    edge_mask_1 = self.filter_edge(source)
   #print("edge_mask")
    #print(edge_mask_1.dtype)

    #cv2.imwrite("edge_mask.png",edge_mask_1)
    harris_mask_1 = self.filter_corner(source)
    #harris_mask_1 = harris_mask_1.astype(np.uint8)
    #print("harris_mask")
    #print(harris_mask_1.dtype)
    #cv2.imwrite("harris_mask.png",harris_mask_1)

    #combined_mask = 
    combined_mask = np.zeros((480, 848), dtype = "uint8")

    combined_mask = cv2.bitwise_or(edge_mask_1,harris_mask_1)
    #print(combined_mask)
    mask1 = cv2.morphologyEx(combined_mask, cv2.MORPH_CLOSE,
                             cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3)))
    #print(mask1)
    #cv2.imwrite("mask1.png",mask1)
    return mask1

def filter_edge(self, frame):
    source = frame
    #skinCrCbHist = np.zeros((256, 256, 1), dtype = "uint16")
    source = np.int16(source)
    #print("filter edge source")
    #print(source)
    # cv::Scharr(area->decimated_ir, area->scharr_x, CV_16S, 1, 0);
    scharr_x = cv2.Scharr(source, -1, 1, 0)
    #print("filter edge scharr_x")
    #print(scharr_x)
    abs_scharr_x = cv2.convertScaleAbs(scharr_x)
    #print("abs_scharr_x")
    #print(abs_scharr_x)
    # cv::convertScaleAbs(area->scharr_x, area->abs_scharr_x);
    # cv::Scharr(area->decimated_ir, area->scharr_y, CV_16S, 0, 1);
    scharr_y = cv2.Scharr(source, -1, 0, 1)
    #print("abs_scharr_y")
    #print(abs_scharr_y)
    abs_scharr_y = cv2.convertScaleAbs(scharr_y)
    #print("absolute y")
    #print(abs_scharr_y)
    # cv::convertScaleAbs(area->scharr_y, area->abs_scharr_y);
    edge_mask = cv2.addWeighted(abs_scharr_y, 0.5, abs_scharr_x, 0.5, 0)
    # cv::addWeighted(area->abs_scharr_y, 0.5, area->abs_scharr_y, 0.5, 0, area->edge_mask);
    retval, binary = cv2.threshold(edge_mask, 192, 255, cv2.THRESH_BINARY)
    #print(retval)#f(retval == True):
    return binary
    # cv::threshold(area->edge_mask, area->edge_mask, 192, 255, cv::THRESH_BINARY);

def filter_corner(self, frame):
    source = frame
    float_ir = np.float32(source)
    # area->decimated_ir.convertTo(area->float_ir, CV_32F);
    corners = cv2.cornerHarris(float_ir, 2, 3, 0.04)
    # cv::cornerHarris(area->float_ir, area->corners, 2, 3, 0.04);
    retval, binary = cv2.threshold(corners, 300, 255, cv2.THRESH_BINARY)
    # cv::threshold(area->corners, area->corners, 300, 255, cv::THRESH_BINARY);
    #if(retval == True):
    harris_mask = np.uint8(binary)
    return harris_mask
    # area->corners.convertTo(area->harris_mask, CV_8U);`

I then access the same in another function using: ` filtered_ir_image = self.main_filter(ir1_image)

scale_percent = 60 # percent of original size

            width = int(filtered_ir_image.shape[1]/decimation_scale)
            height = int(filtered_ir_image.shape[0]/decimation_scale)
            dim = (width, height)
            # resize image
            resize = cv2.resize(filtered_ir_image, dim, interpolation = cv2.INTER_AREA)

            #print(filtered_ir_image)
            ##########
            new_depth = new_depth * (resize.astype(new_depth.dtype))`

I was hoping this would be enough to get a new distance array from my obstacles. I would like to understand how the SDK portion of the guide be implemented in python ? Is that even necessary ? Any pointers would be great.

https://github.com/IntelRealSense/librealsense/blob/7a3ffaf412401849894a4d69791b98559f624d47/wrappers/opencv/depth-filter/readme.md#sdk-integration

dorodnic commented 5 years ago

Hi @alter-sachin We will probably port this example in future to python. For now, you can skip parts you don't need (like SDK integration). Overall, this is just one of possible heuristics to improve depth for collision avoidance, but it proven very useful in real-life applications on drones. Let me know if you have questions about the code

alter-sachin commented 5 years ago

@dorodnic Thankyou for the prompt reply. I am not sure about the SDK integration part. I have applied the mask generated using the main filter(edge and corner combined) and applied it to the original decimated depth.

Is that all I need to do ?

If yes, my decimated frame needs to use the mask , how do I apply it properly is where I am worried.

Thanks :)