search

dusty-nv / jetson-inference

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
https://developer.nvidia.com/embedded/twodaystoademo
MIT License
7.71k stars 2.97k forks source link

[OpenGL] failed to create X11 Window. #1712

Open Aamnastressed2 opened 1 year ago

Aamnastressed2 commented 1 year ago

import cv2 from jetson_inference import detectNet from jetson_utils import videoSource, videoOutput

Create object detection network

net = detectNet("ssd-mobilenet-v2", threshold=0.5)

Create video sources for each camera

camera1 = videoSource("csi://0")
camera2 = videoSource("csi://1")

Create video outputs for each camera

display1 = videoOutput("display://0") display2 = videoOutput("display://1")

Camera specifications

baseline_length = 0.06 # 60 mm baseline length in meters focal_length_pixels = 3280 2.6 / (2 1.4) # Using the formula: Focal Length (in pixels) = (Image Width Focal Length) / (2 CMOS size) pixel_size = 1.4e-6 # 1/4 inch CMOS size, converted to meters

Stereo Block Matching parameters

block_size = 5 min_disparity = 0 num_disparities = 64 unique_ratio = 5

Create the stereo block matching object

stereo_bm = cv2.StereoBM_create(numDisparities=num_disparities, blockSize=block_size)

while display1.IsStreaming() and display2.IsStreaming():

Capture frames from each camera

img1 = camera1.Capture()
img2 = camera2.Capture()

if img1 is not None and img2 is not None:
    # Detect objects in the image from the first camera
    detections1 = net.Detect(img1)
    # Detect objects in the image from the second camera
    detections2 = net.Detect(img2)

    # Convert images to grayscale (block matching works on grayscale images)
    gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
    gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

    # Perform stereo block matching to compute the disparity map
    disparity = stereo_bm.compute(gray1, gray2)

    # Iterate through the detections and calculate distance for each detected object
    for detection1, detection2 in zip(detections1, detections2):
        # Calculate the center pixel coordinates of the detection in the left image
        x1, y1 = int(detection1.Center[0]), int(detection1.Center[1])

        # Get the disparity value for the corresponding point in the right image
        disparity_value = disparity[y1, x1]

        # Calculate the distance using the disparity value and camera-specific parameters
        distance = (baseline_length * focal_length_pixels) / (disparity_value * pixel_size)

        # Convert the distance to a string and add it to the label
        label = "{:.2f} meters".format(distance)

        # Draw a bounding box and label for the detection in the first image
        cv2.rectangle(img1, (detection1.Left, detection1.Top), (detection1.Right, detection1.Bottom), (0, 255, 0), 2)
        cv2.putText(img1, label, (detection1.Left, detection1.Top - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)

        # Draw a bounding box and label for the detection in the second image
        cv2.rectangle(img2, (detection2.Left, detection2.Top), (detection2.Right, detection2.Bottom), (0, 255, 0), 2)
        cv2.putText(img2, label, (detection2.Left, detection2.Top - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)

    # Render the image and detections in the first display
    display1.Render(img1)
    # Render the image and detections in the second display
    display2.Render(img2)

    # Set the status of the first display
    display1.SetStatus("Object Detection | Network {:.0f} FPS".format(net.GetNetworkFPS()))
    # Set the status of the second display
    display2.SetStatus("Object Detection | Network {:.0f} FPS".format(net.GetNetworkFPS()))

ERRjetson@nano:~/jetson-inference/build/aarch64/bin$ python3 depth.py jetson.inference -- detectNet loading custom model '(null)'

detectNet -- loading detection network model from: -- model networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff -- input_blob 'Input' -- output_blob 'NMS' -- output_count 'NMS_1' -- class_labels networks/SSD-Mobilenet-v2/ssd_coco_labels.txt -- threshold 0.500000 -- batch_size 1

[TRT] TensorRT version 8.0.1 [TRT] loading NVIDIA plugins... [TRT] Registered plugin creator - ::GridAnchor_TRT version 1 [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1 [TRT] Registered plugin creator - ::NMS_TRT version 1 [TRT] Registered plugin creator - ::Reorg_TRT version 1 [TRT] Registered plugin creator - ::Region_TRT version 1 [TRT] Registered plugin creator - ::Clip_TRT version 1 [TRT] Registered plugin creator - ::LReLU_TRT version 1 [TRT] Registered plugin creator - ::PriorBox_TRT version 1 [TRT] Registered plugin creator - ::Normalize_TRT version 1 [TRT] Registered plugin creator - ::ScatterND version 1 [TRT] Registered plugin creator - ::RPROI_TRT version 1 [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1 [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1 [TRT] Could not register plugin creator - ::FlattenConcat_TRT version 1 [TRT] Registered plugin creator - ::CropAndResize version 1 [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1 [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1 [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1 [TRT] Registered plugin creator - ::Proposal version 1 [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1 [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1 [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1 [TRT] Registered plugin creator - ::Split version 1 [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1 [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1 [TRT] detected model format - UFF (extension '.uff') [TRT] desired precision specified for GPU: FASTEST [TRT] requested fasted precision for device GPU without providing valid calibrator, disabling INT8 [TRT] [MemUsageChange] Init CUDA: CPU +202, GPU +0, now: CPU 230, GPU 3812 (MiB) [TRT] native precisions detected for GPU: FP32, FP16 [TRT] selecting fastest native precision for GPU: FP16 [TRT] found engine cache file networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.1.1.8001.GPU.FP16.engine [TRT] found model checksum networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.sha256sum [TRT] echo "$(cat networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.sha256sum) networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff" | sha256sum --check --status [TRT] model matched checksum networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.sha256sum [TRT] loading network plan from engine cache... networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.1.1.8001.GPU.FP16.engine [TRT] device GPU, loaded networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 264, GPU 3881 (MiB) [TRT] Loaded engine size: 33 MB [TRT] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 264 MiB, GPU 3881 MiB [TRT] Using cublas a tactic source [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +158, GPU -87, now: CPU 440, GPU 3799 (MiB) [TRT] Using cuDNN as a tactic source [TRT] [MemUsageChange] Init cuDNN: CPU +241, GPU +45, now: CPU 681, GPU 3844 (MiB) [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 681, GPU 3853 (MiB) [TRT] Deserialization required 8153995 microseconds. [TRT] [MemUsageSnapshot] deserializeCudaEngine end: CPU 681 MiB, GPU 3855 MiB [TRT] [MemUsageSnapshot] ExecutionContext creation begin: CPU 681 MiB, GPU 3855 MiB [TRT] Using cublas a tactic source [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU -4, now: CPU 681, GPU 3851 (MiB) [TRT] Using cuDNN as a tactic source [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU -1, now: CPU 681, GPU 3850 (MiB) [TRT] Total per-runner device memory is 22143488 [TRT] Total per-runner host memory is 137024 [TRT] Allocated activation device memory of size 14256640 [TRT] [MemUsageSnapshot] ExecutionContext creation end: CPU 683 MiB, GPU 3874 MiB [TRT]
[TRT] CUDA engine context initialized on device GPU: [TRT] -- layers 124 [TRT] -- maxBatchSize 1 [TRT] -- deviceMemory 14256640 [TRT] -- bindings 3 [TRT] binding 0 -- index 0 -- name 'Input' -- type FP32 -- in/out INPUT -- # dims 3 -- dim #0 3 -- dim #1 300 -- dim #2 300 [TRT] binding 1 -- index 1 -- name 'NMS' -- type FP32 -- in/out OUTPUT -- # dims 3 -- dim #0 1 -- dim #1 100 -- dim #2 7 [TRT] binding 2 -- index 2 -- name 'NMS_1' -- type FP32 -- in/out OUTPUT -- # dims 3 -- dim #0 1 -- dim #1 1 -- dim #2 1 [TRT]
[TRT] binding to input 0 Input binding index: 0 [TRT] binding to input 0 Input dims (b=1 c=3 h=300 w=300) size=1080000 [TRT] binding to output 0 NMS binding index: 1 [TRT] binding to output 0 NMS dims (b=1 c=1 h=100 w=7) size=2800 [TRT] binding to output 1 NMS_1 binding index: 2 [TRT] binding to output 1 NMS_1 dims (b=1 c=1 h=1 w=1) size=4 [TRT]
[TRT] device GPU, networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff initialized. [TRT] W = 7 H = 100 C = 1 [TRT] detectNet -- maximum bounding boxes: 100 [TRT] loaded 91 class labels [TRT] detectNet -- number of object classes: 91 [TRT] loaded 0 class colors [TRT] didn't load expected number of class colors (0 of 91) [TRT] filling in remaining 91 class colors with default colors [gstreamer] initialized gstreamer, version 1.14.5.0 [gstreamer] gstCamera -- attempting to create device csi://0 [gstreamer] gstCamera pipeline string: [gstreamer] nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=30/1, format=(string)NV12 ! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM) ! appsink name=mysink [gstreamer] gstCamera successfully created device csi://0 [video] created gstCamera from csi://0

gstCamera video options:

-- URI: csi://0

dusty-nv commented 1 year ago

@Aamnastressed2 can you try putting the import cv2 after you create the videoOutput() objects? cv2 module likes to mess with the OpenGL bindings