search

luxonis / robothub

MIT License
1 stars 4 forks source link

Issue in getting object detection on live stream #53

Open Drakshayani-123 opened 10 months ago

Drakshayani-123 commented 10 months ago

I have accessing live stream of OAK -D-POE in RobotHuB. I have a yolov8 custom object detection .best file to detect various parts of robot. I converted that to blob and onnx file through tools.luxonis. Then used blob file in depthai code and copied that code in code edit of robotHub. But not getting live detection

PetrNovota commented 10 months ago

Hi @Drakshayani-123 Please can you share a code example of what you are doing?

Drakshayani-123 commented 10 months ago

Hi Luxonis/Robothub Below is the code I am using for custom object detection robotic arm movement. I trained yolov8 model. Generated best.pt file- converted to blob file that I am using in code. import cv2 import depthai as dai import numpy as np

nnPath = "best-300_openvino_2022.1_6shave.blob" # Change blob path labelMap = ["Upper arm","Lower arm","Gripper"] # Change labelMap

Create pipeline

pipeline = dai.Pipeline()

Define sources and outputs

monoRight = pipeline.create(dai.node.MonoCamera) manip = pipeline.create(dai.node.ImageManip) nn = pipeline.create(dai.node.MobileNetDetectionNetwork) manipOut = pipeline.create(dai.node.XLinkOut) nnOut = pipeline.create(dai.node.XLinkOut)

manipOut.setStreamName("right") nnOut.setStreamName("nn")

Properties

monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT) monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)

manip.initialConfig.setResize(384, 384) # Input frame shape manip.initialConfig.setFrameType(dai.ImgFrame.Type.GRAY8) # Model expects Grayscale image

nn.setConfidenceThreshold(0.5) nn.setBlobPath(nnPath) nn.input.setBlocking(False)

monoRight.out.link(manip.inputImage) manip.out.link(nn.input) manip.out.link(manipOut.input) nn.out.link(nnOut.input)

with dai.Device(pipeline) as device:

qRight = device.getOutputQueue("right", maxSize=4, blocking=False)
qDet = device.getOutputQueue("nn", maxSize=4, blocking=False)

frame = None
detections = []

def frameNorm(frame, bbox):
    normVals = np.full(len(bbox), frame.shape[0])
    normVals[::2] = frame.shape[1]
    return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)

def displayFrame(name, frame):
    color = (255, 255, 0)
    for detection in detections:
        bbox = frameNorm(frame, (detection.xmin, detection.ymin,

detection.xmax, detection.ymax)) cv2.putText(frame, labelMap[detection.label], (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, color) cv2.putText(frame, f"{int(detection.confidence * 100)}%", (bbox[0] + 10, bbox[1] + 40), cv2.FONT_HERSHEY_TRIPLEX, 0.5, color) cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color, 2) cv2.imshow(name, frame)

while True:
    frame = qRight.get().getCvFrame()
    frame =  cv2.cvtColor(frame, cv2.COLOR_GRAY2BGR) # For colored

visualization detections = inDet = qDet.get().detections displayFrame("right", frame)

    if cv2.waitKey(1) == ord('q'):
        break

Thanks Drakshayani

On Thu, Dec 14, 2023 at 2:44 PM Petr Novota @.***> wrote:

Hi @Drakshayani-123 https://github.com/Drakshayani-123 Please can you share a code example of what you are doing?

— Reply to this email directly, view it on GitHub https://github.com/luxonis/robothub/issues/53#issuecomment-1855463664, or unsubscribe https://github.com/notifications/unsubscribe-auth/BES7SSTJXATP7WZYNVM5RZLYJK7QPAVCNFSM6AAAAABAUIG442VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJVGQ3DGNRWGQ . You are receiving this because you were mentioned.Message ID: @.***>

PetrNovota commented 10 months ago

@Drakshayani-123 the reason it doesn't work might be that you have a Yolo model but you are using the MobileNet node. Try using the YoloDetection node from the depthai library instead.

Drakshayani-123 commented 10 months ago

Now I am trying below yoloDetection code but this time error is: 2023-12-18T14:15:18.410Z | stdout | [1844301031F7FF0800] [10.10.138.21] [243.087] [DetectionNetwork(1)] [error] Input tensor 'images' (0) exceeds available data range. Data size (519168B), tensor offset (0), size (1228800B) - skipping inference

below is the code on live stream on robothub:

!/usr/bin/env python3

""" The code is the same as for Tiny Yolo V3 and V4, the only difference is the blob file

from pathlib import Path import sys import cv2 import depthai as dai import numpy as np import time

Get argument first

nnPath = str((Path(file).parent / Path('best-300_openvino_2022.1_6shave.blob')).resolve().absolute()) # change blob file location

labelMap = [ "Upper arm", "Lower arm", "Gripper" ]

syncNN = True

Create pipeline

pipeline = dai.Pipeline()

Define sources and outputs

camRgb = pipeline.create(dai.node.ColorCamera) detectionNetwork = pipeline.create(dai.node.YoloDetectionNetwork) xoutRgb = pipeline.create(dai.node.XLinkOut) nnOut = pipeline.create(dai.node.XLinkOut)

xoutRgb.setStreamName("rgb") nnOut.setStreamName("nn") A

Properties

camRgb.setPreviewSize(416, 416) camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P) camRgb.setInterleaved(False) camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR) camRgb.setFps(40)

Network specific settings

detectionNetwork.setConfidenceThreshold(0.5) detectionNetwork.setNumClasses(80) detectionNetwork.setCoordinateSize(4) detectionNetwork.setAnchors([10, 14, 23, 27, 37, 58, 81, 82, 135, 169, 344, 319]) detectionNetwork.setAnchorMasks({"side26": [1, 2, 3], "side13": [3, 4, 5]}) detectionNetwork.setIouThreshold(0.5) detectionNetwork.setBlobPath(nnPath) detectionNetwork.setNumInferenceThreads(2) detectionNetwork.input.setBlocking(False)

Linking

camRgb.preview.link(detectionNetwork.input) if syncNN: detectionNetwork.passthrough.link(xoutRgb.input) else: camRgb.preview.link(xoutRgb.input)

detectionNetwork.out.link(nnOut.input)

Connect to device and start pipeline

with dai.Device(pipeline) as device:

# Output queues will be used to get the rgb frames and nn data from the

outputs defined above qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False) qDet = device.getOutputQueue(name="nn", maxSize=4, blocking=False)

frame = None
detections = []
startTime = time.monotonic()
counter = 0
color2 = (255, 255, 255)

# nn data, being the bounding box locations, are in <0..1> range - they

need to be normalized with frame width/height def frameNorm(frame, bbox): normVals = np.full(len(bbox), frame.shape[0]) normVals[::2] = frame.shape[1] return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)

def displayFrame(name, frame):
    color = (255, 0, 0)
    for detection in detections:
        bbox = frameNorm(frame, (detection.xmin, detection.ymin,

detection.xmax, detection.ymax)) cv2.putText(frame, labelMap[detection.label], (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255) cv2.putText(frame, f"{int(detection.confidence * 100)}%", (bbox[0] + 10, bbox[1] + 40), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255) cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color, 2)

Show the frame

    cv2.imshow(name, frame)

while True:
    if syncNN:
        inRgb = qRgb.get()
        inDet = qDet.get()
    else:
        inRgb = qRgb.tryGet()
        inDet = qDet.tryGet()

    if inRgb is not None:
        frame = inRgb.getCvFrame()
        cv2.putText(frame, "NN fps: {:.2f}".format(counter /

(time.monotonic() - startTime)), (2, frame.shape[0] - 4), cv2.FONT_HERSHEY_TRIPLEX, 0.4, color2)

    if inDet is not None:
        detections = inDet.detections
        counter += 1

    if frame is not None:
        displayFrame("rgb", frame)

    if cv2.waitKey(1) == ord('q'):
        break

On Fri, Dec 15, 2023 at 9:43 PM Petr Novota @.***> wrote:

@Drakshayani-123 https://github.com/Drakshayani-123 the reason it doesn't work might be that you have a Yolo model but you are using the MobileNet node. Try using the YoloDetection https://docs.luxonis.com/projects/api/en/latest/components/nodes/yolo_detection_network/ node from the depthai library instead.

— Reply to this email directly, view it on GitHub https://github.com/luxonis/robothub/issues/53#issuecomment-1858126736, or unsubscribe https://github.com/notifications/unsubscribe-auth/BES7SSSIHE3GRQ26VDDQMRDYJRZJJAVCNFSM6AAAAABAUIG442VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJYGEZDMNZTGY . You are receiving this because you were mentioned.Message ID: @.***>

PetrNovota commented 10 months ago

Than you. @alex-luxonis or @themarpe can you assist here? The model input size seems to be correct.

alex-luxonis commented 10 months ago

[DetectionNetwork(1)] [error] Input tensor 'images' (0) exceeds available data range. Data size (519168B), tensor offset (0), size (1228800B) - skipping inference

Received image to NN matches camera settings, 519168 = 416 * 416 * 3, but looks like the NN model expects something else: 1228800 = 1024 * 400 * 3 or 800 * 512 * 3, etc. Please check the NN model input requirements. If it really needs some different resolution, try configuring the camera preview size to match.

Drakshayani-123 commented 10 months ago

Hi Petr

I did not get your solution.

Thanks Drakshayani

On Tue, Dec 19, 2023 at 9:27 PM Petr Novota @.***> wrote:

Than you. @alex-luxonis https://github.com/alex-luxonis or @themarpe https://github.com/themarpe can you assist here? The model input size seems to be correct.

— Reply to this email directly, view it on GitHub https://github.com/luxonis/robothub/issues/53#issuecomment-1863031735, or unsubscribe https://github.com/notifications/unsubscribe-auth/BES7SSUT2IREGJTCTOLX253YKG2MZAVCNFSM6AAAAABAUIG442VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRTGAZTCNZTGU . You are receiving this because you were mentioned.Message ID: @.***>

PetrNovota commented 10 months ago

I think the nn input size is correct - if incorrect then depthai throws a different error

st 20. 12. 2023 v 4:30 odesílatel Drakshayani-123 @.***> napsal:

Hi Petr

I did not get your solution.

Thanks Drakshayani

On Tue, Dec 19, 2023 at 9:27 PM Petr Novota @.***> wrote:

Than you. @alex-luxonis https://github.com/alex-luxonis or @themarpe https://github.com/themarpe can you assist here? The model input size seems to be correct.

— Reply to this email directly, view it on GitHub https://github.com/luxonis/robothub/issues/53#issuecomment-1863031735,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/BES7SSUT2IREGJTCTOLX253YKG2MZAVCNFSM6AAAAABAUIG442VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRTGAZTCNZTGU>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/luxonis/robothub/issues/53#issuecomment-1863793724, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXZ6YOJDEMOFMHE5TRZBTFLYKJLWXAVCNFSM6AAAAABAUIG442VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRTG44TGNZSGQ . You are receiving this because you commented.Message ID: @.***>