Imagemanip resized frame display in greyscale from fullfov NN demo

[justin-larking-pk]

Copied the code from the fullFOV NN inference demo on mobilenet ssd and the preview frame showing NN inferencing results is showing up in greyscale.

the image is rgb as detected bounding boxes show up in colour. However all r, g and b values output from the imagemanip node are the same.

[Luxonis-Brandon]

Thanks for the report. We'll investigate and fix.

[justin-larking-pk]

In addition to this when modifying the full FOV code to capture full FOV 4k+, the imagemanip nodes output also appears to be freezing.

#!/usr/bin/env python3

import cv2
import depthai as dai
import numpy as np
import time
import datetime

capture = False
count = 0

# Create pipeline
pipeline = dai.Pipeline()

print("creating colour camera")
# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setInterleaved(False)
#camRgb.setIspScale(1, 5) # 4032x3040

xoutIsp = pipeline.create(dai.node.XLinkOut)
xoutIsp.setStreamName("isp")
camRgb.isp.link(xoutIsp.input)

print("initialising imagemanip")
# Use ImageManip to resize to 1024x768x3 and convert YUV420 -> RGB
manip = pipeline.create(dai.node.ImageManip)
manip.setMaxOutputFrameSize(2359296) # 1024x768x3
manip.initialConfig.setResizeThumbnail(640, 480)
manip.initialConfig.setFrameType(dai.RawImgFrame.Type.RGB888p) # to display
camRgb.isp.link(manip.inputImage)

xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutRgb.setStreamName("rgb")
manip.out.link(xoutRgb.input)

print("starting device")
with dai.Device(pipeline) as device:
    #time.sleep(0.5)
    qRgb = device.getOutputQueue(name='rgb')
    qIsp = device.getOutputQueue(name='isp')
    print("initialised queues")
    now = datetime.datetime.now()
    prev_frame_time = 0
    new_frame_time = 0
    rgb = np.zeros((768, 1024, 3), np.int8)
    while True:
        new_frame_time = time.time()
        framergb = qRgb.get()
        rgb = framergb.getCvFrame()
        print("captured frame")
        fps = 1/(new_frame_time-prev_frame_time)
        prev_frame_time = new_frame_time
        cv2.putText(rgb, str(int(fps)), (900 ,50), cv2.FONT_HERSHEY_SIMPLEX, 2, (255,255,255), 4, 2)
        if capture:
            print("Saving to File")
            frameisp = qIsp.get()
            f = frameisp.getCvFrame()
            cv2.imwrite(now.strftime('%Y%m%d%H%M') + "image"+str(count)+".jpg", f)
            count+=1
            capture = False
            cv2.putText(rgb, "Image Captured", (10 ,550), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2, 2)
        cv2.imshow("rgb", rgb)
        if cv2.waitKey(1) == ord('c'):
            capture=True
        if cv2.waitKey(1) == ord('q'):
            break

With this code, maybe i'm making some obvious mistake here but this runs for a number of frames before stopping on the .get() blocking function. Indicating its no longer getting any frames. Does this even with a 0.5s wait so I'm not polling it to quickly.

The number of frames its stopping at appears to be consistent across runs but then changes sometimes. I was getting 6 frames then the code would be stuck on .get() however now its getting 21 frames.

[Erol444]

Hello @justin-larking , did you install requirements? python3 -mpip install -r requirements.txt, as there is a special version of depthai needed for this demo to run.

[justin-larking-pk]

Works fine on pc however not on RPi 4 running raspberian.

PC:

RPi4: Code is slightly different, but both images are resized by imagemanip and then displayed using opencv. (the part in the centre view is orange)

when running requirements.py on the RPi4 I got this error:

python3 -m pip install -r requirements.txt Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/ Collecting opencv-python==4.5.1.48 (from -r requirements.txt (line 1)) Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/opencv-python/

So I removed opencv from the requirements.txt list and ran:

python3 -m pip install -r requirements.txt Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/ Requirement already satisfied: numpy in /home/pi/.local/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (1.20.3) Requirement already satisfied: depthai==2.13.2.0.dev0+87552f67f0820a6eef058550af2c27724795fefe in /home/pi/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (2.13.2.0.dev0+87552f67f0820a6eef058550af2c27724795fefe)

Same version of depthai but the rpi4 imagemanip resized image is in greyscale.

[themarpe]

CC: @Erol444 can you help out on this?

[maxsitt]

Same problem here on RPi Zero 2 W. rgb output is only greyscale. The requirements do not install properly on RPi, which might be the source of this problem. ERROR: Failed building wheel for depthai Failed to build depthai ERROR: Could not build wheels for depthai, which is required to install pyproject.toml-based projects

[Erol444]

Hello @maxsitt , could you run uname -a on your RPI Zero 2? We should have prebuilt wheels for 3.7 and 3.9 (see here), but only for armv7 I believe.

[maxsitt]

I'm running Bullseye (Python 3.9) on the RPi Zero 2. I used this fix from Alex from the Discord: "If not, as an alternative, I updated the app to use the latest release, on the branch full_fov_nn_main https://github.com/luxonis/depthai-experiments/compare/full_fov_nn_main There's only a minor horizontal crop happening on the original frame, of 1.5%, from 812 to 800 pixels. We'll fix the FW for this." (https://discord.com/channels/790680891252932659/924800254535143464/939636496045060136) Now everything is working fine!