using the feed in opencv

[csaail]

how can i use this feed in opencv for image processing

[nobu835]

you can use h264decoder: https://github.com/DaWelter/h264decoder

[nobu835]

This is the simplest example. Combined the h264decoder sample and the pylwdrone sample.

import cv2 # you can use opencv!

# pylwdrone setup
import pylwdrone
drone = pylwdrone.LWDrone()

# based on the "H264 Decoder Python Module" Example (copy and pasted)
import h264decoder
import numpy as np

#f = open(thefile, 'rb')
decoder = h264decoder.H264Decoder()
while 1:
#  data_in = f.read(1024)
#  if not data_in:
#    break
#  framedatas = decoder.decode(data_in)

  for _frame in drone.start_video_stream(): # pylwdrone "Stream live video" Example
    framedatas=decoder.decode(bytes(_frame.frame_bytes))
    # Note that the variable name "frame" is the same in both pyledrone and h264decoder.
    # Here, we change the "frame" of pylwdrone to "_frame".
    # `type(_frame.frame_bytes)`  is `<class 'bytearray'>`.
    # `decoder.decode()`  takes <class 'bytes'> as argument.
    # Here we explicitly convert it to 'bytes' type.

    # Let's go back to the "H264 Decoder Python Module" Example.
    for framedata in framedatas:
      (frame, w, h, ls) = framedata
      if frame is not None:
          #print('frame size %i bytes, w %i, h %i, linesize %i' % (len(frame), w, h, ls))
          frame = np.frombuffer(frame, dtype=np.ubyte, count=len(frame))
          frame = frame.reshape((h, ls//3, 3))
          frame = frame[:,:w,:]
          # End of "H264 Decoder Python Module" Example

          frame=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB) # cv2 uses BGR
          cv2.imshow("window",frame)
          cv2.waitKey(1)

By using the threading module, you can do something else (such as piloting a drone) while watching the video. If you want a sample of this, please let me know. I plan to try making it from now on too.

[csaail]

ok i'll give it a try, but i was not able to build the h264decoder for python so i shifted to ffmpeg and created a flaskapp which basically gives you a video streaming link which i can acces in anyopencv imageprocessing, so rn i am able to control my drone using gestures/ signs and colors, etc.. the only issue i am facing is there is a delay in feed

import cv2
import numpy as np
import subprocess
import threading
from flask import Flask, render_template, Response

#cap = cv2.VideoCapture("http://127.0.0.1:5000/video_feed")  # Use the URL as the video source

app = Flask(__name__)

frame = None  # Global variable to store the current frame

def capture_video():
    global frame
    process = subprocess.Popen(
        ["plutocam", "stream", "start", "--out-file", "-"],
        stdout=subprocess.PIPE
    )

    ffmpeg_process = subprocess.Popen(
        ["ffmpeg", "-i", "-", "-f", "rawvideo", "-pix_fmt", "bgr24", "-"],
        stdin=process.stdout,
        stdout=subprocess.PIPE
    )

    while True:
        # Read the frame from ffmpeg output
        raw_frame = ffmpeg_process.stdout.read(2048 * 1152 * 3)  # Adjust dimensions based on your camera's resolution
        if not raw_frame:
            break

        # Convert the byte data to a numpy array
        frame = np.frombuffer(raw_frame, dtype=np.uint8).reshape((1152, 2048, 3))

        # Display the frame using OpenCV
        #cv2.imshow('Video Stream', frame)

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

    # Clean up: terminate the process and close all OpenCV windows
    process.terminate()
    ffmpeg_process.terminate()
    cv2.destroyAllWindows()

# Flask route to serve the video feed
@app.route('/video_feed')
def video_feed():
    def generate():
        global frame
        while True:
            if frame is None:
                continue
            _, jpeg = cv2.imencode('.jpg', frame)
            yield (b'--frame\r\n'
                   b'Content-Type: image/jpeg\r\n\r\n' + jpeg.tobytes() + b'\r\n\r\n')
    return Response(generate(), mimetype='multipart/x-mixed-replace; boundary=frame')

@app.route('/')
def index():
    return render_template('index.html')

if __name__ == '__main__':
    # Start video capture thread
    video_thread = threading.Thread(target=capture_video)
    video_thread.daemon = True
    video_thread.start()

    # Start Flask app
    app.run(host='0.0.0.0', port=5000)

[csaail]

import cv2
import numpy as np
from pyzbar.pyzbar import decode

cap = cv2.VideoCapture("http://127.0.0.1:5000/video_feed")
cap.set(3,640)
cap.set(4,480)

while True:

    success, img = cap.read()
    for barcode in decode(img):
        myData = barcode.data.decode('utf-8')
        print(myData)
        pts = np.array([barcode.polygon],np.int32)
        pts = pts.reshape((-1,1,2))
        cv2.polylines(img,[pts],True,(255,0,255),5)
        pts2 = barcode.rect
        cv2.putText(img,myData,(pts2[0],pts2[1]),cv2.FONT_HERSHEY_SIMPLEX, 0.9,(255,0,255),2)

    cv2.imshow('Result',img)
    cv2.waitKey(1)

[csaail]

plutocam is same as pywdrone, i changed the port numbers which were different in my cam

[nobu835]

Sorry, I'm not familiar with flask or pyzbar, so I can only give advice on the h264decoder.

but i was not able to build the h264decoder for python

If you don't mind, could you tell me what happened? might be I can help you.

For reference, here is a summary of how to build the h264decoder (for Windows):

1. get vcpkg git clone https://github.com/microsoft/vcpkg.git
2. install vcpkg cd vcpkg .\bootstrap-vcpkg.bat
3. install ffmpeg via vcpkg vcpkg.exe install ffmpeg:x64-windows Note: this step take some hours. Note: It's not frozen. You can check CPU usage in Task Manager (≒60% ?). Note: It consumes several GB of disk space.
4. get h264decoder source cd /any/path/you/like git clone https://github.com/DaWelter/h264decoder cd h264decoder
5. Fix the code open h264decoder/src/h264decoder_python.cpp on code editor. replace "ssize_t" to "SSIZE_T". Note: Do not change Py_ssize_t at line 110 Note: You should use the Match Whole Word feature (like ↓)
6. build h264decoder python setup.py build_ext --cmake-args="-DCMAKE_TOOLCHAIN_FILE=[path to vcpkg]/scripts/buildsystems/vcpkg.cmake"

[csaail]

Actually, i tried this few months back, so i am not sure what was tbe issue, I'll give it another try and let you know what isue i am facing..