Is it possible to connect with OpenCV?

cgbahk commented 6 years ago

I want to get streaming video from mambo FPV, and want to process it with powerful opencv library (via python). So it would be best to connect rtsp streaming server with opencv module and what I am trying to find out is its availability.

As far as I understand, @amymcgovern tried to connect using opencv first in /pymambo repo. Then it is replaced by vlc because of some error in /pyparrot repo here.

So is it still unresolved problem, or there is some way to break through, not shown in source code? If it’s impossible, I would catch frames using vlc then send it to opencv and do vision-things.

Thanks in advance :)

amymcgovern commented 6 years ago

The bug is NOT in pyparrot. The bug is in opencv. I do not think it is fixed but feel free to try. Last I tried (a month or two ago), it was still there. It will open the stream and process about 10 frames and then stop.

cgbahk commented 6 years ago

The symptom is exactly same in my case. (10 frames and stop)

amymcgovern commented 6 years ago

Yeah, not my bug, alas. Well acknowledged inside opencv but no one working to fix it, so far as I can tell.

amymcgovern commented 6 years ago

Either ffmpeg or VLC will work though

PascPeli commented 6 years ago

Excuse me if I am off topic but I get using the demoMamboVisionGUI.py I cannot even see 1 frame. the GUI pops up but it's just black. I'm using Py 3.6.6

amymcgovern commented 6 years ago

The mambo turns off its vision stream after one minute of inactivity. To reactivate it, take off (using RC or code) or reboot it.

lengyuyu2333 commented 5 years ago

Hello. Have you ever met the problem that cannot import cv2? Now, can we use opencv and ros kinetic in the same environment?

Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 21:41:56) [GCC 7.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import cv2 Traceback (most recent call last): File "", line 1, in ImportError: /opt/ros/kinetic/lib/python2.7/dist-packages/cv2.so: undefined symbol: PyCObject_Type

lengyuyu2333 commented 5 years ago

Using pyparrot, is there any ways to obtain videos or images without opencv? (with ffmpeg or VLC maybe) I am looking forward to your reply :)

amymcgovern commented 5 years ago

ffmpeg and VLC are how you do it now. So I'm confused by your question... Look at droneVision and demoMamboVision and demoMamboVisionGUI.

amymcgovern commented 5 years ago

Also your issue with importing is that you are trying to import in python 2.7. it shows that clearly in your packages. pyparrot is python 3.

mmaz commented 5 years ago

Hi @amymcgovern I have been able to connect to the front camera of the Mambo FPV via OpenCV using gstreamer, which I think bypasses the need to include libvlc.py -- but there's probably not much of a difference in practice between the two approaches.

I wrote up some instructions here (for 16.30/16.31 students at MIT): http://mmaz.github.io/mambo_notes

Hope this helps folks using the Mambo!

amymcgovern commented 5 years ago

Have you done this on multiple OS's? What I found was it worked ok on linux but not consistently on Mac or windows. If so, I'll look into this!

mmaz commented 5 years ago

I’ve only tested on Linux unfortunately - I will ask students if any of them have tried compiling OpenCV against GStreamer on windows or mac

amymcgovern commented 5 years ago

Thanks! If this solves the streaming issue for multiple OS's, it would be great. I don't like the delay introduced by the current approach but it was the only multi-OS approach I could get to work.

malfonsoNeoris commented 5 years ago

Hi there... i have found a solution to the OPENCV whitelist, while trying to get videocapture (bebop.sdp) here the forum where i find the solution https://stackoverflow.com/questions/55456870/emgucv-opencv-whitelist-rtp-protocol https://github.com/opencv/opencv/issues/10328

i reproduced in windows and works flawless. More over... i added the enviroment variable (to both general and user). Enviroment add variable OPENCV_FFMPEG_CAPTURE_OPTIONS value

furthermore... if you just do this... you may get some bad quality imgs... so i added protocol_whitelist;file,rtp,udp fflags;nobuffer flag;low_delay

it works.. on ffplay and on videocapture (bebop.sdp) in python

amymcgovern commented 5 years ago

Thanks, I'll try it out!

hsuRush commented 5 years ago

@malfonsoNeoris Hi, I've tried

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = 'protocol_whitelist;file,rtp,udp fflags;nobuffer flag;low_delay'

but this appear. Selection_002 How to fix it?

malfonso0 commented 5 years ago

is that in windows or linux? On windows.. doing that doesnt work.. try to directly add the env variable to windows System ->advanced-<enviroment variables

on ubuntu i didnt tryed

amymcgovern commented 5 years ago

On my Mac, I get a ton of errors go by but then the vision stream pops up. I just ignore all the errors. They are just on startup and sometimes it drops a frame but no streaming vision is perfect.

hsuRush commented 5 years ago

Linux Ubuntu16.04 . It work well if only protocol_whitelist;file,rtp,udp

Cozmo25 commented 4 years ago

On Ubuntu 18.04.1, OpenCV 4.1.2 (built with FFMPEG), Python 3.6.3 I can use:

import os os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;udp"

To open the stream on Anafi (should work with other Parrot drones also).

Note: I couldn't get the stream to open with the protocol_whitelist settings.

Bixiii commented 4 years ago

@malfonsoNeoris How to fix it?

If someone else needs this. I think it is supposed to be:

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = 'protocol_whitelist;file,rtp,udp | fflags;nobuffer | flag;low_delay'

use | between different options

Cesaire-Njietche commented 4 years ago

@Cozmo25 I tried your solution with parrot anafi on ubuntu.

My input stream from the sky controller3 INPUT_STREAM = "rtsp://192.168.53.1/live"

and later in the code

cap = cv2.VideoCapture(INPUT_STREAM, cv2.CAP_FFMPEG)

  while cap.isOpened():
      # Read the next frame
       flag, frame = cap.read()
       cv2.imshow('frame', frame)

I have a very poor image quality. I also have this error:

[h264 @ 0x55d4f67fab00] SPS unavailable in decode_picture_timing [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] SPS unavailable in decode_picture_timing [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] decode_slice_header error [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] decode_slice_header error [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] decode_slice_header error ...

Have you worked around this please?

malfonso0 commented 4 years ago

@Cozmo25 I tried your solution with parrot anafi on ubuntu.

My input stream from the sky controller3 INPUT_STREAM = "rtsp://192.168.53.1/live"

and later in the code
cap = cv2.VideoCapture(INPUT_STREAM, cv2.CAP_FFMPEG)

while cap.isOpened():
    # Read the next frame
     flag, frame = cap.read()
     cv2.imshow('frame', frame)
I have a very poor image quality. I also have this error:

[h264 @ 0x55d4f67fab00] SPS unavailable in decode_picture_timing [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] SPS unavailable in decode_picture_timing [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] decode_slice_header error [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] decode_slice_header error [h264 @ 0x55d4f67fab00] non-existing PPS 0 referenced [h264 @ 0x55d4f67fab00] decode_slice_header error ...

Have you worked around this please?

when you open the videocapure.. a lot of pps error will go out.. buy then you shold have a good image. If you have like "broken" images.. try to change to imutils.webcamvideocapture.. that captures the image on a thread and then you can grab that whenever you want. its quite transparent. for me, that was the solution..

hope this helps

amymcgovern / pyparrot

Is it possible to connect with OpenCV? #102