Computer Vision Support

[rohinrohin]

I am looking to buy a parrot mambo fpv for a computer vision project involving person tracking and following and was looking at library support for this. You had mentioned that MamboVision was still under development.

Just wanted to know if there is basic support and what is working with respect to vision. Any suggestions are also welcome! Thanks!

[amymcgovern]

The vision is up and running for the mambo. You can get the live streaming into open cv, frame by frame. I intend to work on blob vision detection but that is lower on my priority list than getting some of the Bebop stuff up and running (specifically Bebop vision). So what is working right now is very basic: you can grab the frames and do whatever you need with them.

[edmondja]

Hello, same for me, I wanted to buy the mambo just because of your API. I don't know it very well yet, but can you confirm me it is easily possible to make a drone moving in any direction we want and capture images with very low latency through the wifi on python 3.6 and OSX please ? I just wanted to make sure that I could reimplement this https://arxiv.org/pdf/1704.05588.pdf with the mambo. Thanks.

[amymcgovern]

Sorry, I forgot to update this. Vision is much happier now! I have the code checked in and I'll get more documentation on it this weekend.

[freaad]

Dear Author Do you think bebop is work with vison? Also do you have any idea to get the imu sensor information inside of Bebop? Thank you.

[amymcgovern]

testing bebop vision is on my agenda this weekend.
I wish I had an idea how to get the internal data at a higher frequency!

[freaad]

I see. I try to find the bebop vision becuase there is "import bebop vision" but I can not find it.... Thank you for answering.

[amymcgovern]

I will check the code in this weekend. I have been focused on ensuring it works on the mambo. But it should be a seamless transition to the bebop.

[amymcgovern]

Good news! Vision is working on both Bebop and Mambo now! Unified into one module in DroneVision.py. I'll work on the documentation shortly. The code is all checked in!

[freaad]

Could you update which version of opencv did you install using anaconda. It could be needed ffmpeg and opencv. I know that there are various version of opencv and very few version support the ffmpeg actuall. Or please give the command for installation. Thank you.

[amymcgovern]

conda list says this: ffmpeg 3.4 h766ddd1_0
opencv 3.3.1 py27h60a5f38_1

[freaad]

Thank you so much

[edmondja]

Can you confirm me the possibility to use the lib for that please ?

[amymcgovern]

edmondja, the library will certainly do the movements and the vision is now being handled through opencv. The latency on the vision is around 0.5s. I am working to get that down but I have other things on the todo list as well. Also, that paper was fascinating and I appreciate you sharing it. One thing to note: they used an AR, which has a nice robust outer shell. The mambo will not appreciate being run into things thousands of times in a row. The propeller guards are useful but nothing like the shell of an AR. I own an AR also and it is more suited to that part of the task. With that said, the mambo is far more agile. A bebop might also work if you 3D printed guards for it. It has a pan/tilt camera, which is also useful. The mambo's points straight ahead always.

[edmondja]

Thank you and you are welcome. I think I will take a mambo to implement a blend of this paper and alphago zero’s paper. Because AR drone doesn’t fit in my appartment.

[freaad]

Do you think that bebop can give better latency? If it is just 0.2s or 0.3s it could be enough.

[amymcgovern]

The latency is introduced by having to bypass opencv and go through ffmpeg first. The solution I was going to tackle (and just haven't had a chance to do so yet) is to try a RAMdisk. I think that will bring the latency down for both drones to around 0.1-0.2s. Given that the mambo's sensors only run at 2Hz, a 0.2s latency would be just fine.

[amymcgovern]

Memory is always much faster than disk. If I can find a way to make the ffmpeg write out to memory instead of disk, then opencv could read directly from memory and it would be much faster. Just haven't looked to see how easy that is to do in python.

[freaad]

Opening ffmpeg ffmpeg -protocol_whitelist "file,rtp,udp" -i C:\Users\dongho\Desktop\marvelmindr\pyparrot-master\utils/bebop.sdp -r 30 image%03d.png & starting vision thread Opening non-blocking readers ffmpeg version 3.4.1 Copyright (c) 2000-2017 the FFmpeg developers

built with gcc 7.2.0 (GCC)

configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-libmfx

libavutil 55. 78.100 / 55. 78.100

libavcodec 57.107.100 / 57.107.100

libavformat 57. 83.100 / 57. 83.100

libavdevice 57. 10.100 / 57. 10.100

libavfilter 6.107.100 / 6.107.100

libswscale 4. 8.100 / 4. 8.100

libswresample 2. 9.100 / 2. 9.100

libpostproc 54. 7.100 / 54. 7.100

[udp @ 000001c552268ea0] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)

[h264 @ 000001c55225b440] SPS unavailable in decode_picture_timing

[h264 @ 000001c55225b440] non-existing PPS 0 referenced

[h264 @ 000001c55225b440] SPS unavailable in decode_picture_timing

[h264 @ 000001c55225b440] non-existing PPS 0 referenced [h264 @ 000001c55225b440] non-existing PPS 0 referenced

[h264 @ 000001c55225b440] decode_slice_header error

[h264 @ 000001c55225b440] non-existing PPS 0 referenced

[h264 @ 000001c55225b440] decode_slice_header error

[h264 @ 000001c55225b440] non-existing PPS 0 referenced

[h264 @ 000001c55225b440] decode_slice_header error

Traceback (most recent call last): File "demoBebopVision.py", line 38, in success = bebopVision.open_video() File "C:\Users\dongho\Desktop\marvelmind_r\pyparrot-master\DroneVision.py", line 128, in open_video line = stderr_reader.readline() File "C:\Users\dongho\Desktop\marvelmind_r\pyparrot-master\utils\NonBlockingStreamReader.py", line 50, in readline timeout = timeout) File "C:\ProgramData\Anaconda3\envs\pyparrot6\lib\queue.py", line 158, in get with self.not_empty: Any idea of this error in the window. I install necessary package and try to run the demovision.py

[freaad]

In the linux it works fine I just need to figure it out how to get each frame of image. Do you have any reason that you do not use the image show function?

[amymcgovern]

Use the VisionServer to watch the vision live

[freaad]

Is it right that save the certain number of image. Also it looks like that img = self.vision.get_latest_valid_picture() This one is the img object that what I have to change as numpy array for the image processing. Could you give any hint?

[amymcgovern]

Yes, get_latest_valid_picture will return the latest image. I had a bug report that it sometimes returns None so you should check for that. I was going to investigate that this weekend and fix but go ahead and check. But the VisionServer is simple and easy if what you want to do is see the vision. You just tell it where your images are saved and then you can watch them live in the browser. If you want to process images, yes, the get_latest_valid_picture returns a single picture. It is an opencv object though, not a numpy array.

[freaad]

Make sense I think that converting numpy array is not the big deal.