Access Stream Frames within demo_mambo_user_vision_function

[jebediahmemer]

I am somewhat new to pyparrot and am looking to analyze the streamed frames. In what way can I keep up with the VLC stream and analyze those frames. Basically, what would be the frame variable for cv2.imshow('frame', ????) or cv2.Canny(?????, 5, 5) or how would I access such a thing.

[amymcgovern]

The documentation shows you examples of how to access the images.

https://pyparrot.readthedocs.io/en/latest/vision.html

[jebediahmemer]

I am using the VLC program and have been able to control drone via keyboard and things such as that. However, I have been unable to integrate OpenCV analysis. What class statement/ method do I use to get the latest valid frame, thus returning it and storing in a variable such as img for analysis within the user demo function. I am very new, so if possible please include the command. Thanks so much for your work.

[jebediahmemer]

if (mambo.sensors.flying_state != "emergency"): print("flying state is %s" % mambo.sensors.flying_state) i = 1 while True: img = mamboVision.get_latest_valid_picture() cv2.imshow('frame', img) if (keyboard.is_pressed('w')): mambo.fly_direct(roll=0, pitch=40, yaw=0, vertical_movement=0, duration=0.2) continue if (keyboard.is_pressed('a')): mambo.fly_direct(roll=-40, pitch=0, yaw=0, vertical_movement=0, duration=0.2) continue if (keyboard.is_pressed('d')): mambo.fly_direct(roll=40, pitch=0, yaw=0, vertical_movement=0, duration=0.2) continue if (keyboard.is_pressed('q')): mambo.turn_degrees(-20) continue if (keyboard.is_pressed('e')): mambo.turn_degrees(20) continue if (keyboard.is_pressed('s')): mambo.fly_direct(roll=0, pitch = -40, yaw = 0, vertical_movement=0, duration=0.2) continue if (keyboard.is_pressed('shift')): mambo.fly_direct(roll=0, pitch=0, yaw = 0, vertical_movement = 40, duration=0.2) continue if (keyboard.is_pressed('ctrl')): mambo.fly_direct(roll = 0, pitch=0, yaw = 0, vertical_movement = -40, duration=0.2) continue if (keyboard.is_pressed('esc')): break

        mambo.smart_sleep(5)

    print("landing")
    print("flying state is %s" % mambo.sensors.flying_state)
    mambo.safe_land(5)

This is what I have tried so far.

[jebediahmemer]

Keyboard stuff works fine. It is just the img that turns up empty.

[jebediahmemer]

The cv2.imshow error spits out that the width and height are not greater than 0. When I have tried cv2.imwrite the same way, the image file has been empty. VLC stream is working, so a clear alternative is fullscreening that and simply screengrabbing with OpenCV. However this would cause a huge reduction in performance.

[amymcgovern]

So you are using the DroneGUI approach for your VLC window? I have certainly used opencv to open a new window in this scenario. I did it using multi-threading. The code is too specific to my application to post here but here is a summary:

in main I created a new namedWindow (cv2.namedWindow)

then I set a user callback function to call my image processing stuff (in the background, in a separate thread from the other control, in your case the keyboard controller).

This function then did the grabbing the latest frame (just as you did) and then it processed it and drew stuff in the namedWindow I had created in main. I did that using cv2.imshow

[jebediahmemer]

So, I would just place my function in mamboVision.set_user_callback_function?

[amymcgovern]

yes. Your drawing code but not your keyboard listening code! That should go in the user_code_to_run function

[jebediahmemer]

Roger, had that working good. The keyboard library is quite intuitive for this application.

[jebediahmemer]

main decoder error buffer deadlock when using uservision.save_picture callback function with the addition of cv2.imshow(). I created named window fine. I feel dumb because I am at university and cant get this working.

[jebediahmemer]

Program continues to run but is not displaying the frames to the named window.

[amymcgovern]

can you show your code (or the relevant segments)? I can try to pare mine down to the relevant bits also (it's a LOT of code as this was for a contest we were participating in that used vision and coordinated the flight with vision).

[jebediahmemer]

Printing img before the cv2.imshow lists "none", thus evidencing the fact that self.vision.get_latest_valid_picture() is not working properly.

[jebediahmemer]

class UserVision: def init(self, vision): self.index = 0 self.vision = vision

def save_pictures(self, args):
    # print("in save pictures on image %d " % self.index)

    img = self.vision.get_latest_valid_picture()
    print("This is image")
    print(img)

    if (img is not None):
        filename = "test_image_%06d.png" % self.index
        cv2.imshow('frame', img)
        # uncomment this if you want to write out images every time you get a new one
        #cv2.imwrite(filename, img)
        self.index +=1
        #print(self.index)

[jebediahmemer]

It never reaches the img is not none branch, as image always prints none.

[jebediahmemer]

userVision = UserVision(mamboVision) mamboVision.set_user_callback_function(userVision.save_pictures, user_callback_args=None)

This is literally the example with printing img. This is where you had set the callback function.

[amymcgovern]

So this is without any of the callback and named windows. I don't see that part in your code. So the default example is never getting an image??

[jebediahmemer]

Yes. I create the named window too with cv2.namedWindow('frame'). I use the example code to set the callback function, and it is evidently entering it as it is printing out This is Image \n None.

[jebediahmemer]

VLC stream is displaying as well and keyboard control simultaneously works flawlessly.

[jebediahmemer]

Framed window is opening but is blank of course.

[amymcgovern]

Let me try porting my code over to a simple example to show for the mambo (mine was bebop). I'll post on that tomorrow.

[jebediahmemer]

Ok, thanks. I just need a means of getting that frame, and then I am good because I have quite a bit of experience with OpenCV analysis techniques. Once again thank you so much.

[amymcgovern]

I hope you are not using a Mac. The latest upgrade (to 10.14) seems to have broken the ability to have two GUI windows open at once (SIGH) even using the namedWindow function. However, it should work on other OS's. I only have access to a Mac right now (I actually have parallels installed but the opencv installation wants to be rebuilt to run cv2.namedWindow and I definitely don't have time for that tonight). It does display the window but it only does it at the end of the main thread, which clearly is sub-optimal (and this code worked just fine before my latest upgrade to Mojave).

Let me just check the code into the repository and link to it. it's being funny on formatting in here (parsing it through markdown in odd ways)

[amymcgovern]

Ok, look at:

https://github.com/amymcgovern/pyparrot/blob/master/examples/demoMamboVisionGUITwoWindows.py

[jebediahmemer]

Thanks so much. Pretty similar to what I was trying minus the arg[0], I was trying to reference DroneVisionGui."get frame function".

[jebediahmemer]

Damn, img is still showing as none, and window is not being populated.

[jebediahmemer]

Once again it is concurrently running the second window function while VLC correctly streams. However, img is still none after getting the latest valid image. I have no idea what is going on.

[jebediahmemer]

I am not having the threading issue with it not running till the end. It is simply a matter of getting that frame, which appears to be broken in some way. I am running exactly the same code as you.

[amymcgovern]

This is very strange. I also have no idea what is happening. I was having threading issues (as I mentioned from the Mac) but it was otherwise working.

[jebediahmemer]

Thanks. I will try some more stuff myself. Hopefully we can get this working. I am trying to effectively build an object recognition system on the parrot mambo platform.

[amymcgovern]

One of my MS students just finished his thesis doing gesture recognition in real time on the mambo. So your object idea should work fine!

http://www.mcgovern-fagg.org/idea/theses/tdavis/index.html