search

marklauter / TelloAPI-SDK-2.0

Flight Controller for Ryze/DJI Tello Drone in C#
MIT License
28 stars 8 forks source link

Video Stream #2

Closed mrcarbook closed 3 months ago

mrcarbook commented 5 years ago

Hi Mark, great work, just a question have you developed the video stream capabilities? Thanks Regards

marklauter commented 5 years ago

I got video working in a UWP app in this repo, but in my opinion it's a kludge and it doesn't get me what I need, which is decoded frames that I can push to an ML subsystem: https://github.com/marklauter/TelloLib

Look at MainPage.xaml.cs here: https://github.com/marklauter/TelloLib/tree/master/src/NetStandard/Tello.Video.UWP

I originally forked from the "aTello" project, but then found out he was using FFMPEG for the video decoding. The UWP media component will accept the h264 data direct from the drone. There's no reason to parse out individual frames like I wasted time doing. The only thing you have to do is tell the decoder the width and height.

My intent is to use OpenCV for decoding and image processing. I will be experimenting as soon as the refactor is complete. Hopefully this weekend, but definitely by next I should have at least some understanding of how this works so I can get it into the project by the end of May 2019.

mrcarbook commented 5 years ago

Hi Mark,

thanks for your reply, i have 2 question;

  1. I don’t understand if now below project is working with Tello ? https://github.com/marklauter/TelloLib/tree/master/src/NetStandard/Tello.Video.UWP
  2. It’s possible to capture frame from above project and use it with Custom Vision for object detection ?

Thanks

Regards

Antimo Musone IT Manager & Architect +39 345 36 36 642

Da: Mark Lauter notifications@github.com Inviato: Wednesday, May 15, 2019 5:48:07 PM A: marklauter/TelloAPI-SDK-2.0 Cc: Antimo Musone; Author Oggetto: Re: [marklauter/TelloAPI-SDK-2.0] Video Stream (#2)

I got video working in a UWP app in this repo, but in my opinion it's a kludge and it doesn't get me what I need, which is decoded frames that I can push to an ML subsystem: https://github.com/marklauter/TelloLib

Look at MainPage.xaml.cs here: https://github.com/marklauter/TelloLib/tree/master/src/NetStandard/Tello.Video.UWP

I originally forked from the "aTello" project, but then found out he was using FFMPEG for the video decoding. The UWP media component will accept the h264 data direct from the drone. There's no reason to parse out individual frames like I wasted time doing. The only thing you have to do is tell the decoder the width and height.

My intent is to use OpenCV for decoding and image processing. I will be experimenting as soon as the refactor is complete. Hopefully this weekend, but definitely by next I should have at least some understanding of how this works so I can get it into the project by the end of May 2019.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/marklauter/TelloAPI-SDK-2.0/issues/2?email_source=notifications&email_token=AALCI5X44CEBOAOCVLZQRRLPVQWDPA5CNFSM4HNARAE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPC5IY#issuecomment-492711587, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AALCI5SY4MZUOFIGQKRFVTTPVQWDPANCNFSM4HNARAEQ.

marklauter commented 5 years ago

Hi Antimo, you might want to edit your last comment as it includes your personal contact information.

  1. That project, or rather the solution (below) should build and run with Tello, though I abandoned the fork and haven't touched it in a while. I don't remember if the full solution builds. https://github.com/marklauter/TelloLib/blob/master/Tello.sln

  2. No, you can't capture a frame, which is why I abandoned the project and started from scratch. I had hoped that the UWP media control would have some useful frame output, but unfortunately it doesn't. It was still a good project for learning how the Tello responds to commands and how the H264 video stream works.

mrcarbook commented 5 years ago

Hi Mark, I have seen in the program.cs of project Tello.SDKV2.Udp.Demo that code: var transceiver = new UdpTransceiver(IPAddress.Parse("192.168.10.1"), 8889); var stateReceiver = new UdpReceiver(8890); var videoReceiver = new UdpReceiver(11111); _tello = new Drone(transceiver, stateReceiver, videoReceiver); _tello.Controller.ConnectionStateChanged += Controller_ConnectionStateChanged; _tello.Controller.ExceptionThrown += Controller_ExceptionThrown; _tello.Controller.ResponseReceived += Controller_ResponseReceived; _tello.StateObserver.StateChanged += StateObserver_StateChanged; _tello.VideoObserver.VideoSampleReady += VideoObserver_VideoSampleReady;

The "Video Sample" Event work?

Thanks

Regards

marklauter commented 5 years ago

I haven't tested it yet, but it should. It's just going to push raw h264 video segments as byte arrays. If I remember correctly there are 8 or 16 segments per frame. I pushed a lot of code yesterday and I should have the UWP app buildable again tonight.

mrcarbook commented 5 years ago

Respect this Monday do you have done some changes about video stream? Regards

Antimo Musone IT Manager & Architect +39 345 36 36 642

Da: Mark Lauter notifications@github.com Inviato: Friday, May 17, 2019 5:44:59 AM A: marklauter/TelloAPI-SDK-2.0 Cc: Antimo Musone; Author Oggetto: Re: [marklauter/TelloAPI-SDK-2.0] Video Stream (#2)

I haven't tested it yet, but it should. It's just going to push raw h264 video segments as byte arrays. If I remember correctly there are 8 or 16 segments per frame. I pushed a lot of code yesterday and I should have the UWP app buildable again tonight.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/marklauter/TelloAPI-SDK-2.0/issues/2?email_source=notifications&email_token=AALCI5SM6SZ5S6BQ4AOT3S3PVYS3XA5CNFSM4HNARAE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVTUV2Y#issuecomment-493308651, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AALCI5WUZ2FZPZSJN6RGHB3PVYS3XANCNFSM4HNARAEQ.

marklauter commented 5 years ago

I've gotten enough done on the rest of the project to focus on OpenCV this weekend, so I should have something for you to play with soon. By the way, the demo program is throwing not implemented exceptions in the video segment received handler. I will change it to write the byte length of the segment to the console.

mrcarbook commented 5 years ago

I try to test it this afternoon. Do you know image parameters for the conversation of bytes array received?

Keep me updated on news.

Thanks

Antimo Musone IT Manager & Arc

From: Mark Lauter notifications@github.com Sent: Friday, May 17, 2019 3:29:10 PM To: marklauter/TelloAPI-SDK-2.0 Cc: Antimo Musone; Author Subject: Re: [marklauter/TelloAPI-SDK-2.0] Video Stream (#2)

I've gotten enough done on the rest of the project to focus on OpenCV this weekend, so I should have something for you to play with soon. By the way, the demo program is throwing not implemented exceptions in the video segment received handler. I will change it to write the byte length of the segment to the console.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/marklauter/TelloAPI-SDK-2.0/issues/2?email_source=notifications&email_token=AALCI5QF6IR54XR3FVILC53PV2XKNA5CNFSM4HNARAE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVUYDYA#issuecomment-493453792, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AALCI5XLTMELTZCE55GAVBTPV2XKNANCNFSM4HNARAEQ.

mrcarbook commented 5 years ago

Hi, i have tested the project Tello.SDKV2.Simulator.Demo and in RunDemo method i have added this: Log.WriteLine("> start video battery"); _tello.Controller.StartVideo(); But I didn't receive any event in the method: private static void VideoObserver_VideoSampleReady(object sender, Events.VideoSampleReadyArgs e) { if (e != null && e.Message != null) { if (e.Message.Data != null) Console.WriteLine(e.Message.Data.Length); } } Regards

marklauter commented 5 years ago

Okay. I'll have a look at it in a few minutes.

marklauter commented 5 years ago

Oh, simulator - it probably isn't sending any video data in the simulator.

marklauter commented 5 years ago

I just pushed a change that will send 12 bytes of fake video data every 1/5 second (5hz). The fake video data is just the string "this is fake video data" encoded as UTF8.

marklauter commented 5 years ago

livedemovideo I just tested Tello video stream with the latest push via the UDP demo and actual drone. It appears to be working.

mrcarbook commented 5 years ago

Have you done some changes about code? I'll test it tomorrow. Do you have also saved some image?

Antimo Musone IT Manager & Arc

From: Mark Lauter notifications@github.com Sent: Friday, May 17, 2019 9:26:51 PM To: marklauter/TelloAPI-SDK-2.0 Cc: Antimo Musone; Author Subject: Re: [marklauter/TelloAPI-SDK-2.0] Video Stream (#2)

[livedemovideo]https://user-images.githubusercontent.com/10211200/57951488-2f09d580-78b8-11e9-8a95-346cb71758f3.png I just tested Tello video stream with the latest push via the UDP demo and actual drone. It appears to be working.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/marklauter/TelloAPI-SDK-2.0/issues/2?email_source=notifications&email_token=AALCI5XH5YA3LUMO2XOLGATPV4BHXA5CNFSM4HNARAE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVU4DI#issuecomment-493571597, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AALCI5X4MHATIOXADZZNIGTPV4BHXANCNFSM4HNARAEQ.

marklauter commented 5 years ago

Yes, I've got the entire solution building with all projects loaded. The UWP application builds and runs, though I haven't tested it thoroughly and there are some weird things in the XAML rendering.

I moved the all the core demo into Tello.Demo. The only code remaining in the two demo Program.cs files is custom instantiation (udp objects for live Tello vs. simulation objects for emulated Tello).

I'm not decoding the video yet. With luck I'll have that done before I crash for the night.

mrcarbook commented 5 years ago

Hi Mark, any update for video decoding? Regards

marklauter commented 5 years ago

No yet. I'm giving Media Foundation a try, but it's poorly documented and experimentation is slow.

marklauter commented 5 years ago

I have video playback working. You can't do anything useful with it as it doesn't generate decoded frames or anything. It just plays in the media element. There's a nice delay because the H264 is a terrible format for real time video and the media element is kinda klunk. But at least there's video output. I'm really frustrated trying to decode the stream into useful frames. I don't have time to work on it tomorrow, but maybe later in the week I can get it working.

marklauter commented 5 years ago

I was able to build the Cisco OpenH264 project. The sample app successfully decoded raw data I saved from a Tello session. I just need to learn how to use the newly created DLL in the Tello project.

https://github.com/cisco/openh264/tree/v1.8.0

mrcarbook commented 5 years ago

Great news, keep me updated

Antimo Musone IT Manager & Arc

From: Mark Lauter notifications@github.com Sent: Tuesday, May 21, 2019 2:09:23 AM To: marklauter/TelloAPI-SDK-2.0 Cc: Antimo Musone; Author Subject: Re: [marklauter/TelloAPI-SDK-2.0] Video Stream (#2)

I was able to build the Cisco OpenH264 project. The sample app successfully decoded raw data I saved from a Tello session. I just need to learn how to use the newly created DLL in the Tello project.

https://github.com/cisco/openh264/tree/v1.8.0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/marklauter/TelloAPI-SDK-2.0/issues/2?email_source=notifications&email_token=AALCI5SMQRINJ7IIS55XSLLPWM4THA5CNFSM4HNARAE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV2MPAQ#issuecomment-494192514, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AALCI5TF5UAPB2OO3LUC4QDPWM4THANCNFSM4HNARAEQ.

mrcarbook commented 5 years ago

Hi Mark , any news? Regards

marklauter commented 5 years ago

not yet. got busy with work.

mrcarbook commented 5 years ago

Hi Mark, any news :) ?

mrcarbook commented 4 years ago

Hi Mark, how are you? I have seen your last updates, do you do progress with video? Regards

marklauter commented 4 years ago

Hi. I started a new job in September and had to take a break from this project. I haven't abandoned it, just dealing with real life for a bit.

sblanchard commented 2 years ago

Hi there have you had any chance to look at this project again or is it still shelved?

marklauter commented 2 years ago

It's not shelved, but it's very low priority as work and life has been in the way. I never made any progress on useful image decoding. If someone points me at a working and not too complex h264 decoder for C# I'll move this project up the priority list.

marklauter commented 3 months ago

Rewrite in progress to celebrate version 3 of the SDK and the apparent discontinuation of the Tello product. New repo here: https://github.com/marklauter/tello.io

marklauter commented 3 months ago

project continuing here: https://github.com/marklauter/tello.io