No video on UDP 5600

[tr0043t]

In the project description it notes:

View drone video feeds in QGC or forward RTP to an IP address of your choice (currently Mavic Pro 2 only)

Does the Mavic Pro 2 restriction apply to both UDP 5600 video and RTP? So far using the Mini 2 (emulating Air 2) I can see video in the RD app itself, but have been unable to see any video at all on QGC (or any other UDP video app, either on the same device or an external device). Of course I've set the parameters in QGC properly for UDP video streams on 5600, and I do have the telemetry linkage to QGC -- just no video.

Is there something I'm missing or is this simply not supported yet? Thanks.

[m4xw]

I broke that for good when working on the ffmpeg video decoding, but tbh it wasnt properly working prior either (didnt get it working via Mini 2) Someone needs to take a look.

[m4xw]

Checkout the videostream branch, that sacrifices the dji sdk (no in-app video) but stream seems to work. Still needs lots of testing, tested it with qgc on android localhost

[tr0043t]

I have the video functioning both on QGC localhost and on a remote QGC host. The video is significantly laggy but that's to be expected. Great work!

[m4xw]

I suspect we might be doing video playback at half the expected rate

[m4xw]

@laurenweinstein1 Did u have decoding issues on remote QGC?

[tr0043t]

The decoding on remote QGC was fine. The lag is what's noticeable, and that lag seemed to be quite similar on the remote and local QGC (that's just an impression, I'll look at it in more detail in a few days when I have chance to spend more time with it).

Side notes:

I've noticed that video doesn't appear at all (in the app for the devel version, or on the stream for the vs version) unless the UI switch is set to video, not photos. That is, I always need to change that setting (sometimes more than once) to get the video to appear (since it defaults to photo each time). This seems odd, since it means photo mode is not really usable since you can't monitor the video while setting up photo shots.

This may not be practical, but it would be useful if there was a way to switch (even if only at app startup) between the in-app video and streaming video modes.

But yeah, other than the lag (which again is also present on a local QGC), the remote QGC video looked fine.

[m4xw]

Hmm I still get decoding issues with my S21

As for the side notes, yea i am aware. It causes it to send a keyframe (full video frame) instead of a diff and so the video stream can recover. Sadly i cant force it with this SDK to the best of my knowledge. Weirdly enough that behaviour does change once takeoff, i thought maybe its a bw issue on top... got a few other local changes that tweak that behaviour correctly but it made no diff for me. Feel free to test more and report back.

Also technically its very much possible to have both stream and display, issue rn is the usage of singletons so that stuff needs to be split up into the appropriate instances

[m4xw]

also my "solution" rn is just to fly in VLOS ;)

[m4xw]

@laurenweinstein1 please test videostream_2 branch, video should hopefully now work at the same time as GC! However it needs to recreate the surface so if u minimize it goes black. I really think i should just switch video -> cam -> whatever target is or smth before each action to ensure....

[m4xw]

actually merged. please confirm everything works , if not i reopen

[tr0043t]

Videostream merged into develop, correct? I just built a develop from earlier today and was about to test videostream2. Just build/test latest develop now?

[m4xw]

pull it!

[tr0043t]

Will do very shortly!

[m4xw]

2am here soon, any findings?

[tr0043t]

Will build/test sometime over the next few hours (PDT here).

On Mon, Oct 18, 2021 at 4:46 PM m4xw @.***> wrote:

2am here soon, any findings?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/m4xw/rosettadrone_mini2/issues/5#issuecomment-946251965, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC3AUEAZACTJTDHMVDPNYHDUHSWU7ANCNFSM5FL6F56A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[tr0043t]

So, built and initial static tests. Short answer: It essentially "works" -- long answer, a bit more complex:

First, two side notes:

1) Localhost QGC connect to RD often has very slow initial loading of parameters (remote QGC is pretty much always very fast). Occasionally the localhost QGC will load parameters quickly, but usually it takes a minute or more -- on that order of magnitude, anyway. (This is all testing on my tablet, by the way -- my phone would likely be significantly faster in various respects.)

2) Mission uploading appears to work, but I don't currently see how to verify if they're actually in RD (I haven't really played with the simulator much at all yet). If you try to upload again too quickly there will be a mission upload error. Mission downloads (back to QGC) don't work at all (one way I wanted to verify if they were loaded in RD), but I didn't check to see if downloads are even implemented at this point.

More generally:

I was able to get mission uploads from remote QGC to RD and apparently from the localhost QGC as well, in separate tests (pretty much requiring reloading the apps each time).

I was able to get streaming video working to both localhost QGC and remote QGC. Latency seems more or less the same, maybe a tiny bit better. Turning on low latency mode in QGC may be helping a bit also.

I was able to get simultaneous video on both RD and the video stream, with the latter tested both on local QGC and remote QGC. However, switching these modes around generally involved toggling the photo/video toggle between the two positions, and not necessarily in a predictable way. Basically, when the video didn't seem to work I used that toggle to get it going. In some cases when there was trouble, I killed the apps and restarted them. BTW, I was able to get both the RD and QGC localhost app to display the same video simultaneously on the tablet split screen.

So, I wouldn't call it stable, but basically it does seem to work, with the proviso that you have to play around a bit in some cases to get things going, and I am uncertain if missions are actually being saved or not at this time (though as noted QGC claims the mission uploads completed).

Definitely nice to have everything merged now. I hope to be able to do a non-static test later this week. As usual, great work!

[m4xw]

Go try cache_last_iframe branch. Its just a hack and causes decoding issues that seem to fix themselves after a while. Anything else needs far more work. Maybe i can use the djicodecmanager again if i make a extra hook to change codec type hmm Also added some more latency stuff

[tr0043t]

Hi. I'd call this a regression. Very bad tearing and freezing, without any obvious signs of clearing up over time. No other changes.

[m4xw]

Hi. I'd call this a regression. Very bad tearing and freezing, without any obvious signs of clearing up over time. No other changes.

It clears up if the frame contents change. You cant be possibly moving around the menu enough for this to be a issue, lol.

ffmpeg for android ships without encoders so i cant make easy builds with a transcoder for homecompiler

[m4xw]

Cant be a regression if alternative was black screen :P

[tr0043t]

Yeah, that's true! So I moved the camera around a lot to try get it to clear, but it just kept tearing up and freezing. The develop version from last night was far more stable and would be usable.

[m4xw]

Define stable.

[tr0043t]

"Don't answer. It's a trap!" // I'll try take the current develop version out for a test ASAP.

[m4xw]

And I am gonna take a nap :P Fwiw i wrote the code already to fix it, dont have ffmpeg on hand tho with the encoder. I will probably need to move some stuff around or just have it take the raw frame from the ffmpeg instance (its happy and keeps decoding). Just the presenting has issues because the surface gets erased and that kills the codec instance and then requires a keyframe that we dont have. Hence i just went for caching it. Here the artifacts fix itself if the whole frame changes. Anyway there should be no other downside. It would always break when opening the map or minimizing, now you see a "change". At least on my end. As for the initial action requirement to get the decoding starting, shrug, no idea. DJI samples just set photo everytime and i guess that does the job on startup? Who knows. Could be mini 2 shenanigans as well.

[tr0043t]

So long as the video is relatively "stable", most of those other issues are pretty much de minimis. This already goes so far beyond what else is out there. More actions for the waypoints would be fun -- rotate would get my vote -- but really this is doing excellent.

[m4xw]

I will get to each possible action after nailing the basics

[m4xw]

Tbh i am more frustrated the tools doesnt allow us to plan custom actions! We will need to define a order tho. If a waypoint has a heading, is it when moving to it, or when getting there. what with rotate action, is it before goto starts or when at the target and stuff.. I will ask a bud if he can double check behaviour on the drones that actually support it

[m4xw]

Also note: The "basic stability" with the video feed when doing nothing should not have been impacted and was likely general rng

[tr0043t]

I'll look at this again in better light. Low light is probably making this more difficult.

[m4xw]

@laurenweinstein1 yo check "change_outputsurface" branch. I think i got it! Needs a bunch of cleanup and i need to doublecheck for mem leaks. lmk how it goes for u

[m4xw]

I think i can squeeze out a bit more latency wise. Localhost qgc seems half the latency. fwiw I dont have a fix for the initial keyframe missing when qgc connects, nothing i can do on my part for now.. Other than recording thousands of packets and keep them in mem

[m4xw]

FWiw there still seems to be one condition that may the stream bug out. Need to find it.

[m4xw]

Ok i think sometimes its at a wrong state (seems to happen mostly when streaming, maybe some delay?) Should be able to add checks for that later / tomorrow

[m4xw]

yea seems stable to me / condition doesnt trigger if qgc isnt on hm

[m4xw]

Pushed some more improvements to the branch. Actually seems to still shit itself, but only when flying. I really wonder if delays cause timing issue & drops for the decoder hmmm

[m4xw]

Made some more progress, theres quite a few bitrate drops, probably need to change how cmd's are pushed a bit. I got it decently stable now tho, give f7849281d4f88f6db07ef70534eb5acc9695f9cd a shot

[m4xw]

Pushed some more stability improvements. Hope weather clears tomorrow for a fieldtest. There were some reliability issue on startup but i cant repro them right now anymore and a 30min mission passed the video stream test!

[tr0043t]

Lots of trouble with this one. Most of the time no video at all, not on the app and not on local or remote streams. I got the video going a couple of times after lots of restarts of the app -- toggling the photo/video switch didn't seem to make any difference. On those times I got the video, it looked pretty good on the app (very little tearing and only the usual latency) but was freezing up and dropping out completely on QGC both local and remote.

[tr0043t]

From my standpoint right now the best build was the most recent develop branch.

[m4xw]

Did u try the last commit? Works absolutely reliable now here

[m4xw]

Also with the latest commit (sry thought it was clear when i said "pushed more stability" stuff), are issues are occuring on ground/sim or just in air? Motor interference can cause a drop in bitrate / so does distance. Adjusting it on the fly causes stutters tho (updates too much), so might need to change it if it changes by 25% compared to last value or smth. Also if you still have issues, go to video mode and make sure to not select 4k30 (my samsung device just hates it, doesnt work in dji fly either), that even applies when photo mode is on.

[tr0043t]

Yes, this was testing of change_outputsurface less than 12 hours ago (and I see right now your last commit was well before that). The video just was basically not working at all. I'm unsure what video setting you're referring to. I haven't changed any of them. These are all static tests on the bench. Thanks.

[m4xw]

Confirm with my build https://m4xw.net/nextcloud/index.php/s/tDa5BxjzmCGD2rL

As for the settings, whatever u last used in dji fly applies

Which device did u test on?

[tr0043t]

I'll test your build asap. All testing currently on my Samsung tablet (S6 Lite).

[tr0043t]

Yeah, I've never touched the video setting in dji fly.

[m4xw]

I'll test your build asap. All testing currently on my Samsung tablet (S6 Lite).

I can test against my S6 edge plus if needed.

Yeah, I've never touched the video setting in dji fly.

Confirm whats used then.

I def. had startup issues yesterday but those are fixed on my end now and basically no cutoffs.

You could maybe try to 5x tap the icon on the connect screen and then click open (might need to do twice, its the "test mode") and then connect when the main screen is already open

[m4xw]

I can only repro absolutely no video feed if 4k30 is selected in the video mode (i use 2.7k30 currently) Thats a issue with the samsung hw codec and dji's sdk transcoder. earlier vers skipped it but then i have no chance to fix the keyframe issue..... another possibility would be some GLES/EGL issue, in which case i need a Run log

[tr0043t]

I'll check but I'm almost certain it's just 1080.

[m4xw]

i just did a field test, theres def a bitrate issue when going fast Was pretty stable otherwise tho

[tr0043t]

I'm pulling down your apk now.