search

eerimoq / moblin

Moblin, a free iOS app for IRL streaming.
MIT License
182 stars 20 forks source link

Add RTMP SampleBuffer Processor/Overhaul RTMP #32

Closed ErAzOr2k closed 5 months ago

ErAzOr2k commented 6 months ago

This is a complete overhaul of the RTMP implementation. Incoming sample buffers are now buffered for a certain time.

Benefits:

Tests and feedback is welcome.

ErAzOr2k commented 6 months ago

Unfortunately, DJI Action 4 and Pocket 3 behave completely differently. While Action 4 runs perfectly in sync, the timestamps with the Pocket 3 drift apart. I'm currently running out of ideas on how to solve this :(

This is what Pocket 3 it looks like at the beginning: 05:40:08,004 Int. Video TimeStamp: 1432.116388666 05:40:08,035 RTMP Video TimeStamp: 1432.822444524 05:40:08,038 Int. Video TimeStamp: 1432.149723708 05:40:08,050 RTMP Video TimeStamp: 1432.855777857 05:40:08,070 Int. Video TimeStamp: 1432.18305875 05:40:08,097 RTMP Video TimeStamp: 1432.88911119

few minutes later: 05:43:35,388 Int. Video TimeStamp: 1639.493705375 05:43:35,424 RTMP Video TimeStamp: 1636.922442483 05:43:35,424 Int. Video TimeStamp: 1639.527040375 05:43:35,453 RTMP Video TimeStamp: 1636.955775816 05:43:35,458 Int. Video TimeStamp: 1639.560375458

And this with Action 4: 23:22:04,920 Int. Video TimeStamp: 139.21645325 23:22:04,928 RTMP Video TimeStamp: 139.314406901 23:22:04,948 RTMP Video TimeStamp: 139.347740234 23:22:04,957 Int. Video TimeStamp: 139.249787916 23:22:04,965 RTMP Video TimeStamp: 139.381073567

23:25:08,819 RTMP Video TimeStamp: 323.014405064 23:25:08,833 Int. Video TimeStamp: 323.124154583 23:25:08,840 RTMP Video TimeStamp: 323.047738397 23:25:08,863 RTMP Video TimeStamp: 323.08107173 23:25:08,864 Int. Video TimeStamp: 323.157489416 23:25:08,892 RTMP Video TimeStamp: 323.114405063

Of course I tested with 30fps on both devices...

Do you have other devices to test with?

eerimoq commented 6 months ago

Rumors says RTMP in DJI products is bugged. I noticed it myself when testing with my DJI drone. The Pocket 3 probably outputs frames at a slightly different frame rate, or drops frames occasionally. So incrementing the PTS with 1 / fps for every frame will likely not work for that camera.

It might be a good idea to start a periodic timer with 1 / fps as timeout, and use that to pace it all (and discard/duplicate frames if needed). I mean, if I am not mistaken received frames are processed immediately in this PR, which means the video preview on phone screen will be glitchy if the RTMP connection has hickups. A smallish jitter buffer is probably a good idea to mitigate such problems.

eerimoq commented 6 months ago

Another advantage of adding a timer is that the stream from Moblin might be configured for 60 FPS, but the RTMP server cam only inputs 30 FPS. So one have to duplicate frames to get to 60 FPS. This is only really needed if switching from RTMP cam to builtin cam during a stream.

ErAzOr2k commented 6 months ago

you are a genius. That seems to be the right approach. because now it's much better. It's not quite perfect yet, but I think it can still be optimized. And yes, you're right, the preview will also be much smoother

eerimoq commented 6 months ago

Amazing! I really appreciate you helping out making Moblin better!

eerimoq commented 5 months ago

Does the user have to configure the correct RTMP server stream FPS now? Settings it to 0 will not work? Just asking.

If the configured RTMP server stream FPS is set lower than the real FPS produced by the RTMP publisher, Moblin will run out of memory as sampleBufferQueue in videounit will grow.

ErAzOr2k commented 5 months ago

Does the user have to configure the correct RTMP server stream FPS now? Settings it to 0 will not work? Just asking.

If the configured RTMP server stream FPS is set lower than the real FPS produced by the RTMP publisher, Moblin will run out of memory as sampleBufferQueue in videounit will grow.

yes, at this state of the PRs it is still necessary. This controls the processing speed of the queue. But since we can now do without specifying the fps when calculating the timestamps, I think I will find a solution here too. so that fps no longer needs to be set manually in the future

eerimoq commented 5 months ago

The timestamps received over RTMP from my DJI drone were incorrect last time I tested and to get any sense of timing the excepted FPS has to be configured somehow. Or possibly estimated over time.