Open fkaa opened 2 years ago
I have a IP camera that supports B-frames.
Interesting! What's the make/model? I've never seen this before. I'm sure I have .mp4
files around with B frames though so hopefully I can point gstreamer at them to replicate this for testing.
Not sure what the warning about TCP is about
Yeah that seems wrong. It's supposed to say this only for certain live555 versions.
Interesting! What's the make/model?
It's an AXIS Q3536-LVE, but basically any Axis camera with their latest SoC should work.
I've never seen this before.
Same! FWIW it seems like FFmpeg struggles a bit with the stream as well
I can provide a Wireshark capture if you'd be interested, but AFAICT it's just sending video packets in decode order, with the RTP time indicating when it should be presented. It should be identical to how H.264/H.265 is stored in Matroska files.
Good news: I think I can reproduce this behavior with open source software rather than having to buy a camera. That will help with implementation.
.mp4
with B-frames. Big Buck Bunny doesn't use them but you can re-encode it to do so: ffmpeg -i BigBuckBunny.mp4 -codec:v libx264 -preset veryslow $HOME/bbb-reencoded.mp4
ffmpeg -re -stream_loop -1 -i bbb-reencoded.mp4 -c copy -f rtsp rtsp://localhost:8554/mystream
cargo run -p client -- mp4 --url rtsp://localhost:8554/mystream out.mp4
Bad news: I'm confused about (or maybe just disappointed by) how this works:
When writing a .mp4
file, we have to make up the DTS somehow. (For background, see ISO/IEC 14496-12 section 8.6.1.1. There's an example in Table 2. This wiki page has links to a free download of that standard and several others.) I guess we need to basically assume DTS=PTS as we go through, but then as we see an entry that jumps backwards in PTS, we retroactively pull up the DTS for all frames we've already seen with a later PTS so they precede this DTS. In fact at least in theory, that can require bumping up DTSs all the way to the beginning of the file if they're dense, because two frames can't have the same DTS. (In practice, the times advance at 90kHz, and we'd never actually support that framerate, so maybe we can just error out if that happens.)
It's weird if at any point in the stream, you don't know if you've seen all the frames with PTS up to the current RTP timestamp or if more are yet to come. In particular:
.mp4
, you might not actually have a prefix of the stream, but you might be missing B frames. That's surprising.It also just seems to complicate the situation with cameras' flaky timestamps. They sometimes jump back in time due to (incorrectly) basing the RTP time on the wall clock (rather than a monotonic clock as they ought to), and stepping that back when using SNTP. Now when we see that, we have to wonder if it's supposed to jump back due to PTS/DTS differences or if it's the time jump behavior.
I can provide a Wireshark capture if you'd be interested
Yes, please, I think that would be helpful, just in case it is doing something differently than in my ffmpeg scenario. E.g. maybe it conveys the intended DTS via a RTP extension or something.
I believe MPEG-TS conveys both DTS and PTS. (See spec, Table 2-21, PTS_DTS_flag
.) Seems weird if H.264-over-RTP doesn't.
I believe MPEG-TS conveys both DTS and PTS. (See spec, Table 2-21, PTS_DTS_flag.) Seems weird if H.264-over-RTP doesn't.
I think RTP really doesn't. There's an expired draft for conveying the dts via an RTP extension, but afaik no one has implemented it.
Curid's PR #81 adds support for inferring the DTS for H.264 by copying the approach used by gortsplib
. gstreamer also has something for this (h264timestamper
). As mentioned in PR comments, I haven't compared the two yet.
I made a docker B-frame test stream.
docker run -it --network=host curid/test-stream
ffplay rtsp://127.0.0.1:8554/1
The WebRTC example doesn't work for me on any branch, but it may work for you with some tweaking. The mp4 muxer in the client example would need major changes to support B-frames.
I have a IP camera that supports B-frames. Running the client example doesn't seem to support non-monotonic timestamps:
Not sure what the warning about TCP is about