H.265 recorded videos cannot be played back, H.264 works ok.

[seanch80]

Hi. I compiled the sample program "FlyleafPlayer (WPF Control) (WPF)" and tested it with streams from my Hikvision IP camera. When using the low latency function it worked amazingly well. This project seems extremely promising for my use.

However, if I record videos from these streams (H.265), the resulting mp4 files cannot be played back.

ffprobe gives this error: [extract_extradata @ 0000017c706d2a00] No start code is found. h.265 record ip camera.mp4: Invalid data found when processing input

ffplay gives this error: [extract_extradata @ 000002439e292d40] No start code is found.0/0 h.265 record ip camera.mp4: could not find codec parameters

ffmpeg gives this error repeatedly a hundred times: [hevc @ 0000026965e01c00] No start code is found. [hevc @ 0000026965e01c00] Error splitting the input into NAL units.

Windows explorer recognizes the vertical and horizontal resolution and frame rate of the file.

Vlc recognizes the codec but playback is all black Codec: MPEG-H Part2/HEVC (H.265) (hev1) Type: Video Video resolution: 2560x1440 Buffer dimensions: 2560x1440 Frame rate: 25.085616 Orientation: Top left

mpv just exists immediately

mpc-hc says "cannot render the file"

built-in Windows app "Movies & TV" just skips to the end of the file immediately

However, if I switch my IP camera to use H.264 instead and record that, then everything works perfectly.

A short sample file (non-working) can be downloaded from here: https://drive.google.com/drive/folders/1LdwVmWe0muQ3JfkqjhGx-MourqJMuHrh?usp=sharing

[SuRGeoNix]

Hi @seanch80, I cannot reproduce this. I never came across with a HEVC (H.265) recording so far.

Can you possible record from another application and send me a sample so I can identify the issue (just make sure the recorded sample will still have the issue)? It's possible that the camera uses different timestamps with HEVC, is the playback normal?

[seanch80]

I didn't quite understand what you meant there. If I record from another application, the sample will not have the same issue. The issue with the unplayable file only occurs when I record from the flyleaf sample program I referred to previously.

I recorded a couple of seconds from my IP camera using ffmpeg.exe (version 4.4), and the resulting mp4 file works well. ffmpeg -rtsp_transport tcp -i "rtsp://....." -c copy working_h265_recorded_with_ffmpeg.mp4

Interestingly, ffmpeg provided the following messages while recording: [mp4 @ 000001aa0be993c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [mp4 @ 000001aa0be993c0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.

Nevertheless the file plays with no issues. I have uploaded it as well to the same google drive folder: https://drive.google.com/drive/folders/1LdwVmWe0muQ3JfkqjhGx-MourqJMuHrh?usp=sharing

I zip'ed the mp4 files this time, to make sure google drive doesn't do any processing on the video files.

As far as the question of playback - yes, playback works great everywhere, also in the flyleaf sample app.

[seanch80]

@SuRGeoNix I tried playing back the ffmpeg.exe recorded file (working_h265_recorded_with_ffmpeg.mp4) I referenced in my previous comment with the flyleaf sample program and then doing a recording of that with that program. The resulting file worked fine. I think maybe that is what you meant to ask me. (?)

You are probably aware of this already, but to clarify: the issue I am having is when I record h.265 from an rtsp stream live from my IP camera using the flyleaf sample program, FlyleafPlayer (WPF Control) (WPF). I right-click and use the "Paste URL" function.

[SuRGeoNix]

OK, so the issue will not appear when you try to record it from the already recorded with ffmpeg (working ... mp4). Which means that happens only from your live stream camera. I'm not sure if you could configure the camera for that to use timestamps, it seems that it does not set them at all. I'm not sure how I can fix this one as I don't have a similar sample to reproduce it. I will try to find one and fix it in the future. Just to clarify, there is nothing to do with the h.265, it's your camera when it uses h.265 codec it will set the timestamps.

[seanch80]

@SuRGeoNix Hi thanks for your reply.

There is no option for timestamps anywhere. Your theory is probably true, yet everything works in flyleaf if I switch to h.264, and both h.264 and h.265 live camera streams can be recorded with ffmpeg.exe directly, no matter if I mux it to mp4, ts or mkv. The camera might use different software for their h.264 and h.265 implementations, not sure.

This probably happens with thousands of ip cameras, not just mine. I am a software developer myself, and sadly dealing with various hw/sw implementations in the wild is something that I've had to bang my head against many times too.

My particular IP camera is a Hikvision DS-2DE5432IW-AE (R7 Series), I believe these cameras were launched in 2019, but my camera was produced in July 2021 and has the newest firmware from late 2021. So it's not very old, outdated or obscure.

I tested older ffmpeg versions, and all versions from ffmpeg 2.4.5 to 5.0 can record these streams fine.

I checked the ffmpeg source code and found this section directly after the code that logs the warning message "Timestamps are unset in a packet for stream %d. ":

if (pkt->pts == AV_NOPTS_VALUE && pkt->dts != AV_NOPTS_VALUE && delay == 0)
    pkt->pts = pkt->dts;

So it seems ffmpeg just uses the DTS as the timing source instead if there is no PTS. I read that flyleaf is based off of ffmpeg, but I am not sure if that particular piece of code is used (ffmpeg\libavformat\mux.c).

When I test with ffmpeg.exe directly I use no other command line options other than -c copy (and -rtsp_transport tcp), so I am not sure why these two pieces of software work differently.

Your software seems very promising and I would love to be able to use it for my app, so I hope we can sort this out. What would perhaps help if I could give you access to the camera, however it is not located in my house and the people that live there was not really "enthusiastic" when I asked them, so I think that's a no-go.

[seanch80]

@SuRGeoNix Quick update. I logged the pts and dts values from each of the streams (h264 and h265), in Demuxer.cs (Demuxer.RunInternal)

TotalBytes += packet->size; // existing line System.IO.File.AppendAllText(@"c:\dev\ptslog.txt", $"pts={packet->pts}, dts={packet->dts}\r\n");

h.264 stream: pts=21510, dts=21510 pts=3600, dts=3600 pts=7200, dts=7200 pts=10710, dts=10710 pts=14310, dts=14310 pts=17910, dts=17910 pts=21510, dts=21510 pts=25110, dts=25110 pts=28710, dts=28710 pts=32310, dts=32310 ... ...

h.265 stream: pts=-9223372036854775808, dts=-9223372036854775808 (numeric value of AV_NOPTS_VALUE constant) pts=3600, dts=3600 pts=7200, dts=7200 pts=10800, dts=10800 pts=14400, dts=14400 pts=17910, dts=17910 pts=21510, dts=21510 pts=25110, dts=25110 pts=28710, dts=28710 pts=32310, dts=32310 pts=35910, dts=35910 pts=39510, dts=39510 pts=43110, dts=43110 ... ...

..so at that point in the code there are in fact pts and dts values in there, but there was a single AV_NOPTS_VALUE in the beginning of the h.265 stream which was not there in the h.264 stream. I tried the hack below, but it had no effect:

                if(packet->pts < 0)
                    packet->pts = 0;
                if (packet->dts < 0)
                    packet->dts = 0;

The recorded h265 file was still unplayable.

Not sure if these dts/pts files are generated by something else, or if I am logging at the correct location. If was interesting to see the difference though, with the single NOPTSVALUE (-9223372036854775808) initially in the h.265 stream which was not seen in the h.264 stream.

Do you have any insights on what to look at tnext? Thanks in advance.

[seanch80]

@SuRGeoNix Last update for today.

I compiled the latest ffmpeg from source, and experimented with the function compute_muxer_pkt_fields() in libavformat\mux.c which is the one responsbile for complaining about no pts value. This function has a lot of workarounds for missing and invalid pts, but I was able to delete all of that code and ffmpeg still produces mp4 files that can be played back.

Also it was interesting to see that in the newly compiled ffmpeg there was no difference between the pts values for h264 and h265 streams from my camera. Here both types of streams had the first packet come in with NOPTSVALUE for pts, but interestingly enough 0 for dts.

newly compiled ffmpeg with no pts/dts kludges:

h264 test: PTS = -9223372036854775808, DTS = 0 [mp4 @ 0x55b6622ff5c0] pts has no value [mp4 @ 0x55b6622ff5c0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file. PTS = 1, DTS = 1 PTS = 3600, DTS = 3600 PTS = 7200, DTS = 7200 PTS = 10800, DTS = 10800 PTS = 14400, DTS = 14400 PTS = 18000, DTS = 18000 ...

h265 test: PTS = -9223372036854775808, DTS = 0 [mp4 @ 0x559d47831b80] pts has no value [mp4 @ 0x559d47831b80] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file. PTS = 1, DTS = 1 PTS = 3600, DTS = 3600 PTS = 7200, DTS = 7200 PTS = 10800, DTS = 10800 PTS = 14400, DTS = 14400 PTS = 18000, DTS = 18000 PTS = 21600, DTS = 21600 ...

.. exactly the same.

I also removed the code that is associated with the warning "Non-monotonous DTS in output stream" in fftools\ffmpeg.c, so no fixups or alterations are made to the pts/dts values if that happens. Still the output files worked fine.

Finally I removed the code associated with the warning "pts has no value" in libavformat/movenc.c, that didn't change anything either. All saved files could be played back with no issues.

So to sum it up, no matter what I try to break in ffmpeg in relations to pts/dts the output files are still playable. I think it is proven now that both h.264 and h.265 streams actually do have timing information in them.

The most glaring difference right now is that ffmpeg gets the same pts/dts values for the first packet on either stream type h.264 or h.265, while in Flyleaf the pts/dts values for the first frame is different for h.264 and h.265. I think this is an important clue to figure out why the flyleaf-saved h264 files are playable whilst the saved h265 files are not.

ffmpeg h264 packet 1: pts = AV_NOPTS_VALUE, dts = 0 ffmpeg h265 packet 1: pts = AV_NOPTS_VALUE, dts = 0 flyleaf h264 packet 1: pts=21510, dts=21510 flyleaf h265 packet 1: pts=AV_NOPTS_VALUE, dts=AV_NOPTS_VALUE

ffmpeg h264 packet 2: pts = 0, dts = 0 ffmpeg h265 packet 2: pts = 0, dts = 0 flyleaf h264 packet 2: pts=3600, dts=3600 flyleaf h265 packet 2: pts=3510, dts=3510

ffmpeg h264 packet 3: pts = 3510, dts = 3510 ffmpeg h265 packet 3: pts = 3600, dts = 3600 flyleaf h264 packet 3: pts=7200, dts=7200 flyleaf h265 packet 3: pts=7110, dts=7110

The ffmpeg version used here was the one with all the pts/dts workarounds/fixes removed from the code, so it should show more or less what comes directly out of the camera.

[SuRGeoNix]

Well @seanch80, that's a great investigation. I didn't read all your details yet and I'm not currently available to investigate as well (especially without being able to reproduce it). I'm not quite sure why you try to change the ffmpeg's source code instead of flyleaf's. Just to clarify, as I understand the issue is during the recording and not during the playback of the recorded video. So, you should check flyleaf's Remuxer when it writes the packets.

Possible there is a bug with the AV_NOPTS_VALUE for dts/pts.

[seanch80]

@SuRGeoNix Well, I did both. The working theory was the my camera didn't send correct timing data, so I tried to break ffmpeg to identify the source code that corrected this (to be able to write playable mp4 files), but I also tried to debug, log, inspect and modify the pts/dts values in FlyLeaf too. I tried a "fake wallclock" approach to produce synthetic pts/dts values based on the system time, and I also tried to make sure no pts/dts values <1 was ever introduced into the packet processing. Nothing I did to FlyLeaf helped.

For a novice in a field, it is much easier to take a sledgehammer and break something that works, rather than to take a scalpel to surgically correct something that doesn't.

Initially I tried a binary search for all ffmpeg versions the last 10 years in the hope of identifying which revision introduced a dts/pts fix for my "corrupt ip camera timing data" (which was actually correct all along). It would have been very easy if say ffmpeg 3.3.1 wrote non-playable files and ffmpeg 3.3.2 wrote correct ones. Then I could just diff the source code to find the "magic" piece of code that repaired the timing data. It was interesting to find that ALL ffmpeg versions wrote working files.

By removing all kludges/sanity checking/hacks/fixes regarding pts/values I came to find that the camera actually outputs valid pts/dts data, and that there is no difference between the h.264 stream and h.265. I believe this is very important, as FlyLeaf actually gets different data for pts/dts when I switch from h.264 to h.265. That leads me to believe there is a bug somewhere in the h.265 handling that introduces the problem, but I have little experience with this type of code, and zero experience with how FlyLeaf works, so I am don't really know how to proceed.

I will take a look at the flyleaf remuxer, but I will generally be looking very closely for code that runs differently and produces different results with h.264 data vs h.265 data. We're already "off course" by the time the data gets into FlyLeaf, so I thought it would be most important to fix that before we get to the writing side of things, although of course the real problem could be present there. If you have any more tips, I would really appreciate it.

[SuRGeoNix]

As the playback works fine you should focus on mux that writes/produces the broken recorded video. I'm pretty sure that there is a 'bug' of those 'non dts/pts' in my remuxer write method which translates the input dts/pts to the output dts/pts values.

[seanch80]

@SuRGeoNix Thanks, I will have a look at that later today.

[seanch80]

@SuRGeoNix I've had a look at the remuxer code. pts and dts values seem fine. The only weird thing I could see was packet->duration=1800 for the h.265 stream when in actuality it was 3600. However, the duration fields in the output file was set to 3600 anyway, so that was probably fine. I also tried to override packet->duration to 3600 when writing the packets, but it made no difference.

Where I found a difference was in the mp4 file codec profile/parameters. In the h.265 mp4 file recorded by flyleaf the following parameters were missing: -Format profile : Main@L5.1@Main -Color space : YUV -Chroma subsampling : 4:2:0 -Bit depth : 8 bits

I've attached a diff screenshot. Obviously there are many fields here that are supposed to be different (file size, stream duration, etc..), but it might be a problem that the full format profile and color/bitrate information is not present.

I tried playing around with the probesize parameter, and waiting a long time before recording and doing longer recordings to allow the ffmpeg format probe to guess the format, but it didn't make a difference. Since it can play everything correctly, I believe the input format probing works well, but there might be a problem transferring all the format parameters to the output stream being recorded. I had a look at how ffmpeg implements "-c copy" streamcopy, but I was unable to implement these functions in flyleaf, as I didn't really know how to.

See the init_output_stream_streamcopy() function from ffmpeg here: https://github.com/FFmpeg/FFmpeg/blob/master/fftools/ffmpeg.c#L3033

EDIT: I also analyzed the mp4 file that is written when flyleaf rerecords an already working ffmpeg-recorded file, and in that instance the parameters I mentioned above are included, so it seems to get all the info when the source is a working file.

[SuRGeoNix]

Just a quick though until I will review all your information above. Did you try the downloader (just the sample directly) to see if it works? They both (recorder and downlaoder) use the remuxer but the recorder might have some issue with the keyframe initially.

[seanch80]

@SuRGeoNix I tried the downloader now. Got the same result. The h265 file cannot be played. back.

The h264 file contains format profile, color space, chroma subsampling, etc. , while the h265 file does not (or it cannot be located in the file).

h264

Video ID : 1 Format : AVC Format/Info : Advanced Video Codec Format profile : Main@L4.2 Format settings, CABAC : Yes Format settings, ReFrames : 1 frame Format settings, GOP : M=1, N=75 Codec ID : avc1 Codec ID/Info : Advanced Video Coding Duration : 15s 535ms Bit rate : 861 Kbps Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode : Variable Frame rate : 25.426 fps Minimum frame rate : 25.000 fps Maximum frame rate : 90 000.000 fps Original frame rate : 25.000 fps Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Scan type : Progressive Bits/(Pixel*Frame) : 0.016 Stream size : 1.59 MiB (100%) Encoded date : UTC 1904-01-01 00:00:00 Tagged date : UTC 1904-01-01 00:00:00 Color range : Full Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709

h265

General Complete name : flyleafDownload_h265.mp4 Format : MPEG-4 Format profile : Base Media Codec ID : isom (isom/iso2/mp41) File size : 10.5 MiB Duration : 16s 712ms Overall bit rate : 5 287 Kbps Encoded date : UTC 1904-01-01 00:00:00 Tagged date : UTC 1904-01-01 00:00:00 Writing application : Lavf58.76.100

Video ID : 1 Format : HEVC Format/Info : High Efficiency Video Coding Codec ID : hev1 Codec ID/Info : High Efficiency Video Coding Duration : 16s 712ms Bit rate : 5 286 Kbps Width : 2 560 pixels Height : 1 440 pixels Display aspect ratio : 16:9 Frame rate : 25.132 fps Bits/(Pixel*Frame) : 0.057 Stream size : 10.5 MiB (100%) Encoded date : UTC 1904-01-01 00:00:00 Tagged date : UTC 1904-01-01 00:00:00

[seanch80]

@SuRGeoNix I have been working on debugging why ffprobe parses one of the files successfully and the other not. I have so far not found the root cause, but I can see HEVC is detected, and width/height information, but it never finds the correct pixel format or color space, and it never starts to parse HEVC nal units VPS, SPS and PPS.

I have read something about an hevc_mp4toannexb filter, maybe that is needed ("start code prefixed mode" instead of "length prefixed mode"). When trying to play the file with ffplay there is an error message "No start code is found". EDIT: What I wrote here is probably false, just thinking aloud. It could simply be caused by ffmpeg just not recognizing the data as an MP4 file and trying to read it as a raw h265 bitstream instead.

I installed MP4Box and did a diff using that too, not sure if it adds any more information, but that tool also is not able to find the format profile and color space info and parsing fails:

[SuRGeoNix]

@seanch80 can you provide me (even with email for private) a camera for testing this? I will try to test it tomorrow if you have one.

[seanch80]

@SuRGeoNix As I mentioned before I don't own the camera, and it's not at my house, and when I asked the people that live there, they were not willing to provide access to the camera to others. I know this is a huge inconvenience, but sadly there is nothing I can do about it right now. I have been trying to look for other public rtsp streams that we can test with.

A while ago I found this post. It looks very similar to what's happening here, however in the bug report it was not working with ffmpeg, and it does in my case. The inputs and protocols used and the resulting output and error codes are identical though.

https://trac.ffmpeg.org/ticket/7974

[seanch80]

@SuRGeoNix Interestingly enough, if I write .ts files with flyleaf instead of .mp4 it works perfectly. string defaultExtenstion = "ts"; "hevc" also works.

It's very weird. It seems with flyleaf now we have the exact same issue that other guy (Levinson) was having in my previous post. I saw he was using ffmpeg 4.1.3, so I downloaded that as well but it worked perfectly so I was not able to reproduce the same issue. Maybe he had compiled ffmpeg with different flags or included modules, not sure. It's a big mystery.

How did you supply the ffmpeg libraries for flyleaf?

At least being able to write .ts files I have a workaround now.

[SuRGeoNix]

Glad you came up with a workaround. Check Flyleaf's Wiki to see how you will parse ffmpeg's libraries. I have compiled and patched them with this patch.

So, is anything that I can fix in Flyleaf for this issue or just an ffmpeg issue? Even for the default extension, I'm using this logic here but I didn't have any issues so far.

Note: Flyleaf does remuxing and not transcoding which means that if you change the format container, the new one must support the original codecs and formats.

Note 2: You can use Engine to enable trace logs so you don't have to add your own. You can change the recording extension by passing an argument to the function (or just the filename with the .ext not really remember) and not need to change it from the library's code manually.

[seanch80]

@SuRGeoNix Thanks for the feedback and info.

Yes, it was good to find that .ts can be used. For my use now that's an acceptable workaround. Most of the recordings will be short and it's no big deal to do a quick remux to .mp4 afterwards.

The root cause of this is a big mystery to me, but I won't have time to check into this any more in the coming weeks. Maybe we can keep this issue on the back-burner for a while. (I don't necessarily mean "github issue", you can close it if you like). I would like to do some work on experimenting with a new set of ffmpeg libraries, find a public RTSP stream we can test with, and do other things to continue to try to find the root cause, and I'm always hoping to wake up some day and find the answer to this on my next google search. :-)

Seemingly there is a intersection of ffmpeg use with live HEVC ip camera streams and the MP4 HEVC file structure (VPS, SPS, PPS NAL units, extradata, order in hvcC box, (?), info per frame or only in the beginning of the stream, etc, ...) that causes this type of problem. I have been trying very hard to reproduce it outside of FlyLeaf to find the smoking gun, but have not been able to. That is something I really don't understand. If ffmpeg is the problem, why does it work when I use ffmpeg only? :-) It's really weird I think, and I don't understand it.

Then finally last night I found someone (Levinson) that actually had seen this exact same problem just with ffmpeg (4.1.3), and even with very similar circumstances I could not reproduce it with that either.

From my comments here it might look like I have spent all my time on ffmpeg, but I have actually done a lot of experiments in FlyLeaf as well, trying to set global header flag earlier, supplying codec extradata, copying codec context using various method, setting codec flags (profile/colorspace, etc..) manually, but nothing I have tried to do there has made any change to the result.

I have used these tools and technologies many times over the years, but I don't really have any expertise on HEVC, MP4, ffmpeg, ffmpeg libraries, etc., so it is difficult to try to troubleshoot a problem in an area were I know very little. I will try to read up on these things as well.

Ok, I will leave it there for now. I might come back here to hassle you some more when I start the real work of integrating flyleaf into my app. :-)

[SuRGeoNix]

Nice, sorry I was not able to follow your detailed analysis but it's not the best time for me currently. By the way, It's possible that ffmpeg's command line tool has a fix but not their API (just trying to figure out why some people have the issue and others do not). Anyways, I will close this and we re-open it in the future if it comes to flyleaf's side.

[SuRGeoNix]

Hi @seanch80, this should be fixed with the latest version. No reason to add the .ts extension anymore. It was an issue with the demuxer and this format that it would not provide the right extension.

[seanch80]

I am not sure if I understand. My issue was with writing mp4 files. When I switched to writing ts files the problem went away. The filenames always had the correct extension. (?) Sorry if I misunderstood your message, I am just very confused right now.

[SuRGeoNix]

@seanch80 My mistake, I still need to find time and check your analyses though.