search

NewChromantics / PopH264

Low-level, minimal H264 decoder & encoder library for windows, hololens/uwp, ios/tvos/macos, linux, android/quest/magic leap. CAPI for use with c#, unreal, swift
http://poph264.com/
Mozilla Public License 2.0
84 stars 15 forks source link

Support for profiles above baseline #22

Open nick-cr opened 3 years ago

nick-cr commented 3 years ago

I am currently working on integrating PopH264 into a Unity project.

Loading Main Level 5 file thow's an error in Mp4.cs PopH264 currently only supports baseline profile. This is Main level=5. The video does not play.

In Windows I would expect Poph264 to be using MediaFoundation which works with Main Level 5.

Is this limitation because of the limitations of eg broadway? If this is the case I would expect it to warn but still work.

SoylentGraham commented 3 years ago

Yeah it's a warning for broadway, MediaFoundation should still work, but I do have someone using it where it doesn't output anything (baseline 3), but no errors (unity's log file will record STDERR from the plugin which doesn't show up in the console)

Do you have an nvidia card? (it should select any compatible codec, but I feel like there's one that doesn't quite respond)

Can you have a look at the log file (from top right icon of the console) and paste it here (do a quick ctrl-f to see if there's any mention of media foundation)

When MediaFoundation fails to decode packets (like, out of order packets), it doesn't error, just doesn't output any frames

SoylentGraham commented 3 years ago

In fact, if you have a sample encoding that doesn't work as an example that would be great. (I'm moving out of my home for a while and without a PC to test on, but might be able to find a way)

nick-cr commented 3 years ago

The video file I am trying to use is this one:

https://user-images.githubusercontent.com/68970441/104579992-9f03cc80-5654-11eb-97d5-5459c240a090.mp4

It was encoded using ffmpeg with the nvenc extensions built in.

From the log this is the only part which mentions media foundation:

XInput1_3.dll not found. Trying XInput9_1_0.dll instead...
<RI> Input initialized.

D3D11 device created for Microsoft Media Foundation video decoding.
[Project] Loading completed in 105.565 seconds
    Project init time:              0.688 seconds
        Template init time:         0.000 seconds
        Package Manager init time:      0.000 seconds
        Asset Database init time:       0.000 seconds
        Global illumination init time:  0.000 seconds
        Assemblies load time:           0.000 seconds
        Unity extensions init time:     0.001 seconds
        Asset Database refresh time:    0.000 seconds
    Scene opening time:             2.455 seconds
SoylentGraham commented 3 years ago

I forgot, the debug is hidden unless Verbose debug is enabled

This isn't in the latest package, but it's a one line addition, can you add VerboseDebug and enable it, and then you might see some MediaFoundation stuff (at the very least it should least all compatible adaptors/codecs) https://github.com/NewChromantics/PopH264/blob/master/Unity/PopH264/Assets/PopH264/PopH264.cs#L122

SoylentGraham commented 3 years ago

I'm assuming this doesn't play with broadway (if you set the decoder name to Broadway) - sometimes encoding won't actually use features broadway doesnt support and will continue (and sometimes will decode resolutions above waht it's supposed to support) This variable should be set to Broadway https://github.com/NewChromantics/PopH264/blob/master/Unity/PopH264/Assets/PopH264/PopH264.cs#L120

Just in case it does decode and it will confirm its a MediaFoundation bug

SoylentGraham commented 3 years ago

This video is now decoding in MediaFoundation. Need to make it clear which platforms support which profiles... automatically. #32

SoylentGraham commented 3 years ago

This video currently not decoding in ios.

nick-cr commented 3 years ago

@SoylentGraham did you confirm this on your own device or are you going on my report?

SoylentGraham commented 3 years ago

I've confirmed it now on my SE.(Testing after skipping SEI packets) I get the keyframe decoding, then it errors with baddata

SoylentGraham commented 3 years ago

Just double checking same error on mac. VideoToolbox is really strict, data on the end of a nalu will give bad data

SoylentGraham commented 3 years ago

After converting mp4 to annexb ffmpeg -i test.flv -vcodec copy -an -bsf:v h264_mp4toannexb test.h264 and running it through test app, it decoded 16 frames on mac.

SoylentGraham commented 3 years ago

Also worked on ios!

Maybe a bug in the mp4 c# code

SoylentGraham commented 3 years ago

Test app ios output with working main5 h264 stream

Popped Nalu SupplimentalEnhancementInformation x15bytes
Popped Nalu SupplimentalEnhancementInformation x11bytes
Popped Nalu SequenceParameterSet x38bytes
Popped Nalu PictureParameterSet x8bytes
Popped Nalu Slice_CodedIDRPicture x162288bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x106138bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x78331bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x68782bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x63235bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x60337bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x52333bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x50118bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x50716bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x49969bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x44499bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x45220bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x43533bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x43212bytes
Popped Nalu SupplimentalEnhancementInformation x12bytes
Popped Nalu Slice_NonIDRPicture x45636bytes
Popped Nalu EndOfStream x5bytes
SoylentGraham commented 3 years ago

Err, now it's working. I did have Decode SEI checked (from previous testing) I think maybe it's fine since SEI skipping 699bf972b4cee0f6d5466a0a96f66f29875bfe04

@nick-cr try 1.3.30

SoylentGraham commented 3 years ago

I think this is all working now