kVideoTexture mode and interlaced h264 media

leonardo204 commented 1 year ago

Hi,

I have a few questions about playing H264 media.

1) When playing in kMediaElement mode, interlaced video doesn't seem to play, how do I get it to play?

2) When I changed to kVideoTexture mode for sw interlaced filtering, it doesn't work even if I put the h264 es packet that I used when playing kMediaElement mode. I checked the behavior for VP8 content. Is there a codec limitation in kVideoTexture mode?

Thanks!

GrandTeaBrewer commented 1 year ago

Hello,

Ad 1. Playing interlaced h264 video with hardware decoding is currently possible only when using kVideoTexture mode. When kMediaElement mode is used WASM Player on Samsung TV internally selects a decoder that has no support for interlaced video. Unfortunately there is no way to affect it's decision at this moment. We'll look into this and enable interlaced video playback in the future for all modes.

Setting decoding to software should enable interlaced video playback in kMediaElement mode, however performance of SW decoder is limited.

Ad 2. Did you manage to get working playback at all (e.g. using progressive video or when using HW decoding)?

There are indeed some limitations on configurations that are supported in various modes, you can test configs using ElementaryMediaTrack::ValidateVideoConfig() (it may not be available on some older devices, though). In general, player capabilities are usually lower in Video Texture mode. Both h264 and VP8 should be playable at 1920x1080 @ 30fps.

Other things that come to mind if you have issues with starting playback:

Is your video stream Annex B?
What interlacing mode is your content using?
What error codes does the player produce when playback fails? Did it initialize properly and fail asynchronously after an attempt to start the playback?

leonardo204 commented 1 year ago

Hi !

First, thank you very much for taking the time to answer our questions. I'd like to provide more detailed test details and results, as well as some additional questions.

You answered my first question, so I guess I can summarize it as follows. 1) interlace stream can only be played in kVideoTexture mode (except for the option to use sw decoder) 2) In kVideoTexture mode, the decoder performance is more limited than in kMediaElement mode. 3) However, it is possible to play h264/vp8 1920x1080 @ 30 fps.

Please check if this is correct, and I would like to ask the following questions.

Q1) kVideoTexture mode can play interlace streams by default. Is this correct? Q2) When using kVideoTexture mode, various callbacks are not called and then stop, why? (See test result below for details) Q3) What should be the ElementaryAudioTrackConfig mimetype for ac3 audio? "audio/ac3; codec=ac3" does not work. Q4) In knormal latency mode, after the first scene is decoded and displayed on the screen, it freezes for a long time before playing or even terminating the program, but there are no errors in the process. I'm doing the appendPacket() faithfully, but it's just freezing and I can't figure out why. Does it matter if it's a network stream? Q5) I am calling appendPacket() faster than 30 times per second at 30fps, is there a problem?

Below I've shared my tests and the streams I did.

Let me know if you need any additional information :)

[ Test Equipment (TV) Information ]

model name: KQ43QB60AFXKR
tizen version: 6.5
Release date: March, 2022.
Other: UHD + 43 inch

[ Test procedure ]

I couldn't find a way to test by reading a large file (mpeg-ts) directly from the TV, so I tested it in the following order.

< Demuxing test on TV > 1) Send ts file from PC to TV by udp or rtp using a tool like tsduck / vlc, etc. 2) Read data from TV to UDP socket at wasm level, demux TS packet to es packet and append packet to wasm player.

< Test for playback on TV only >. 1) Send ts file from PC to localhost by udp or rtp 2) Join localhost on PC to demux TS packet and send es packet to TV. (When demuxing, use FFmpeg's h264_mp4toannexb bsf filter to customize the packet to Annex B format) 3) The TV simply appends the received es packet to the wasm player by calling the

< H264 ES Packet >

Passed to appendPacket() in NAL units of only SPS/PPS/IDR/Non-IDR type.
PTS/DTS starts at 0 for the first packet and increments by the packet duration thereafter.

[ Test Result ]

< kMediaElement mode > 1) knormal latency mode

interlace stream does not play (appendPacket() result fail)
Progressive streams are checked for playback, but most of the time the first frame is shown and then it takes a very long time (more than a minute) to play the next frame (sometimes it freezes and the program crashes).

2) klow mode

interlace stream is not played (appendPacket() result fail)
Checked progressive stream playback

< kVideoTexture mode >.

onCanPlay callback is not called-
After the onCanPlay callback is called, the next callback for Draw() in FillTextureWithNextFrame() is not called.
When callbacks such as above both are not called, the appendPacket() and the entire thread are aborted, causing the program to freeze.

< ElementaryMediaTrackConfig >

Since interlace streams cannot be tested, I used the video/audio config below to test kVideoTexture mode with a progressive stream.


const auto videoTrackConfig = samsung::wasm::ElementaryVideoTrackConfig{
    "video/mp2t; codecs=\"avc1.4D401E\"",       // h264 main profile
    //"video/mp2t; codecs=\"avc1.64001F\"",                // h264 high profile
    std::vector<uint8_t>{},  // codec extradata
    720,
    480,
    30,     // framerateNum
    1,                            // framerateDen
};

    const auto audioTrackConfig = samsung::wasm::ElementaryAudioTrackConfig{
    "audio/mp4; codecs=\"mp4a.40.2\"",                  // mimeType
    {},                                      // sideData
    samsung::wasm::SampleFormat::kF32,            // SampleFormat
    samsung::wasm::ChannelLayout::kStereo,        // ChannelLayout
    0
    };

For config validation, call it like this and get a success return

    auto vret0 = mediaTrack.ValidateVideoConfig(m_videoTrackConfig, samsung::wasm::EmssLatencyMode::kNormal, samsung::wasm::EmssRenderingMode::kVideoTexture);
    auto vret1 = mediaTrack.ValidateVideoConfig(m_videoTrackConfig, samsung::wasm::EmssLatencyMode::kLow, samsung::wasm::EmssRenderingMode::kVideoTexture);
    auto vret2 = mediaTrack.ValidateVideoConfig(m_videoTrackConfig, samsung::wasm::EmssLatencyMode::kNormal, samsung::wasm::EmssRenderingMode::kMediaElement);
    auto vret3 = mediaTrack.ValidateVideoConfig(m_videoTrackConfig, samsung::wasm::EmssLatencyMode::kLow, samsung::wasm::EmssRenderingMode::kMediaElement);

( Note )

$ emcc --version
emcc (Emscripten gcc/clang-like replacement) 1.39.4.7 ((unknown revision))
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

in elementary_media_track.h

  /// Checks if a config can be used to create a valid video track.
  ///
  /// @param[in] video_config A video track configuration.
  /// @param[in] source_mode A playback mode the config will be used in. Some
  /// configurations may be supported only in particular playback modes.
  ///
  /// @return `Result<void>` with `operation_result` field set to
  /// `OperationResult::kSuccess` if the config can be used to create a valid
  /// video track. Otherwise `operation_result` will contain a code describing
  /// a problem with the config.
  ///
  /// @remark
  /// This method is supported only on devices which have
  /// `EmssVersionInfo::has_config_validation` set to `true`.
  static Result<void> ValidateVideoConfig(
      const ElementaryVideoTrackConfig& video_config,
      EmssMode source_mode);

[ Stream Information ]

Here are the streams we used for testing (you can download them by clicking the links)

1) progressive + aac https://drive.google.com/file/d/1aCWE77j6FEBzhVHphd9haZycsPMFh50h/view?usp=share_link

< ffprobe result >
  Duration: 00:10:40.04, start: 1.400000, bitrate: 3083 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
  Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn
  Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 198 kb/s

2) interlace + ac3 https://drive.google.com/file/d/1jeVlRiRFKoJBZbpdCNHiRtHWICaIvB9g/view?usp=share_link

< ffprobe result >
  Duration: 00:05:00.20, start: 75935.112378, bitrate: 13054 kb/s
  Program 155
  Stream #0:0[0xc9]: Data: scte_35
  Stream #0:1[0x613]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, top first), 1920x1080 [SAR 1:1 DAR 16:9], Closed Captions, 29.97 fps, 59.94 tbr, 90k tbn
  Stream #0:2[0x614](kor): Audio: ac3 ([129][0][0][0] / 0x0081), 48000 Hz, stereo, fltp, 448 kb/s
  No Program
  Stream #0:3[0x615]: Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s

GrandTeaBrewer commented 1 year ago

Hello,

thanks for extra information and test streams. Also, your summary is correct! :) kMediaElement mode will play media according to Samsung SmartTV Video Specifications, kVideoTexture puts some restrictions on HW decoder and also induces some OpenGL overhead.

I've tested both streams and managed to play both of them in normal latency mode. Interlaced streams works in Video Texture mode. So I can definitely say the content is playable. Hopefully we can debug the problem, please see answers to questions below.

Yes, kVideoTexture supports interlaced video.
kVideoTexture should behave like kMediaElement in terms of emitted events, with some extra callbacks related to texture delivery added on top of what's emitted in standard mode.
- If events are not delivered to the app, maybe some code blocks main thread? WASM code running in Web environment should avoid blocking the main thread, because it also blocks JS running on the page. WASM Player also uses main JS thread to deliver events, so event handlers execution time should be brief. If the main thread is blocked events will wait in internal event queue. It's best to offload packet append loop and data download to a worker thread.
- Other thing that comes to mind, please make sure the app uses either -s NO_EXIT_RUNTIME=1 linker flag or EM_ASM(noExitRuntime = true); to prevent Emscripten from terminating WASM module when main function exits.
audio/ac3; codecs="ac-3" should be fine. Is config that uses this mime rejected or is there a problem with playback? If it's the latter then maybe it's a problem related to an other issue.
Freeze reason may be connected to what I've described above in the point (2). If program terminates, maybe player is not accepting data and packets buffered in the app makes it go OOM? Please make sure the AppendPacket() operations succeed. Error code should explain the reason most of the time, however I believe two scenarios are possible in your case:
- Player internal buffer is full and packets are no longer accepted. If there is no playback, this is likely. Please note if packets were rejected due to the buffer being full, the app should retry appends as the playback progresses.
- You've mentioned the first frame is rejected when interlaced content is played. Please make sure the first frame appended after WASM Player is opened is marked as a keyframe. Also, normal latency mode will not start playback by default. Please make sure to either set autoplay on the media element or call play. Regarding origin of data, network stream is perfectly fine. Actually, it was developed to be useable alongside sockets - so there shouldn't be any problems here.
Behavior depends on latency mode:
- WASM Player buffers packets when running in EmssLatencyMode::kNormal so packets can be appended at any rate. However, if internal buffer fills up, AppendPacket() will return OperationResult::kAppendBufferFull. Append should be retried after playback progresses. It's generally a good idea to append a fixed amount of packets ahead of the current playback position (e.g. +2 seconds ahead).
- On the other hand, there is no buffering in EmssLatencyMode::kLow (or ::kUltraLow). Player will try to display frames ASAP, so it's up to the app to keep appends according to the stream framerate.

leonardo204 commented 1 year ago

Hello,

Thank you very much to check my answer. For your information, I tested it with "EM_ASM(noExitRuntime = true);” as likes example code.

Could you please possible to share with me how you test it with sample code? If you share sample of code snippets, it will help to solve what I missed because I tested it according to this example codes.

For ac3 audio, I will give you futher information of error messages what I met after I reproduce it for clear.

Thanks again!

GrandTeaBrewer commented 1 year ago

I've tested with the sample code published in this repo, in wasm_player_sample/ and video_decoder_sample/. The only change was to replace packet data and config in sample_data.cc.

For that you'd need to dump stream data of packets and change sample_data.cc accordingly. I believe your demuxing method is correct since you got video frame visible on the screen. Maybe you can also try to either incorporate demuxing code into the demo or replace fixed sample data structures with dynamic ones and (1) fill them with demuxed packets before starting playback and (2) start playback?

leonardo204 commented 1 year ago

I'm glad to hear you can play with sample code of this repository but I'm still face the problem to play my stream.

in emss_sdf_sample.cc:386
386:                    auto append_result = video_track_.AppendPacket(pkt);
387:                    if (!append_result) {
388:                        printf("|zerolive| append video track error \n");

in video_decoder_sdf_sample.cc:101
96: void VideoDecoderTrackDataPump::OnDrawCompleted() {
97:   printf("|zerolive| VideoDecoderTrackDataPump::OnDrawCompleted \n");
98: 
99:    video_track_.RecycleTexture(texture_);
100:    RequestNewVideoTexture();
101:  }

As shown above, even though I'm testing with sample code, I get an error when I call appendPacket(). I have two kinds of problems to play my stream.

1) For interlaced stream, it makes error as same as above. 2) For progressive stream, application is terminated without any errors.

Finally, as I said before, the stream is being tested by demuxing it on the PC using FFmpeg and sending it to the TV on a udp socket. In this way, I checked the normal playback for VP8 stream, and for ts h264 stream, I checked the playback in kMediaElement mode as a progressive stream. The same method in kVideoTexture mode is not working.

Thanks!

GrandTeaBrewer commented 1 year ago

Hm, I think the most likely explanation of an app terminating without an error is the app hitting memory limit. System will terminate apps that use too much memory. Re-reading your test procedure, does entire ts (~235MB or ~467MB) gets transferred to TV and then is demuxed over a short period of time? I believe this can indeed lead to the app hitting memory limit. I've actually played only several first seconds of each clip, because loading entire ts into a source file is not very practical just to check if a content is valid... maybe that's why we get different results. Can you try testing it on smaller data chunks?

Regarding "The append was ignored" error, that's actually normal behaviour when seeking or when app suspends but it's surprising if it happens when the playback starts. Packets are ignored based on the sessionId they are stamped with. The procedure is:

When player starts for the first time, seek completes or app resumes, a track opens (signified by ElementaryMediaTrackListener::OnTrackOpen()). It's assigned a new sessionId before it opens (ElementaryMediaTrackListener::OnSessionIdChanged()).
All packets appended should then be stamped with the new sessionId or kIgnoreSessionId. Session id should be used in multithreaded programs, as ignoring it may lead to a number of subtle problems with synchronizing appends with player operations.
Session lasts until end of stream is appended, a seek starts or app suspends. Appends of packets with session id that has ended will result in OperationResult::kAppendIgnored ("The append was ignored" in JS).

If packets are dropped due to non-current session from the get go, maybe they are stamped with the bad value? You can try changing id kIgnoreSessionId for testing and see if it helps. If it does, please make sure packets are assigned correct session id as reported by ElementaryMediaTrackListener.

leonardo204 commented 1 year ago

Hi,

I tried the test using kIgnoreSessionId as you suggested, and the results were the same. I changed the test method slightly to use the test code as it is, to make sure it works in my environment without me touching the code as much as possible.

I took the first second or two of a packet that I demuxed to Annex B type using FFmpeg's bsf filter on my PC and added it to the test sample code's sample_data.cc and sample_data.h in the same format.

When I created a progressive / intrelaced stream like this respectively and ran it back to the test sample code, it still doesn't work and is in the same state as before.

Could you please test it using the attached sample_data.cc and see what happens? For reference, these are files that were displaying frames normally with ffplay.

Also, I tried changing to kSoftware / kHardwareWithFallback option via decoder option and tested it. Both were slow but playable, and based on the CPU usage, it seems that kHardwareWithFallback also works like kSoftware (sw decoding). In this case, the interlaced stream also plays, but it is not deinterlaced and the representation is as interlaced. (Is there a deinterlace option?)

If I unzip the sample_data.zip, it looks like this . ├── interlaced_h264.cc ├── interlaced_h264.h ├── progressive_h264.cc ├── progressive_h264.h

When testing interlaced_h264, you can do so without including progressive_h264 (and vice versa).

Thank you very much!

leonardo204 commented 1 year ago

I did some more testing with AC3 audio. audio/ac3; codecs="ac-3" -> As you suggested, I put a '-' between the 'ac' and '3' and it works. (Incidentally, I had written it as "ac3" before that, and it didn't work when I did that).

By any chance, have you tried this with the same sample code for the sample data I shared (progressive / interlaced)? I'd be curious to see the results.

Also, looking at the developer page, there's a mention of a PARTNER SDK, and if you become a PARTNER and get the emscripten sdk, you can use the if I become a partner and get the new emscripten sdk? (Is there a partner emscripten sdk?)

GrandTeaBrewer commented 1 year ago

Hi, sorry for taking long time to reply. I'm having some some intense priority project activities recently :). I'm glad to hear you've managed to solve the issue with audio.

I did check your sample data and it actually worked for me. I believe I must check software version that is used on your device. Could you execute samsung::wasm::GetAvailableApis() and let me know what version is associated with "ElementaryMediaStreamSource"?

Regarding Partner SDK, it's not related to WebAssembly development. The only SDK for WASM on Samsung TV is Emscripten SDK extended with platform API support which is publicly available.

For a partner status, I believe you can contact Samsung using https://seller.samsungapps.com/tv/. However I'm more on a technical/development side of things, so I can't really discuss what are the benefits of being a partner (although that's required to publish TV app in Samsung TVs Store AFAIK).

leonardo204 commented 1 year ago

Hi,

I’m ok because I appreciate for your answer Let me share information of samsung::wasm::GetAvailableApis() result.

Code.

    auto ver_info = samsung::wasm::EmssVersionInfo::Create();
    printf("-0x%08x |zerolive| Supports has_emss                      : %d \n", pthread_self(), ver_info.has_emss);
    printf("-0x%08x |zerolive| Supports has_legacy_emss               : %d \n", pthread_self(), ver_info.has_legacy_emss);
    printf("-0x%08x |zerolive| Supports has_video_texture             : %d \n", pthread_self(), ver_info.has_video_texture);
    printf("-0x%08x |zerolive| Supports has_decoding_mode             : %d \n", pthread_self(), ver_info.has_decoding_mode);
    printf("-0x%08x |zerolive| Supports has_low_latency_video_texture : %d \n", pthread_self(), ver_info.has_low_latency_video_texture);
    printf("-0x%08x |zerolive| Supports has_config_validation         : %d \n", pthread_self(), ver_info.has_config_validation);
    auto apis = samsung::wasm::GetAvailableApis();
    for (auto item : apis) {
        if ( item.name == "ElementaryMediaStreamSource") {
            printf(" = ElementaryMediaStreamSource = \n");
            printf("  1) version:  %s \n", item.version.c_str());
            printf("  2) api levels: ");
            for(auto level : item.api_levels) {
                printf("   %d ", level);
            }
            printf("\n");
            printf("  3) features: ");
            for(auto feature : item.features) {
                printf("   %s  ", feature.c_str());
            }
            printf("\n");
        }
    }

Result is

-0x0000ee40 |zerolive| Supports has_emss                      : 1 
 -0x0000ee40 |zerolive| Supports has_legacy_emss               : 0 
 -0x0000ee40 |zerolive| Supports has_video_texture             : 1 
 -0x0000ee40 |zerolive| Supports has_decoding_mode             : 1 
 -0x0000ee40 |zerolive| Supports has_low_latency_video_texture : 1 
 -0x0000ee40 |zerolive| Supports has_config_validation         : 1 
  = ElementaryMediaStreamSource = 
   1) version:  7.fe608e7/7478af5/0639c6b-e6ae208-94e0d91 
   2) api levels:    1    2    3    4    5    6    7 
   3) features:    base-emss     video-texture     software-decoding     construct-with-modes     low-latency-video-texture     ultra-low-latency     config-validation     backend-error

Are you sure play sample data that is provide by me on kVideoTexture mode? If so, I really confused. I also tried to play sample data of mine with sample code of this repository but it can’t.

GrandTeaBrewer commented 1 year ago

Thanks for version info. I was testing on current development software and hardware. I'm in the process of fetching hardware that matches your TV model and I'll check with the provided software version. Hopefully, we can figure out solution to the problem then :).

leonardo204 commented 1 year ago

I appreciate it and look forward to your response! :)

leonardo204 commented 1 year ago

Could you please let me know if you have any updates, I'm waiting for them.

Also, I purchased and received the product below to cross-check with another tizen TV/Monitor, but there is no wired LAN. (I should have checked, but I made a mistake.) Even if I plug in 3 different types of USB LAN, I can't get wired LAN. Is there any way to use wired LAN?

https://www.amazon.it/Samsung-Monitor-3840x2160-Piattaforma-Mirroring/dp/B09TL7VSB1

Thank you.

leonardo204 commented 1 year ago

Hi, Can you share how it's going? Thank you.

SamsungDForum / WebAssemblyDemos

kVideoTexture mode and interlaced h264 media #6