search

bitwiseworks / qtwebengine-chromium-os2

Port of Chromium and related tools to OS/2
9 stars 2 forks source link

Add audio output support #24

Closed dmik closed 3 years ago

dmik commented 3 years ago

Currently Chromium is completely silent on OS/2 because the audio interface is a no-op for now.

The relative part of code is I believe this one: https://github.com/bitwiseworks/qtwebengine-chromium-os2/blob/026c9bf4c34c37bf7fbc48360ee75da8d56b0d86/chromium/media/audio/os2/audio_manager_os2.cc#L1

In particular, this is needed for YouTube support.

Needed for https://github.com/bitwiseworks/qtwebengine-os2/issues/6.

dmik commented 3 years ago

BTW, we will use https://github.com/komh/kai as it seems to be the easiest way. An RPM with the latest version of this library (2.0.0) is already in exp.

dmik commented 3 years ago

I've implemented audio support using libkai, however, it's not stable here. Using the libkai test cases, in DART mode playback just stops suddenly in the middle. In UNIAUD mode I hear no sound at all. Maybe I need a mixer to enable the playback... I have UNIAUD 2020.05.06 installed here (which appears to be the lates one from ArcaNoae).

dmik commented 3 years ago

Well, system sounds (using DART) don't play complete here as well. And Firefox in UNIAUD mode is choppy too (and either not complete or hangs in DART mode — note that Firefox uses libkai too, see https://github.com/bitwiseworks/mozilla-os2/issues/12). So it's a problem of the UNIAUD driver it seems and I will ignore it for now. Having choppy audio for testing is enough.

dmik commented 3 years ago

Ok, I know why I get silence now. It's our OPUS library build. It is somehow broken and decoding any stream results in silence. Also, the time scale seems to be completely broken when decoding (perhaps, nobody actually tested if OPUS works on OS/2). If I select WAV or MP3 on this HTML5 audio test page http://hpr.dogphilosophy.net/test/, I get sounds on a scale with some delays but much closer to the reality. Delays may be caused by extensive logging I added to debug the problem. I will try to rebuild with it disabled to see how well it performs.

Note that YouTube video player seems to use OPUS by default when it's available and since our Chromium build reports it is, it just uses that and hence no audio and the broken time scale. We need to either disable OPUS somehow (not good) or fix our OPUS build (good).

dryeo commented 3 years ago

On 11/29/20 03:26 PM, Dmitriy Kuminov wrote:

perhaps, nobody actually tested if OPUS works on OS/2

OPUS was working fine, at least the lib. Not sure what may have broke, perhaps just need binary mode set

dmik commented 3 years ago

@dryeo which version are you referring to? I used FFMPEG as a test as follows:

ffmpeg -i gs-16b-2c-44100hz.opus test.wav

where the sample opus file is taken from here (from a page with various samples).

The original opus file contains music (and it plays fine on macOS). However, the resulting wav contains silence — this is exactly what I see in Chromium (a bunch of nan samples in float format, 0x7FFFFFFF in integer representation).

I've tried all 4 RPM versions of Opus we have (including the latest 1.3.1-2) and all behave exactly the same.

Note that FFMPEG itself works like a charm — I tried converting an OGG file from the same page to WAV. The result is playable and contains the same music fragment. I also tried FFPMEG on macOS and there it converts OPUS to WAV just fine. This finally rules out that our OPUS DLL is somehow broken.

dmik commented 3 years ago

I've also tried an older Opus from Hobbes (https://hobbes.nmsu.edu/download/pub/os2/dev/mm/opus-1.1.2.zip) and it is also broken. Generates the same empty WAV which is filled with a sequence of 0x00 0x80 (the WAV file generated on macOS from the very same OPUS sample contains valid samples instead).

dmik commented 3 years ago

JFTR, this is a classical show stopper — I think that something is broken in Chromium and then spend a week of hard debugging of its internals just to discover that some third party component is functioning not right on OS/2 instead. And now I have to fix that third party component (putting any Chromium work on hold). This explains why everything is progressing is not that fast as one would wish to.

I of course can disable OPUS in Chromium as an option but that's not the very best thing to do: OPUS is a de-facto (de-jure?) standard for Web audio (because of its good compression/quality ratio).

But first I stabilize the Chromium part to work well with WAV formats (some work with buffer sizing is needed).

dryeo commented 3 years ago

On 11/30/20 03:04 AM, Dmitriy Kuminov wrote:

@dryeo https://github.com/dryeo which version are you referring to? I used FFMPEG as a test as follows:

|ffmpeg -i gs-16b-2c-44100hz.opus test.wav |

where a sample opus file is taken from here https://dl.espressif.com/dl/audio/gs-16b-2c-44100hz.opus (from a page https://docs.espressif.com/projects/esp-adf/en/latest/design-guide/audio-samples.html with various samples).

This plays fine with the FFplay in netlabs-rel and SeaMonkey (using the same code as Firefox) also plays it fine. Likewise "ffmpeg -i gs-16b-2c-44100hz.opus gs.wav" produces a wav file that plays fine on the desktop here.

dmik commented 3 years ago

@dryeo Could you please tell me exactly which version of OPUS0.DLL are you using? (Including the arch if it's an RPM build, and the full timestamp).

dryeo commented 3 years ago

Opus-1.3.1-2.oc00, i686. 9-06-20 11:29p 288,175 124 a--- opus0.dll

But, I don't see libopus in FFmpeg's configure arguments, so probably using its native opus decoder rather then libopus, which, knowing FFmpeg, is likely mostly used for encoding.

ffmpeg version 2.8.6 Copyright (c) 2000-2016 the FFmpeg developers built with gcc 4.9.2 (GCC) configuration: --prefix='/@unixroot/usr' --bindir='/@unixroot/usr/bin' --datadir='/@unixroot/usr/share/ffmpeg' --incdir='/@unixroot/usr/include/ffmpeg' --libdir='/@unixroot/usr/lib' --mandir='/@unixroot/usr/share/man' --arch=i386 --optflags='-O2 -g -march=i686' --enable-bzlib --disable-crystalhd --disable-indev=jack --enable-libfreetype --enable-fontconfig --enable-libvpx --enable-avfilter --enable-avresample --enable-postproc --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir='/@unixroot/usr/lib' --disable-doc --cpu=i686 --enable-runtime-cpudetect --extra-ldflags=-Zhigh-mem --extra-libs='-lpoll -lmmap' libavutil 54. 31.100 / 54. 31.100 libavcodec 56. 60.100 / 56. 60.100 libavformat 56. 40.101 / 56. 40.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 40.101 / 5. 40.101 libavresample 2. 1. 0 / 2. 1. 0 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 2.101 / 1. 2.101 libpostproc 53. 3.100 / 53. 3.100

dmik commented 3 years ago

Thanks. Yes, the latest RPM build of ffmpeg (4.2.2-2) uses an external opus DLL due to --enable-libopus (didn't they drop its native decoder at all?). Anyway, the actual OPUS0.DLL doesn't seem to be used in your case and it explains why the conversion works for you. As I tried the exact same one you have (but with ffpmeg 4.2.2-2) and it produces the same inaudible result as all the rest (including the Hobbes ones). I'm pretty sure you will see the same with the latest ffmpeg. So it seems that it has always been broken on OS/2. I will create a ticket.

dryeo commented 3 years ago

From the FFmpeg docs, libopus decoder wrapper.

libopus allows libavcodec to decode the Opus Interactive Audio Codec. Requires the presence of the libopus headers and library during configuration. You need to explicitly configure the build with --enable-libopus.

An FFmpeg native decoder for Opus exists, so users can decode Opus without this library.

So I would suggest rebuilding FFmpeg without the --enable-libopus configure option. There's also an experimental Opus encoder, needs -strict experimental (or -strict -2) add to the command line. I probably never actually tested libopus, just the FFmpeg decoder

dmik commented 3 years ago

@dryeo I see. I wonder if rebuilding FFmpeg will help with Chromium. Actually, it links to both FFmpeg and Opus directly (and I see sources using Opus functions directly). So unlikely. Which means we have to fix the Opus library anyway.

dryeo commented 3 years ago

On 11/30/20 12:48 PM, Dmitriy Kuminov wrote:

(didn't they drop its native decoder at all?)

FFmpeg strives to have its own versions of codecs, often only decoding so that there are alternative versions. Most codec authors like this redundancy and encourage it. So the native decoder should stick around.

dmik commented 3 years ago

Hmm. I rebuilt FFmpeg 4.2.2 without --enable-libopus but it didn't help even in ffmpeg.exe case. The native opus decoder still generates "silence" (I renamed OPUS0.DLL to make sure it is not used):

D:>ffmpeg -i gs-16b-2c-44100hz.opus t10.wav
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (GCC)
  configuration: --prefix='/@unixroot/usr/local' --enable-bzlib --disable-crystalhd --enable-libfreetype --enable-fontconfig --enable-libvpx --enable-avfilter --enable-avresample --enable-postproc --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --disable-libopus --enable-gnutls --enable-libvorbis
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, ogg, from 'gs-16b-2c-44100hz.opus':
  Duration: 00:00:15.84, start: 0.000000, bitrate: 111 kb/s
    Stream #0:0: Audio: opus, 48000 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc58.34.100 libopus
      title           : Galway
      artist          : Kevin MacLeod
Stream mapping:
  Stream #0:0 -> #0:0 (opus (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, wav, to 't10.wav':
  Metadata:
    ISFT            : Lavf58.29.100
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
    Metadata:
      artist          : Kevin MacLeod
      title           : Galway
      encoder         : Lavc58.54.100 pcm_s16le
size=    2969kB time=00:00:15.83 bitrate=1536.0kbits/s speed=1.79x
video:0kB audio:2969kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.002566%

Then I discovered that even w/o -c opus on the command line, our RPM FFmpeg build uses the native opus codec anyway. I had to restore the libopus build of FFmpeg and specify -c libopus manually to make it use the external codec from OPUS0.DLL:

D:>ffmpeg -c libopus -i gs-16b-2c-44100hz.opus t10.wav
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (GCC)
  configuration: --prefix='/@unixroot/usr/local' --enable-bzlib --disable-crystalhd --enable-libfreetype --enable-fontconfig --enable-libvpx --enable-avfilter --enable-avresample --enable-postproc --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --enable-libopus --enable-gnutls --enable-libvorbis
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, ogg, from 'gs-16b-2c-44100hz.opus':
  Duration: 00:00:15.84, start: 0.000000, bitrate: 111 kb/s
    Stream #0:0: Audio: opus, 48000 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc58.34.100 libopus
      title           : Galway
      artist          : Kevin MacLeod
Stream mapping:
  Stream #0:0 -> #0:0 (opus (libopus) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, wav, to 't10.wav':
  Metadata:
    ISFT            : Lavf58.29.100
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
    Metadata:
      artist          : Kevin MacLeod
      title           : Galway
      encoder         : Lavc58.54.100 pcm_s16le
size=    2969kB time=00:00:15.83 bitrate=1536.0kbits/s speed=82.9x
video:0kB audio:2969kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.002566%

And you know what, the WAV file generated by libopus is CORRECT. I could successfully play it. Also note that the conversion speed in case of the external libopus codec is 45 times (!!) faster than the native one. Several conclusions:

I will try to disable the native codec first to see if it cures Chromium.

dmik commented 3 years ago

Most codec authors like this redundancy and encourage it.

Really? Where did you get that feeling from? This sounds like a nonsense to me because redundancy increases the maintenance bargain and error probability w/o obvious benefits in this particular case...

dmik commented 3 years ago

Ok, disabling the native opus codec in FFmpeg fixes everything:

  1. FFmpeg uses external libopus (OPUS0.DLL) both by default and with either -c opus or -c libopus and the conversion from OPUS to WAV works flawlessly.
  2. Chromium plays OPUS audio flawlessl now. And yes, this means that YouTube now WORKS too! Flawlessly as well (putting numerous UNIAUD problems aside which don't have anything todo with Chromium or FFmpeg or libkai). The time scale is correct too. The playback in general seems to work better than in Firefox to my taste (and gives less CPU load).
dryeo commented 3 years ago

On 12/01/20 02:58 PM, Dmitriy Kuminov wrote:

Most codec authors like this
redundancy and encourage it.

Really? Where did you get that feeling from? This sounds like a nonsense to me because redundancy increases the maintenance bargain and error probability w/o obvious benefits in this particular case...

I see companies like Google actively encouraging FFmpeg adding VPx decoders for example. The idea is that having more then one code base can find errors, more efficient ways to do stuff and such.

dmik commented 3 years ago

This approach obviously doesn't work in case of Opus. What about Google and VPx, provided that the VPx team covers the encoder/decoder source sync, it's manageable, but still, it requires a 2x effort (and no benefits in terms of error searching as the source is the same). Remember that it's more a question of a distribution (supply the product not only with FFmpeg DLLs but also with codec DLLs if they are built externally), something completely different.

dryeo commented 3 years ago

Well, in the case of Google, they obviously felt it was worthwhile having 2 completely different code bases doing the same thing. In the case of Opus, it was likely just some developer scratching an itch and writing his own version, rather then an effort from Xiph. One of FFmpeg's goals seems to be to support all codecs within its codebase. Though it is so fractured now, with no real leadership that it is hard to say if it has a goal anymore.

dmik commented 3 years ago

@dryeo yep, native opus seems to be completely different than libopus from Xiph. BTW, @SilvanScherrer says that native opus was only added in FFmpeg 3.3, so your version 2.8.6 should not have have it and has to use an external library (which as I found out works perfectly well). Could you please double-check that? I will create an FFmpeg ticket in the mean time.

dmik commented 3 years ago

One important note about libkai/UNIAUD used for audio support. It has two modes — DART (MMOS2) and UNIAUD directly. UNIAUD is used unless set KAI_AUTOMODE=DART is done.

On my Asus K75VJ DART mode doesn't work well, neither in OS/2 itself (e.g. PM sound schemes) nor via libkai (tests, Chromium). The symptoms are the same — it plays a bit of an audio, then MMOS2 (?) seems to stops issuing audio callbacks supposed to feed the hardware with new audio data and nothing is played anymore (a restart makes it play a bit of audio again and then it stops working too). In Chromium, the playback timer also stops running because of that (this is how the audio pipeline is implemented in there — it's fixable per se, but I don't feel like spending time on that given there won't be any sound anyway).

In UNIAUD mode, the callback is called regularly as it should (and no problems with the playback timer in Chromium), but there is some irregularity somewhere resulting in small audio glitches (very short periods of silence) in the played audio. This symptom appears both in libkai demo apps and in Chromium. So it's either a problem of libkai itself or UNIAUD (the latter is more likely). But it's at least surely better than silence and the dead timer in case of DART mode.

If @komh is reading us, perhaps he could comment.

I'm putting the new Qt WebEngine DLL and the fixed avcode58.dll for testing — we need to collect more info how it behaves in both DART mode and UNIAUD mode on various platforms — http://rpm.netlabs.org/test/qtwebtest5_2.7z.

dryeo commented 3 years ago

@dmik, @SilvanScherrer probably looked at when the encoder was added. Googling shows news posts about the encoder being added in release 3.3. Here's the commit that added the decoder in 2014, https://git.videolan.org/?p=ffmpeg.git;a=commit;h=b70d7a4ac72d23f3448f3b08b770fdf5f57de222 I already posted the configure arguments, which does not include libopus. ffmpeg -decoders lists, A....D opus Opus with the A meaning audio and the D meaning decoder. No opus listed in ffmpeg -encoders

dryeo commented 3 years ago

One problem with libkai and UNIAUD mode is that often uniaud gets the mixer wrong and needs to be worked around. Here, I need to use PMUniMix and turn up the volume on the front speakers to hear anything. Something for the readme.

dmik commented 3 years ago

@dryeo re FFmpeg, ok I see. Re UNIAUD, yes, I'm using PMUniMix as well (for the same reason). I wanted but forgot to mention it in readme. Did it now and re-uploaded. BTW, testers report that for well supported hardware DART mode works really well in Chromium (I recall it did so on my Thinkpad T410 as well).

komh commented 3 years ago

@dmik For DART mode, it's well-known problem of Uniaud audio driver. Sometimes an event callback is not called by OS/2 DART system due to unknown reason. Then, as you know, a callback for libkai is not called. As a result, playing stops. For Uniaud mode, it seems buffer underrun occurred. You can confirm debug messages with setting KAI_DEBUG=1 before launching apps.

Anyway, Uniaud audio driver is not stable for both DART and Uniaud APIs, but works better with DART APIs, nowadays.

dmik commented 3 years ago

@komh thanks for your info, I did a KAI_DEBUG run as you describe and it is a buffer underrun indeed (which perfectly matches my own guess). Created a kai ticket for your convenience.

dmik commented 3 years ago

This is considered done — audio playback works quite well provided that UNIAUD supports the hardware or there is a native audio driver for it.