Sound playback tracking issue

basilgello commented 3 years ago

Testing the following setup:

Firefox -> Xpra server with private PulseAudio server -> GTK3 client

with various speaker codecs:

$ xpra attach --speaker=on --speaker-codec=help :1
speaker codecs available: vorbis+mka, vorbis, flac, flac+ogg, wav+lz4, wav+lzo, wav, wavpack, speex+ogg, mp3, aac+mpeg4, opus+ogg, opus, opus+mka

Attaching to Xpra server with every possible codec option gives:

vorbis+mka (aka default): choppy sound, often overruns
vorbis: no sound at all, sound level on graph is zero
flac: smooth sound playback
flag+ogg: quieter than plain flac, minor underruns
wav+lz4: smooth sound playback
wav+lzo: quieter than wav+lz4, minor underruns
wav: smooth sound playback
wavpack: smooth sound playback
speex+ogg: low-quality sound, but without chops
mp3: medium quality sound with minor overruns
aac+mpeg4: no sound at all (!!!)
opus+ogg: low quality sound, with noticeable underruns
opus: smooth sound playback
opus+mka: low quality sound, with noticeable underruns

basilgello commented 3 years ago

Firefox running HTML5 client:

aac+mpeg4: clicking 'unmute` button on Xpra HTML5 client pane spews an error:

Invalid audio codec 'unknown' (expected aac+mpeg4), stopping audio stream Utilities.js:1:521
cerror https://localhost:14500/js/Utilities.js:1
cerror https://localhost:14500/js/Client.js:1
error https://localhost:14500/js/Client.js:1
_process_sound_data https://localhost:14500/js/Client.js:1
_route_packet https://localhost:14500/js/Client.js:1
open https://localhost:14500/js/Protocol.js:1
(Async: EventListener.handleEvent)
open https://localhost:14500/js/Protocol.js:1
open_protocol https://localhost:14500/js/Client.js:1
_do_connect https://localhost:14500/js/Client.js:1
connect https://localhost:14500/js/Client.js:1
(Async: EventListener.handleEvent)
connect https://localhost:14500/js/Client.js:1
init_page https://localhost:14500/index.html:1222
onload https://localhost:14500/index.html:1299
(Async: EventHandlerNonNull)
<anonymous> https://localhost:14500/index.html:1291
jQuery 13

while server log shows Gstreamer lacks compliance property:

2021-08-09 09:53:49,062 audio capture Error setting up the sound pipeline:
2021-08-09 09:53:49,063 audio capture  gst_parse_error: no compliance «compliance» property in element «avenc_aac0» (2)
2021-08-09 09:53:49,063 audio capture  GStreamer pipeline for aac+mpeg4:
2021-08-09 09:53:49,063 audio capture   pulsesrc device="Xpra-Speaker.monitor" name="src" ! \
2021-08-09 09:53:49,063 audio capture   queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=50000000000000 leaky=2 ! \
2021-08-09 09:53:49,063 audio capture   volume name=volume volume=1.0 ! \
2021-08-09 09:53:49,063 audio capture   avenc_aac compliance=1 perfect-timestamp=1 ! \
2021-08-09 09:53:49,063 audio capture   mp4mux faststart=1 streamable=1 fragment-duration=20 presentation-time=0 ! \
2021-08-09 09:53:49,063 audio capture   appsink name=sink emit-signals=true max-buffers=10 drop=true sync=false async=false qos=false```

mpeg4+mp3: the following HTML5 client error

audio loadstart MediaSourceUtil.js:1:4849
audio source open: open MediaSourceUtil.js:1:4390
audio media source open Utilities.js:1:648
using audio codec string for mp3+mpeg4: audio/mp4; codecs="mp3" Utilities.js:1:648
audio: requesting mp3+mpeg4 stream from the server Utilities.js:1:648
broadway decoder initialized Utilities.js:1:648
audio-state: playing Utilities.js:1:648
audio loadedmetadata MediaSourceUtil.js:1:4849
Media resource blob:https://localhost:14500/155def2c-320c-47e8-aba5-d7f93d0b5dff could not be decoded. index.html
audio error MediaSourceUtil.js:1:4849

with Xpra server error:

2021-08-09 10:01:02,447 using pulseaudio device:
2021-08-09 10:01:02,447  'Monitor of Xpra Speaker'
2021-08-09 10:01:02,801 Error playing new-stream bell sound:
2021-08-09 10:01:02,801  [Errno 2] Немає такого файла або каталогу: 'gst-launch-1.0'
2021-08-09 10:01:03,139 client   2 broadway decoder initialized
2021-08-09 10:01:03,140 client   2 audio-state: playing
2021-08-09 10:01:03,141 client   2 audio error: [object MediaError]
2021-08-09 10:01:03,142 client   2 DECODE: error occurred when decoding
2021-08-09 10:01:03,146 client   2 audio: stopping stream
2021-08-09 10:01:03,146 client   2 close_audio_mediasource: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], audio=[object HTMLAudioElement]
2021-08-09 10:01:03,146 client   2 audio-state: stopped
2021-08-09 10:01:03,151 audio capture stopping
2021-08-09 10:01:03,426 audio capture using 'mp3' audio codec
2021-08-09 10:01:03,658 client   2 received end-of-stream from server

legacy wav: the following HTML5 client error:

Array [ "sound data error" ]

0: "sound data error"

length: 1

<prototype>: Array []
Utilities.js:1:521
cerror https://localhost:14500/js/Utilities.js:1
cerror https://localhost:14500/js/Client.js:1
exc https://localhost:14500/js/Client.js:1
_process_sound_data https://localhost:14500/js/Client.js:1
_route_packet https://localhost:14500/js/Client.js:1
open https://localhost:14500/js/Protocol.js:1
(Async: EventListener.handleEvent)
open https://localhost:14500/js/Protocol.js:1
open_protocol https://localhost:14500/js/Client.js:1
_do_connect https://localhost:14500/js/Client.js:1
connect https://localhost:14500/js/Client.js:1
(Async: EventListener.handleEvent)
connect https://localhost:14500/js/Client.js:1
init_page https://localhost:14500/index.html:1222
onload https://localhost:14500/index.html:1299
(Async: EventHandlerNonNull)
<anonymous> https://localhost:14500/index.html:1291
jQuery 13

and Xpra server error:

2021-08-09 10:04:29,976 Error playing new-stream bell sound:
2021-08-09 10:04:29,977  [Errno 2] Немає такого файла або каталогу: 'gst-launch-1.0'
2021-08-09 10:04:29,995 client   3 audio-state: playing
2021-08-09 10:04:29,999 client   3 audio-state: error
2021-08-09 10:04:29,999 client   3 AVBuffer@https://localhost:14500/js/lib/aurora/aurora.js:1:12735
2021-08-09 10:04:29,999 client   3 XpraSource.prototype._on_data@https://localhost:14500/js/lib/aurora/aurora-xpra.js:1:531
2021-08-09 10:04:29,999 client   3 XpraClient.prototype.push_audio_buffer@https://localhost:14500/js/Client.js:1:73974
2021-08-09 10:04:29,999 client   3 XpraClient.prototype.add_sound_data@https://localhost:14500/js/Client.js:1:72013
2021-08-09 10:04:29,999 client   3 XpraClient.prototype._process_sound_data@https://localhost:14500/js/Client.js:1:70573
2021-08-09 10:04:29,999 client   3 XpraClient.prototype._route_packet@https://localhost:14500/js/Client.js:1:11200
2021-08-09 10:04:29,999 client   3 XpraProtocolWorkerHost.prototype.open/<@https://localhost:14500/js/Protocol.js:1:733
2021-08-09 10:04:30,000 client   3 EventListener.handleEvent*XpraProtocolWorkerHost.prototype.open@https://localhost:14500/js/Protocol.js:1:582
2021-08-09 10:04:30,000 client   3 XpraClient.prototype.open_protocol@https://localhost:14500/js/Client.js:1:9540
2021-08-09 10:04:30,000 client   3 XpraClient.prototype._do_connect@https://localhost:14500/js/Client.js:1:9222
2021-08-09 10:04:30,000 client   3 XpraClient.prototype.connect/<@https://localhost:14500/js/Client.js:1:8703
2021-08-09 10:04:30,000 client   3 EventListener.handleEvent*XpraClient.prototype.connect@https://localhost:14500/js/Client.js:1:8587
2021-08-09 10:04:30,000 client   3 init_page@https://localhost:14500/index.html:1222:12
2021-08-09 10:04:30,000 client   3 xhr.onload@https://localhost:14500/index.html:1299:6
2021-08-09 10:04:30,000 client   3 EventHandlerNonNull*@https://localhost:14500/index.html:1291:5
2021-08-09 10:04:30,000 client   3 mightThrow@https://localhost:14500/js/lib/jquery.js:3760:29
2021-08-09 10:04:30,000 client   3 resolve/</process<@https://localhost:14500/js/lib/jquery.js:3828:12
2021-08-09 10:04:30,000 client   3 setTimeout handler*resolve/<@https://localhost:14500/js/lib/jquery.js:3866:16
2021-08-09 10:04:30,000 client   3 fire@https://localhost:14500/js/lib/jquery.js:3494:31
2021-08-09 10:04:30,000 client   3 fireWith@https://localhost:14500/js/lib/jquery.js:3624:7
2021-08-09 10:04:30,000 client   3 fire@https://localhost:14500/js/lib/jquery.js:3632:10
2021-08-09 10:04:30,000 client   3 fire@https://localhost:14500/js/lib/jquery.js:3494:31
2021-08-09 10:04:30,000 client   3 fireWith@https://localhost:14500/js/lib/jquery.js:3624:7
2021-08-09 10:04:30,000 client   3 ready@https://localhost:14500/js/lib/jquery.js:4104:13
2021-08-09 10:04:30,000 client   3 completed@https://localhost:14500/js/lib/jquery.js:4114:9
2021-08-09 10:04:30,001 client   3 EventListener.handleEvent*@https://localhost:14500/js/lib/jquery.js:4130:11
2021-08-09 10:04:30,001 client   3 @https://localhost:14500/js/lib/jquery.js:34:10
2021-08-09 10:04:30,001 client   3 @https://localhost:14500/js/lib/jquery.js:38:4
2021-08-09 10:04:30,001 client   3 audio: stopping stream
2021-08-09 10:04:30,001 client   3 audio-state: stopped
2021-08-09 10:04:30,005 audio capture stopping
2021-08-09 10:04:30,007 client   3 received end-of-stream from server
2021-08-09 10:04:30,008 client   3 audio: stopping stream
2021-08-09 10:04:30,452 audio capture using 'wav' audio codec

http stream: mp3: the unmute button stays but the HTML5 client spews error:


GEThttps://localhost:14500/audio.mp3?uuid=5cd196ec-7035-afff-6581-5ff2313b22aa
[HTTP/1.0 404 Path not found 48ms]

HTTP load failed with status 404. Load of media resource https://localhost:14500/audio.mp3?uuid=5cd196ec-7035-afff-6581-5ff2313b22aa failed. index.html All candidate resources failed to load. Media load paused.

while nothing criminal in Xpra server log:

2021-08-09 10:07:16,427 client 4 speaker icon clicked, audio_enabled= true 2021-08-09 10:07:16,427 client 4 audio-state: waiting 2021-08-09 10:07:16,427 client 4 starting http stream from https://localhost:14500/audio.mp3?uuid=5cd196ec-7035-afff-6581-5ff2313b22aa

basilgello commented 3 years ago

Dropping this line:

https://github.com/Xpra-org/xpra/blob/master/xpra/sound/gstreamer_util.py#L159

fixes aac+mpeg4 for me. Sound quality is definitely not the best but no underruns on GTK3 client.

totaam commented 3 years ago

opus : smooth sound playback

As per https://github.com/Xpra-org/xpra/blob/8d5ba7a8b8f8937dcdc3df5274aa3f2e6035ca9f/xpra/sound/gstreamer_util.py#L211-L216 opus should be the default if all the dependencies are installed correctly. I have re-ordered the list based on your findings - since I am unable to test properly "thanks" to pipewire.

I do get opus automagically when I just start the client:

audio playback pipeline elements=['appsrc name=src emit-signals=0 \
    block=0 is-live=0 stream-type=0 format=4', \
    'oggdemux', 'opusdec', 'audioconvert', 'audioresample', \
    'queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=450000000 leaky=2', \
    'volume name=volume volume=0', \
    'pulsesink sync=False async=True qos=True client-name="Xpra" name="sink"']

But you're saying that:

vorbis+mka (aka default): choppy sound, often overruns

So your package is broken somehow. The packages from xpra.org definitely don't do that.

aac+mpeg4 (..) Dropping this line (..) fixes aac+mpeg4 for me

Done, thanks!

mpeg4+mp3: the following HTML5 client error (..) Media resource blob:https://localhost:14500/155def2c-320c-47e8-aba5-d7f93d0b5dff could not be decoded.

Ah. Those errors are completely undecipherable for me. mpeg4+mp3 support in the browser requires very specific settings on the server side. When it fails, the feedback you get from the browser is error or could not be decoded. Very difficult to fix reliably. This used to work, at some point, on some browsers... We also have a blacklist of combinations that don't work. It may also need updating.

with Xpra server error:

That's just the html5 client error being forwarded to the server.

legacy wav: the following HTML5 client error:

Could just be rencodeplus giving bytes instead of strings. I'll have to find a way to try it somehow.

http stream: mp3: the unmute button stays but the HTML5 client spews error

Support for this has been removed from the xpra server a while back, and now also from the html5 client: https://github.com/Xpra-org/xpra-html5/commit/40aa975420e494ac5c8f79bdf88d4c3c77e577f1 (it was only really useful for some versions of IE - if you could tolerate the 5 second audio delay..)

basilgello commented 3 years ago

Ah. Those errors are completely undecipherable for me.

Easier than we both thought :rofl: Proxy server does not forward ttps://localhost:14500/<uuid>.

Audio works with HTML5 client running like this:

xpra start --start='google-chrome --incognito' --speaker=on --bind-tcp=0.0.0.0:10000 --daemon=no

and navigating to http://localhost:10000 in Firefox

basilgello commented 3 years ago

I'll have to find a way to try it somehow.

If you want I can share my Podman containerfilles! Debian+xpra from your repo or my builds (I can supply them, too, since they are not yet merged into Debian by Dmitry).

basilgello commented 3 years ago

Also, I guess invoking HTML5 client with audio-forwarding enabled should instruct Xpra proxy to start Xpra seamless / desktop / shadow server with --speaker=on (and another handle should do the same for --microphone=on). And this should have no effect on HTML5 connect action.

Also, I guess https://xxx/<uuid> must be protected by authentication somehow so an attacker with UUID bruteforce can not hijack the audio stream,

totaam commented 3 years ago

Proxy server does not forward

I don't understand.

If you want I can share my Podman...

I can run in containers or virtual machines, I just don't like doing it when developing or debugging things.

should instruct Xpra proxy to start Xpra seamless / desktop / shadow server with --speaker=on

The default is enabled so the proxy doesn't need to do anything.

Also, I guess https://xxx/<uuid> must be protected...

It's virtually impossible to brute force, the uuid has too many bits. In any case, that's now removed.

basilgello commented 3 years ago

I don't understand.

Easy :) Xpra server directly accessible handles http(s)://host:port/uuid by serving a media stream with proper mimetype. Xpra proxy server serves index.html contents with text/html mimetype. Media DOM element expects sound data stream, not text of index.html.

basilgello commented 3 years ago

In any case, that's now removed

Wait, do you mean aac+mpeg4 or opus do not use uuid scheme at all? If it does not, then I have to investigate why directly accessible Xpra server listening on random TCP port delivers sound packets properly, while proxied server does not

totaam commented 3 years ago

Wait, do you mean aac+mpeg4 or opus do not use uuid scheme at all?

Correct. The only http stream was an mp3 one that was only useful for Internet Explorer and this has now been removed.

Audio streams are handled in a similar way to the python client: audio packets are received through the websocket connection and forwarded to either aurora (javascript decoding) or the native MediaSource decoder.

totaam commented 3 years ago

As for why it doesn't work through the proxy: it could well be something rencodeplus related there.

basilgello commented 3 years ago

As for why it doesn't work through the proxy: it could well be something rencodeplus related there.

HTML5 client never worked for me with python3-rencode, too, so it must be something different. I have to check Xpra GTK3 client connected through proxy instance niw.

basilgello commented 3 years ago

I have to check Xpra GTK3 client connected through proxy instance niw.

This, surprisingly, works!

However, no HTML5 client plays audio if proxy server is in use: server.log

basilgello commented 3 years ago

legacy: wav does not work via proxy because second packet in a sequence (all zeros 0x00) pushed by _on_data as-is and AVBuffer function does not know what type the input is.

totaam commented 3 years ago

Why is this different when going through the proxy? does the proxy strip some attributes that would tell the aurora decoder what type of packet it is? Are we meant to set this.audio_aurora_ctx.format?

More importantly. how did you find that out?

basilgello commented 3 years ago

More importantly. how did you find that out?

For now, I have debugged only missing buffer for legacy wav.

Start "direct" Xpra server with xpra start --start=mpv --speaker=on --bind-tcp=0.0.0.0:10000
Start "proxied" Xpra server with xpra start --start=mpv --speaker=on
Replace /usr/share/xpra/www/js/aurora.js minified by Debian package with unminified one from source tree and delete correspondent *.gz and *.br files so they dont interfere
Navigate to https://localhost:14500 using Google Chrome developer mode
Set HTML5 audio debug checkbox, connect to "direct" Xpra server on port 10000 and set breakpoint on aurora.js / AVBuffer function. See every call to AVBuffer has input argument as valid Uint8Array
Disconnect and reconnect to "proxied" Xpra server on port 14500 with user-password authentication. The same breakpoint shows you that first call to AVBuffer is Uint8Array equal to first sound-data packet, but second call's input is \u0000\u0000\u0000… (1746 times) yielding an error https://github.com/Xpra-org/xpra-html5/blob/master/html5/js/lib/aurora/aurora.js#L629

basilgello commented 3 years ago

Looks the issue is decoded sound packet passed through proxy. setting packet-encoders to bencode does not help either. At least, not with wav

basilgello commented 3 years ago

I debugged the issue!

Since wav is especially hard to troubleshoot, I decided to switch to MediaSource codecs, like aac+mpeg4 or webm+opus.

First of all, I applied the following patch to Client.js:

diff --git a/html5/js/Client.js b/html5/js/Client.js
index 5728faf..2dc86ce 100644
--- a/html5/js/Client.js
+++ b/html5/js/Client.js
@@ -89,6 +89,7 @@ XpraClient.prototype.init_settings = function(container) {

 XpraClient.prototype.init_state = function(container) {
        // state
+       this.audio_packets = 0;
        this.connected = false;
        this.desktop_width = 0;
        this.desktop_height = 0;
@@ -3133,6 +3134,7 @@ XpraClient.prototype._process_sound_data = function(packet, ctx) {
        }
        catch(e) {
                ctx.on_audio_state_change("error", ""+e);
+               ctx.clog(e, "sound data error")
                ctx.exc(e, "sound data error");
                ctx.close_audio();
        }
@@ -3160,6 +3162,7 @@ XpraClient.prototype.add_sound_data = function(codec, buf, metadata) {
                //push metadata first:
                for (let i = 0; i < metadata.length; i++) {
                        this.debug("audio", "metadata[", i, "]=", metadata[i], ", length=", metadata[i].length, ", type=", Object.prototype.toString.call(metadata[i]));
+                       this.clog("metadata[", i, "]=", metadata[i], ", length=", metadata[i].length, ", type=", Object.prototype.toString.call(metadata[i]));
                        this.audio_buffers.push(Utilities.StringToUint8(metadata[i]));
                }
                //since we have the metadata, we should be good to go:
@@ -3205,6 +3209,7 @@ XpraClient.prototype.add_sound_data = function(codec, buf, metadata) {
 };

 XpraClient.prototype._audio_start_stream = function() {
+       this.audio_packets = 0;
        this.debug("audio", "audio start of "+this.audio_framework+" "+this.audio_codec+" stream");
        if (this.audio_state=="playing" || this.audio_state=="waiting") {
                //nothing to do: ready to play
@@ -3256,6 +3261,10 @@ XpraClient.prototype._audio_ready = function() {
 XpraClient.prototype.push_audio_buffer = function(buf) {
        if (this.audio_framework=="mediasource") {
                this.audio_source_buffer.appendBuffer(buf);
+               if (this.audio_packets <= 5) {
+                       this.clog("appendBuffer size=", buf.length, "buf=", Utilities.ArrayBufferToBase64(buf));
+                       this.audio_packets++;
+               }
                const b = this.audio_source_buffer.buffered;
                if (b && b.length>=1) {
                        //for (let i=0; i<b.length;i++) {

The patch just prints metadata and Base64-encoded buffer supplied to appendBuffer, which is later decoded by FFmpeg's DecodeRawMediaPacket.

The good first aac+mpeg4 packet dumped from the HTML5 client directly ttached to Xpra server has the MPEG4-TS headers wrapping the AAC ADTS stream info:

AAAAIGZ0eXBtcDQyAAAAAG1wNDJtcDQxaXNvbWlzbzIAAALJbW9vdgAAAGxtdmhkAAAAAN04D8LdOA/CAAAHCAAAAAAAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAdx0cmFrAAAAXHRraGQAAAAH3TgPwt04D8IAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAE7bWRpYQAAACBtZGhkAAAAAN04D8LdOA/CAACsRAAAAABVxAAAAAAALWhkbHIAAAAAAAAAAHNvdW4AAAAAAAAAAAAAAABTb3VuZEhhbmRsZXIAAAAA5m1pbmYAAAAQc21oZAAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAAqnN0YmwAAABec3RzZAAAAAAAAAABAAAATm1wNGEAAAAAAAAAAQAAAAAAAAAAAAIAEAAAAACsRAAAAAAAKmVzZHMAAAAAAxwAAQAEFEAVAAAAAAAAAAAAAAAFBRIQVuUABgECAAAAEHN0dHMAAAAAAAAAAAAAABBzdHNjAAAAAAAAAAAAAAAUc3RzegAAAAAAAAAAAAAAAAAAABBzdGNvAAAAAAAAAAAAAAA9dWR0YQAAADVtZXRhAAAAAAAAACFoZGxyAAAAAG1obHJtZGlyAAAAAAAAAAAAAAAAAAAAAAhpbHN0AAAAPXVkdGEAAAA1bWV0YQAAAAAAAAAhaGRscgAAAABtaGxybWRpcgAAAAAAAAAAAAAAAAAAAAAIaWxzdAAAADxtdmV4AAAAFG1laGQBAAAAAAAAAAAAAAAAAAAgdHJleAAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAAAAAGBtb29mAAAAEG1maGQAAAAAAAAAAQAAAEh0cmFmAAAAHHRmaGQAAAA4AAAAAQAABAAAAAEWAAAAwAAAABB0ZmR0AAAAAAAAAAAAAAAUdHJ1bgAAAAEAAAABAAAAaAAAAR5tZGF03gIATGF2YzU4LjkxLjEwMABCVambC2EaRX9fu3GT48U9/XXxUJm7cC2VqquJljkb7L5T5N86N86dj9/Vt02e2YmGzoqxFlU5UP5bHyjKeE7+qIAoQ+yWLmzlecskhs5ZQMTeewAybNVVRujKIRTVc5VUYSIynjLAAAAD0gOf3BIAS5n1AAAjjD23yqaQ+AZh5Yi3gW4gbrvXs+0OjjqB/LJYM87EYePt+HliG0nzDh6Ri3gPwDtwDlPXRrjMJxK1wYiPh9FIjd8iyh+oPAHB9RSRnBx2b+77VXDeAE4dl42p+e79AKqAmrCsTT8H8fFSVmvPdPf38/FQmbtpMlVmqq6SAKvOve5AJa0WAAAAAAGQAAACQjw=

Decoding it with Base64 and feeding it to ffprobe shows no errors:

$ base64 -d aac+mpeg4-good01.txt | ffprobe -loglevel repeat+level+trace -
[info] ffprobe version 4.3.2-0+deb11u2 Copyright (c) 2007-2021 the FFmpeg developers
[info]   built with gcc 10 (Debian 10.2.1-6)
... snip ...
[trace] Probing mov,mp4,m4a,3gp,3g2,mj2 score:100 size:1127
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [debug] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'ftyp' parent:'root' sz: 32 8 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [debug] ISO: File Type Major Brand: mp42
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'moov' parent:'root' sz: 713 40 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'mvhd' parent:'moov' sz: 108 8 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] time scale = 1800
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'trak' parent:'moov' sz: 476 116 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'tkhd' parent:'trak' sz: 92 8 468
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'mdia' parent:'trak' sz: 315 100 468
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'mdhd' parent:'mdia' sz: 32 8 307
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'hdlr' parent:'mdia' sz: 45 40 307
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] ctype=[0][0][0][0]
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] stype=soun
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'minf' parent:'mdia' sz: 230 85 307
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'smhd' parent:'minf' sz: 16 8 222
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'dinf' parent:'minf' sz: 36 24 222
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'dref' parent:'dinf' sz: 28 8 28
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [debug] Unknown dref type 0x206c7275 size 12
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'stbl' parent:'minf' sz: 170 60 222
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'stsd' parent:'stbl' sz: 94 8 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] size=78 4CC=mp4a codec_type=1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] audio channels 2
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] version =0, isom =1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'esds' parent:'stsd' sz: 42 8 42
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] MPEG-4 description: tag=0x03 len=28
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] MPEG-4 description: tag=0x04 len=20
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] esds object type id 0x40
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] MPEG-4 description: tag=0x05 len=5
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] Specific MPEG-4 header len=5
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] mp4a config channels 2 obj 2 ext obj 5 sample rate 44100 ext sample rate 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'stts' parent:'stbl' sz: 16 102 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] track[0].stts.entries = 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'stsc' parent:'stbl' sz: 16 118 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] track[0].stsc.entries = 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'stsz' parent:'stbl' sz: 20 134 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] sample_size = 0 sample_count = 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'stco' parent:'stbl' sz: 16 154 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'udta' parent:'trak' sz: 61 415 468
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'meta' parent:'udta' sz: 53 8 53
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'hdlr' parent:'meta' sz: 33 8 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] ctype=mhlr
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] stype=mdir
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'ilst' parent:'meta' sz: 8 41 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'udta' parent:'moov' sz: 61 592 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'meta' parent:'udta' sz: 53 8 53
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'hdlr' parent:'meta' sz: 33 8 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] ctype=mhlr
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] stype=mdir
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'ilst' parent:'meta' sz: 8 41 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'mvex' parent:'moov' sz: 60 653 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'mehd' parent:'mvex' sz: 20 8 52
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'trex' parent:'mvex' sz: 32 28 52
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'moof' parent:'root' sz: 96 753 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] moof offset 2e9
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'mfhd' parent:'moof' sz: 16 8 88
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'traf' parent:'moof' sz: 72 24 88
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'tfhd' parent:'traf' sz: 28 8 64
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] frag flags 0xc0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'tfdt' parent:'traf' sz: 16 36 64
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'trun' parent:'traf' sz: 20 52 64
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] flags 0x1 entries 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [debug] found tfdt time 0, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] first sample flags 0xc0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] AVIndex stream 0, sample 1, offset 351, dts 0, size 278, distance 0, keyframe 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] type:'mdat' parent:'root' sz: 286 849 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] on_parse_exit_offset=849
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [debug] Before avformat_find_stream_info() pos: 849 bytes read:1127 seeks:0 nb_streams:1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] stream 0, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [debug] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] stream 0: start_time: 0 duration: 0.02322
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [trace] format: start_time: 0 duration: 0.02322 (estimate from stream) bitrate=0 kb/s
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x556798307d80] [debug] After avformat_find_stream_info() pos: 1127 bytes read:1127 seeks:0 frames:1
[info] Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:':
[info]   Metadata:
[info]     major_brand     : mp42
[info]     minor_version   : 0
[info]     compatible_brands: mp42mp41isomiso2
[info]     creation_time   : 2021-08-10T11:13:06.000000Z
[info]   Duration: 00:00:00.02, start: 0.000000, bitrate: N/A
[info]     Stream #0:0(und), 1, 1/44100: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 95 kb/s (default)
[info]     Metadata:
[info]       creation_time   : 2021-08-10T11:13:06.000000Z
[info]       handler_name    : SoundHandler
[AVIOContext @ 0x556798310bc0] [verbose] Statistics: 1127 bytes read, 0 seeks

The first aac+mpeg4 packet from Xpra server beyond Xpra proxy server, however, has valid MPEG4-TS headers but AAC (or OPUS) streams are zeroed out:

AAAAIGZ0eXBtcDQyAAAAAG1wNDJtcDQxaXNvbWlzbzIAAALJbW9vdgAAAGxtdmhkAAAAAN04EKndOBCpAAAHCAAAAAAAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAdx0cmFrAAAAXHRraGQAAAAH3TgQqd04EKkAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAE7bWRpYQAAACBtZGhkAAAAAN04EKndOBCpAACsRAAAAABVxAAAAAAALWhkbHIAAAAAAAAAAHNvdW4AAAAAAAAAAAAAAABTb3VuZEhhbmRsZXIAAAAA5m1pbmYAAAAQc21oZAAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAAqnN0YmwAAABec3RzZAAAAAAAAAABAAAATm1wNGEAAAAAAAAAAQAAAAAAAAAAAAIAEAAAAACsRAAAAAAAKmVzZHMAAAAAAxwAAQAEFEAVAAAAAAAAAAAAAAAFBRIQVuUABgECAAAAEHN0dHMAAAAAAAAAAAAAABBzdHNjAAAAAAAAAAAAAAAUc3RzegAAAAAAAAAAAAAAAAAAABBzdGNvAAAAAAAAAAAAAAA9dWR0YQAAADVtZXRhAAAAAAAAACFoZGxyAAAAAG1obHJtZGlyAAAAAAAAAAAAAAAAAAAAAAhpbHN0AAAAPXVkdGEAAAA1bWV0YQAAAAAAAAAhaGRscgAAAABtaGxybWRpcgAAAAAAAAAAAAAAAAAAAAAIaWxzdAAAADxtdmV4AAAAFG1laGQBAAAAAAAAAAAAAAAAAAAgdHJleAAAAAAAAAABAAAAAQAAAAAAAAAAAAAAAAAAAGBtb29mAAAAEG1maGQAAAAAAAAAAQAAAEh0cmFmAAAAHHRmaGQAAAA4AAAAAQAABAAAAAGBAAAAwAAAABB0ZmR0AAAAAAAAAAAAAAAUdHJ1bgAAAAEAAAABAAAAaAAAAYltZGF0AAAAAAAAAAUIAAkBAAEAAAAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAYAAAcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkAAAAAAAAAAAAAAAAAAAAAAAAGAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==

Feeding it to ffprobe shows an error which in fact is a result of zeroed memory area starting offset 0x351 (849):

$ base64 -d aac+mpeg4-bad01.txt | ffprobe -loglevel repeat+level+trace -
[info] ffprobe version 4.3.2-0+deb11u2 Copyright (c) 2007-2021 the FFmpeg developers
[info]   built with gcc 10 (Debian 10.2.1-6)
... snip ...
[pipe @ 0x559c3b348840] [debug] Setting default whitelist 'crypto,data'
[trace] Probing mov,mp4,m4a,3gp,3g2,mj2 score:100 size:1234
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [debug] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'ftyp' parent:'root' sz: 32 8 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [debug] ISO: File Type Major Brand: mp42
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'moov' parent:'root' sz: 713 40 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'mvhd' parent:'moov' sz: 108 8 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] time scale = 1800
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'trak' parent:'moov' sz: 476 116 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'tkhd' parent:'trak' sz: 92 8 468
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'mdia' parent:'trak' sz: 315 100 468
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'mdhd' parent:'mdia' sz: 32 8 307
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'hdlr' parent:'mdia' sz: 45 40 307
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] ctype=[0][0][0][0]
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] stype=soun
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'minf' parent:'mdia' sz: 230 85 307
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'smhd' parent:'minf' sz: 16 8 222
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'dinf' parent:'minf' sz: 36 24 222
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'dref' parent:'dinf' sz: 28 8 28
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [debug] Unknown dref type 0x206c7275 size 12
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'stbl' parent:'minf' sz: 170 60 222
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'stsd' parent:'stbl' sz: 94 8 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] size=78 4CC=mp4a codec_type=1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] audio channels 2
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] version =0, isom =1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'esds' parent:'stsd' sz: 42 8 42
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] MPEG-4 description: tag=0x03 len=28
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] MPEG-4 description: tag=0x04 len=20
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] esds object type id 0x40
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] MPEG-4 description: tag=0x05 len=5
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] Specific MPEG-4 header len=5
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] mp4a config channels 2 obj 2 ext obj 5 sample rate 44100 ext sample rate 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'stts' parent:'stbl' sz: 16 102 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] track[0].stts.entries = 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'stsc' parent:'stbl' sz: 16 118 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] track[0].stsc.entries = 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'stsz' parent:'stbl' sz: 20 134 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] sample_size = 0 sample_count = 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'stco' parent:'stbl' sz: 16 154 162
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'udta' parent:'trak' sz: 61 415 468
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'meta' parent:'udta' sz: 53 8 53
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'hdlr' parent:'meta' sz: 33 8 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] ctype=mhlr
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] stype=mdir
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'ilst' parent:'meta' sz: 8 41 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'udta' parent:'moov' sz: 61 592 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'meta' parent:'udta' sz: 53 8 53
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'hdlr' parent:'meta' sz: 33 8 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] ctype=mhlr
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] stype=mdir
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'ilst' parent:'meta' sz: 8 41 41
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'mvex' parent:'moov' sz: 60 653 705
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'mehd' parent:'mvex' sz: 20 8 52
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'trex' parent:'mvex' sz: 32 28 52
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'moof' parent:'root' sz: 96 753 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] moof offset 2e9
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'mfhd' parent:'moof' sz: 16 8 88
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'traf' parent:'moof' sz: 72 24 88
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'tfhd' parent:'traf' sz: 28 8 64
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] frag flags 0xc0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'tfdt' parent:'traf' sz: 16 36 64
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'trun' parent:'traf' sz: 20 52 64
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] flags 0x1 entries 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [debug] found tfdt time 0, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] first sample flags 0xc0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] AVIndex stream 0, sample 1, offset 351, dts 0, size 385, distance 0, keyframe 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] type:'mdat' parent:'root' sz: 393 849 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] on_parse_exit_offset=849
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [debug] Before avformat_find_stream_info() pos: 849 bytes read:1234 seeks:0 nb_streams:1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] stream 0, sample 0, dts 0
[aac @ 0x559c3b349940] [debug] stereo with SCE
[aac @ 0x559c3b349940] [debug] channel element 0.0 duplicate
[aac @ 0x559c3b349940] [error] channel element 0.0 is not allocated
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [debug] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] stream 0: start_time: 0 duration: 0.02322
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [trace] format: start_time: 0 duration: 0.02322 (estimate from stream) bitrate=0 kb/s
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559c3b347d80] [debug] After avformat_find_stream_info() pos: 1234 bytes read:1234 seeks:0 frames:1
[info] Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:':
[info]   Metadata:
[info]     major_brand     : mp42
[info]     minor_version   : 0
[info]     compatible_brands: mp42mp41isomiso2
[info]     creation_time   : 2021-08-10T11:16:57.000000Z
[info]   Duration: 00:00:00.02, start: 0.000000, bitrate: N/A
[info]     Stream #0:0(und), 1, 1/44100: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 132 kb/s (default)
[info]     Metadata:
[info]       creation_time   : 2021-08-10T11:16:57.000000Z
[info]       handler_name    : SoundHandler
[AVIOContext @ 0x559c3b350bc0] [verbose] Statistics: 1234 bytes read, 0 seeks

Now the question: why does Xpra proxy server rewrite / transcode the audio stream packets?

totaam commented 3 years ago

Now the question: why does Xpra proxy server rewrite / transcode the audio stream packets?

It isn't meant to - and it probably doesn't do it.

My guess is that it has something to do with large strings getting blanked out by the html5 client whereas Uint8Arrays are copied from the worker without problems. (see how draw packets are handled). And this may explain some of the other problems too.

This problem could also be fixed by adding another special case in process_server_packet: https://github.com/Xpra-org/xpra/blob/1d1b1a7e3e1f47efa9a867788ef4cf8f59456f63/xpra/server/proxy/proxy_instance.py#L444, calling self._packet_recompress like the other special cases do. But the html5 client should be fixed too.

basilgello commented 3 years ago

And this may explain some of the other problems too.

Probably you are right because attaching GTK3 client to Xpra proxy like this:

GTK3 - wss -> Xpra proxy (:14500) - socket -> Xpra server

plays sound without issues

basilgello commented 3 years ago

Adding self._packet_recompress(packet, 2, "sound-data") did not help: HTML5 client complains on overflows…

basilgello commented 3 years ago

The default is enabled so the proxy doesn't need to do anything.

Meh... it is Debian packaging again. I guess when @onlyjob gets well we need to discuss some configuration tweaks introduced by him back in 2013-2016, namely https://salsa.debian.org/debian/xpra/-/blob/master/debian/patches/conf-default-codec-order.patch and https://salsa.debian.org/debian/xpra/-/blob/master/debian/patches/conf-default-speaker.patch

basilgello commented 3 years ago

My guess is that it has something to do with large strings getting blanked out by the html5 client whereas Uint8Arrays are copied from the worker without problems. (see how draw packets are handled). And this may explain some of the other problems too.

Now I am going to trace this assumption.

totaam commented 3 years ago

@basilgello how about the fix above for the proxy?

And this should fix the html5 client (for older proxies without the fix above), in 3 commits because I made a complete mess of it (I really need to sort out pipewire support so I can test locally): https://github.com/Xpra-org/xpra-html5/commit/83dc98f88d97523cb06a3253b715f98a289c78d1 + https://github.com/Xpra-org/xpra-html5/commit/226718ba82f524eb7a374b702341c609094ca744 + https://github.com/Xpra-org/xpra-html5/commit/b28eadd3a24a1258012187bb6fe63941aa5aca85

FYI: I made a blooper in the xpra-html5 4.3 release: https://github.com/Xpra-org/xpra-html5/commit/6a2f0b40b7dc33054509f7f222fec36c8d419970 So I intend to release 4.3.1 soon to fix that. Then I'm going to try to move the decoding to a worker thread. (should really help with h264 and Javascript based pre-processing like rgb24)

basilgello commented 3 years ago

how about the fix above for the proxy?

Do you want me to build trunk Xpra + HTML5 now?

basilgello commented 3 years ago

Do you want me to build trunk Xpra + HTML5 now?

When PulseAudio does not die shortly after start-up, it works with HTML5. Now I would like to check why did it start crashing... And not on Bromite for Android (based on Chromium 92).

basilgello commented 3 years ago

As for pulseaudio start, starting Xpra server manually like xpra start --speaker=on starts pulseaudio correctly, but starting it via proxy makes PA terminate with main.c: D-Bus name org.PulseAudio1 already taken error. On deletion, /run/user/1000/xpra/1/pulse does not get removed, too.

This is because Xpra proxy does not allocate D-Bus daemon when starting a session with speaker enabled:

DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus'

while manual start does allocate one:

DBUS_SESSION_BUS_ADDRESS': 'unix:abstract=/tmp/dbus-ZT8nXZrD8a,guid=ff33bdf97324ffdb568294366115661d'

Sound playback works in Firefox browser as client but fails on Bromite for Android.

basilgello commented 3 years ago

As for pulseaudio start, starting Xpra server manually like xpra start --speaker=on starts pulseaudio correctly, but starting it via proxy makes PA terminate with main.c: D-Bus name org.PulseAudio1 already taken error. On deletion, /run/user/1000/xpra/1/pulse does not get removed, too.

This is because Xpra proxy does not allocate D-Bus daemon when starting a session with speaker enabled:
DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus'
while manual start does allocate one:
DBUS_SESSION_BUS_ADDRESS': 'unix:abstract=/tmp/dbus-ZT8nXZrD8a,guid=ff33bdf97324ffdb568294366115661d'
Sound playback works in Firefox browser as client but fails on Bromite for Android.

Possible fix for D-Bus issue above and #3230 :

diff --git a/xpra/scripts/server.py b/xpra/scripts/server.py
index a6e1133..100c97e 100644
--- a/xpra/scripts/server.py
+++ b/xpra/scripts/server.py
@@ -1181,7 +1181,11 @@ def do_run_server(script_file, cmdline, error_cb, opts, extra_args, mode, displa
             except ImportError as e:
                 dbuslog("dbus components are not installed: %s", e)
             else:
-                dbus_pid, dbus_env = start_dbus(opts.dbus_launch)
+                # Speaker and microphone forwarding needs private D-Bus and
+                # PulseAudio daemons
+                from xpra.server import server_features
+                always = server_features.audio
+                dbus_pid, dbus_env = start_dbus(opts.dbus_launch, always)
                 if dbus_env:
                     dbuslog("started new dbus instance: %s", dbus_env)
                     save_session_file("dbus.pid", "%s" % dbus_pid)
diff --git a/xpra/server/dbus/dbus_start.py b/xpra/server/dbus/dbus_start.py
index 2f5f920..5527cb3 100755
--- a/xpra/server/dbus/dbus_start.py
+++ b/xpra/server/dbus/dbus_start.py
@@ -15,13 +15,13 @@ from xpra.log import Logger
 log = Logger("dbus")

-def start_dbus(dbus_launch):
+def start_dbus(dbus_launch, always=False):
     if not dbus_launch or dbus_launch.lower() in FALSE_OPTIONS:
         log("start_dbus(%s) disabled", dbus_launch)
         return 0, {}
     bus_address = os.environ.get("DBUS_SESSION_BUS_ADDRESS")
     log("dbus_launch=%r, current DBUS_SESSION_BUS_ADDRESS=%s", dbus_launch, bus_address)
-    if bus_address:
+    if bus_address and not always:
         log("start_dbus(%s) disabled, found an existing DBUS_SESSION_BUS_ADDRESS=%s", dbus_launch, bus_address)
         return 0, {}
     assert POSIX

PS: I probably have watched Big Buck Bunny a hundred times already :rofl:

totaam commented 3 years ago

Is your proxy running as root? Perhaps we should always start dbus and discard the DBUS_SESSION_BUS_ADDRESS. It could just be remnant for when people were starting xpra via dbus-launch. I can't see a valid reason for re-using an existing dbus session.

basilgello commented 3 years ago

Is your proxy running as root?

Yes, Debian packaging runs it as root.

Perhaps we should always start dbus and discard the DBUS_SESSION_BUS_ADDRESS.

Maybe, except of shadow servers maybe ?

totaam commented 3 years ago

How about 65c8f1c903a3afb7adec871c6909de7acb7895fa ?

Maybe, except of shadow servers maybe ?

The whole dbus section already has this: if not shadowing and POSIX and not OSX:

Did you check if either or both of the fixes above worked? (as per https://github.com/Xpra-org/xpra/issues/3234#issuecomment-897693214)

basilgello commented 3 years ago

Did you check if either or both of the fixes above worked?

If by "fixes above" you mean my patch - I never publush things that don't work for me :)

65c8f1c shoukd be good to go, too :)

basilgello commented 3 years ago

freshly compiled trunk: the dbus + pulseaudio servers are spawned as expected :)

totaam commented 3 years ago

If by "fixes above" you mean my patch

No. I meant the fixes I linked to previously. Specifically:

4091aa1929743227c24631eb40f00fb68a59c041 for the proxy
html5 client: Xpra-org/xpra-html5@83dc98f + Xpra-org/xpra-html5@226718b + Xpra-org/xpra-html5@b28eadd

I would expect either set of fixes to resolve your sound issues with the html5 client when going through the proxy:

the first one is no guarantee, but it should fix things because sound data is always chunked IIRC - and we now preserve that, ensuring there should be no discernible differences between proxied and non-proxied connections when it comes to audio packets
the second deals correctly with audio packets that are not chunked (ie: older versions of the proxy)

basilgello commented 3 years ago

Yes, that worked, thanks! Issues include broken playback on Bromite for Android but it is out of scope since Firefox / Chrome play sound fine (but ~5-6 second delay)

totaam commented 2 years ago

I don't see any issues left here, let's close and open new tickets as needed.

Xpra-org / xpra

Sound playback tracking issue #3234