Eliminate socket exhaustion after playing many streamed audio files by migrating away from HTML5 audio to use a more modern pure Web Audio API approach relying upon AudioBufferSourceNode.

[aaclayton]

Playing multiple large audio files back-to-back eventually ceases to load a new sound.

Experiment with the Michael Ghelfi ambience playlist, after playing 5-6, further sounds will not load properly.

[aaclayton]

marked this issue as related to #4122

[aaclayton]

Comment from @atnoslen

Environment Details

• Foundry VTT Version: 0.7.9
• Operating System: Windows
• How Are You Using Foundry: Chrome, Firefox
• Chrome Version: 87.0.4280.141 (Official Build) (64-bit)
• FireFox Version: 84.0.2 (64-bit)
• Which Game System: pf2e
• Modules Enabled?: no (Disabled all modules)
• Apache proxy: yes
• Client certificate: yes

Issue Description

Repeatedly, after the 6th playlist is played, Chrome client appears to hit socket exhaustion. Here we can see the 6th .ogg file is requested and has a status of pending.

For FireFox, it is the 10th request.

Attempting to refresh the browser does not work for Chrome and shows "Waiting for available socket..." at bottom.

To fix on Chrome, one has to open a new tab and close original.

Apache log shows GET of each file, except one that is shown pending.

Foundry logs (foundrydata/Logs/debug.log and foundrydata/Logs/error.log) do not show request or error.

No log message appears in Console. Pausing Debugger hit's this._tick() as expected.

[aaclayton]

Originally in GitLab by @anathemamask

marked this issue as related to #3679

[aaclayton]

@Zarmian this issue is still present - as mentioned in the previous response the upcoming milestone %"Alpha 0.8.1 - Audio Overhaul" is dedicated to re-engineering some of the audio streaming systems. There are many component issues in that milestone that have to do with that work and would jointly (hopefully) resolve this issue.

[aaclayton]

Originally in GitLab by @Zarmian

Noticed this issue was closed; hope to reopen it. I'm running a new version of a hosted server today and can confirm it still exists. Almost certainly this is caused by the audio board opening multiple concurrent HTTP connections to the domain at once, instead of synching them through a single socket. Browsers limit the number of concurrent connections to 6. So, unless old connections are closed (which would still limit it to max 6, but at least would stop a complete browser crash), or synched into a single stream, this problem will remain. This is a client-side restriction that needs backend revision to the way audio is streamed.

[aaclayton]

Originally in GitLab by @spoidar

@aaclayton I noted an alternative method for delivering such files without the delay in dev chat, I'll ping it to you there.

[aaclayton]

That being said, I don't want to promise miracles. If you're using large (15MB+) audio files there is - under any solution- going to be some friction you will experience due to lengthy loading and parsing times.

[aaclayton]

Hey @TentacleRevenge I agree it's a high priority issue to address, unfortunately a solution here will involve implementing or modifying the audio playback library that is used which constitutes a major alteration to the software, which is why it needs to be handled in 0.8.x. Please note the upcoming milestone https://gitlab.com/foundrynet/foundryvtt/-/milestones/62 which is dedicated to audio improvements.

[aaclayton]

Originally in GitLab by @TentacleRevenge

From recent messaged in the discord bug-report: https://discord.com/channels/170995199584108546/789780874089988116/792788677670731816

It seems this issue persists. If I may add, this issue is the ONLY reason I have been unable to use Foundry since my purchase and initial intent to transition over. Playing audio through the built in playlist is essential and this problem renders it unusable. I'm not complaining, trying to highlight that this issue perhaps carries more weight than initially thought

[aaclayton]

I'm going to solve this in the short term by making the WebAudio API the default behavior for sounds, but adding a configurable option to mark certain sounds as "large files" which require streaming via the HTML5 audio API.

Under web audio default scheme there is an unlimited number of concurrent audio tracks that can be playing, with the disadvantage that each file needs to be fully loaded before playback can begin. For small audio files this is definitely best, but for large files it can crash the browser if you do not properly mark your file as large and require HTML5 streaming.

[aaclayton]

Originally in GitLab by @spoidar

New revision of this patch, it's larger for reasons noted below, but changes to the existing codebase remain minimal.

Some details: HTML5 media elements return a promise from a call to play(), that promise will be rejected based on any number of changes made by writing to the element before the playback starts completely, throwing an exception if that's not handled everywhere. As such, I've ditched the custom element for a wrapper, and guarded all writes with a promise that's by default resolved, unless there is a play request in progress.

Patch for v0.7.1

Patch for v0.6.6

Not expecting any feedback unless you have questions at some stage, but I've deployed the v0.6.6 version to my server to run some games.

[aaclayton]

Originally in GitLab by @spoidar

Turns out it can be a bit racy, I'll get a new revision up soonish with a few fixes.

[aaclayton]

Hey @spoidar looks promising. I'm mentally putting audio changes on the back burner until I get done with 0.7.2, but I will definitely invest more energy in this direction once audio pops to the top of my queue.

[aaclayton]

Originally in GitLab by @spoidar

Hey @aaclayton I've worked up an implementation as a patch against the bundled foundry.js here. The mapping was nearly 1:1 with the Howler API surface we were consuming, so it's a pretty clean cut. I suspect your src repo probably does not store foundry.js as a single file, but there aren't too many changes, so hopefully this is still useful.

Note: It's currently implemented as a custom element that extends HTMLAudioElement, however that won't work in Safari as they refuse to implement that part of the spec. If Safari is a valid target, I can convert this to a wrapper class with all the associated methods/getters/setters we use, to proxy to the standard element (the other option is to polyfill, but that's a trade-off between added weight and maint cost).

I believe I've tested most of the audio functionality, and it's all working well here, as well as fixing this bug.

I noticed that you have audio improvements for 0.7.3 on the roadmap you posted today, and I'm not sure what your thoughts are on the direction you want to take with that - if you're thinking about migrating to the Web Audio API instead of HTML5 audio, that's clearly a whole other kettle of fish, but I started this earlier today, before I read your announcement.

[aaclayton]

For the most part what Howler bought was the ability for me to not need to worry about the inner workings of the HTML5 audio API and to "outsource" that to a library that was very well reviewed for this particular purpose. That being said, I agree that Howler was really a lib designed for pre-HTML5 audio - where many of the APIs that Howler provides are now natively provided. I'm in favor of evaluating the possibility that Howler could be removed entirely - but it's not something I want to look at until I begin work on 0.7.3. If you want to work on a POC API which wraps an interface layer around the native tools, I'm certainly open to reviewing it.

[aaclayton]

Originally in GitLab by @spoidar

Out of curiosity, what is Howler buying us? The upstream issue tracker appears to have gotten well out of control, and it's a lot of code unnecessary for what we're doing. We appear to be enabling HTML5 audio everywhere, and it doesn't look like we're using much of the novel Howler API. Would you be at all interested in replacing it with straight HTMLAudioElements if I was to look at working up a patch?

[aaclayton]

Thanks for sharing your perspective on this @spoidar - this is definitely on the right track. I need to kill off the streamed connections if the song remains stopped for more than a second or two.

[aaclayton]

Originally in GitLab by @spoidar

Stopping an audio stream leaks server connections.

When stopping an audio stream before it plays to completion (ie, via playlist controls in the menu, but also possibly when triggered by other actions, like deleting a playing track, or changing a playing track's path, changing scenes with attached playlists, though I've not confirmed these alternative paths), the call to howl.stop() sets the seek position to 0 and updates some internals, but does not cancel the HTTP request, so the connection stays open forever (as long as there's a reference around to the media element). This is a significant problem, because browsers implement a max limit on concurrent connections to a server - for Chrome this limit is 6, and we already consume one for the websocket - so after playing and stopping a number of audio tracks, the max number of HTTP connections to the server is reached, and the tab can no longer make any HTTP calls to the server. The tab is basically soft-locked at that point, and must be closed, and a new tab opened (reloads on Chrome require a new socket). As audio playback is pushed to all clients, this means that all connected clients may get soft-locked too, requiring every client to close their tab and re-join. On a standard HTMLMediaElement you can set the src to an empty string (or null maybe?) to get it to cancel the request, but with howler.js in the way, I'm not sure what the right approach is.

This bug affects all current versions of Foundry (0.6.5, 0.6.6, 0.7.1).

[aaclayton]

Originally in GitLab by @kimitsu_desu

I wonder if this is caused by some active connection limit, which fail to free up? This may be related:

• https://serverfault.com/questions/708319/chrome-requests-get-stuck-pending
• https://stackoverflow.com/questions/18928945/google-chrome-not-completing-request-if-a-partial-content-header-is-set
• https://stackoverflow.com/questions/49733634/why-are-http-requests-getting-stuck-on-pending-in-chrome

[aaclayton]

Originally in GitLab by @kimitsu_desu

This bug also rendered the application unusable for me: it could not download not only new audio tracks, but also other files, such as graphics. When the bug is active (after playing several tracks) "Network" profiler tab shows that new files are being requested, but are in "Pending state" at 0 bytes transferred, indefinitely. You can't even hit Log Out to reload, have to close Electron and restart.

The bug happens for players in Chrome too, but I could not manage to reproduce it in Firefox. Interesting.

[aaclayton]

Originally in GitLab by @pocaidf

I just experienced this bug myself. I was going through my music files to find one I liked for an upcoming game and it stopped working after a few songs. Songs I had already listened to will still play, but new ones have no sound.

[aaclayton]

Similar issues experience and reproduced on a user's installation. Relatively modest upload speed, several large maps and audio files. Each Scene had an audio playlist set as it's scene playlist. Switching scenes quickly by activating one after the other would exhaust the audio pool and cause the app to freeze up.

[aaclayton]

I haven't experienced issues this severe which make the application "unusable" as you say - but I want to emphasize that this is definitely a priority to address. Milestone 0.7.0 is focused on dice mechanics, but the next milestone in the 0.7.x cycle is going to prioritize audio improvements (including this one).

Just saying that to make sure it's clear to everyone in the thread that I agree this is a priority to address.

[aaclayton]

Originally in GitLab by @TentacleRevenge

This bug essentially renders the audio library/playlist unusable, which is an incredibly important part of the game. As Lawangeen pointed out, seems to have more to do with starting/stopping of files than file size. Disabled all add-on modules and still maaged to replicate the bug. Attached is a screenshot of what frames look like after bug manifests.

[aaclayton]

Originally in GitLab by @siliconsaint

Agreed on this. It's happened 4 times today. After listening to 3-5 songtracks (supplied by Adventure Music & Michael Gelfi compendium) it will "stop" only play previous sounds, and then give a "socket error" if I try to reload. It requires a complete pm2 restart command.

If left long enough it randomly will start playing some of these sounds later (but without the ability to "stop") as if it is unaware it's happening.

This is Lightsail hosted VPS running .62 with Chrome Version 83.0.4103.61 running on MacoS v 10.15.5

[aaclayton]

Originally in GitLab by @sebastianmx2

I experience the same bug. It stops playing the music when I check the songs to adjust their levels after a few songs it stops and I have to restart the game.

[aaclayton]

Originally in GitLab by @RealZeb

I am also having this issue. I have a good number of songs in my playlists, and didn't have the issue when I first added them all. I started having the issue when I went back in to play them all for a few seconds each to adjust the volume. I noticed that after playing 6 songs, the 7th one would not load, and also I would be unable to move between scenes unless I had preloaded them all beforehand. Furthermore, my players saw their character sheet glitch, appearing half transparent. If they tried to roll from the sheet, the roll would process, but no dice rolling sound effect would play. I would be unable to refresh or go back to the main menu. I would have to fully exit Foundry and restart it. My players would not be able to refresh to fix it either, having to wait for me to close Foundry, then rejoin the session and log in again (as Apostol above noted).

I ran a few tests to see if I could pinpoint what was causing the issue.

I created 2 playlists: The "small" playlist, containing songs with a small file size; and the "large" playlist, containing songs with a large file size (mainly FLACs of the Bloodborne OST).

I played the "small" playlist sequentially, letting each track play fully, and in order. I had no issues with this playlist, which contained 31 files for a total of 109 MB. Most were music, some were simple sound effects.

I then played the "large" playlist sequentially, letting each track play fully, and in order. I also had no issues with this playlist, which contained 8 files for a total of 407 MB. All of these were 5-6 minute-long songs.

I then played the first 5 seconds of a song in the "large" playlist, stopped the song, then played the next one, thinking that perhaps loading large files in rapid succession was causing massive slowdown. I again experienced this bug once the 6th song was played. The 7th song did not load, and the character sheets bugged out again.

I then played the first 5 seconds of a song in the "small" playlist, stopped the song, then played the next one. I chose songs which were small in file size but longer than 5 seconds. Similarly as above, the bug happened again, with the 7th song not loading.

My conclusion from this mostly unscientific test is that the file size and song length are not that important. Even playing files as small as 3-6 MB, stopping them after a few seconds, then starting the next one is as likely to trigger the bug as doing the same for files that are 80-88 MB in size.

This is therefore not a bug I think most people would experience in actual use. Most likely it would be seen by people who are similarly going down each song in their track list and adjusting the volume. If anyone else who has seen this bug can confirm that's roughly what they were doing when they experienced it, that'd be great. But that's the only way I can reproduce it. Again, not so much back to back, but starting it, stopping it shortly after, then starting another one right after that seems to be the issue.

[aaclayton]

Originally in GitLab by @apoapostolov

I had similar issues but then I also noticed that when this happens, reloading Foundry web page will stall and the server needs to be killed and restarted, forcing everyone to re-login again, etc.