search

DuckBoss / JJMumbleBot

A plugin-based All-In-One mumble bot solution in python 3.7+ with extensive features and support for custom plugins.
GNU General Public License v3.0
50 stars 10 forks source link

No audio playback from Youtube #238

Closed MacLemon closed 4 years ago

MacLemon commented 4 years ago

Describe the bug Playing a Youtube video seems to do everything else but does not play audio.

To Reproduce Steps to reproduce the behaviour:

  1. Search for a video: !yt IH6C3-GUai8
  2. Play the first item (which is the exact Video ID match) !p 0
  3. See the posting with artwork and title in the chat
  4. No audio is playing

Expected behavior Audio should play. Bot seems to do everything else just fine, no bot icon highlighting that it is emitting audio either.

Screenshots Screen_Shot_2020-07-16_at_23_15_13

Version information:

Additional context Log output by the bot during the interaction described above:

[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !yt IH6C3-GUai8]
[{'title': 'Die beste Fahrstuhlmusik oder entspannende Lounge instrumental Aufzug', 'href': '/watch?v=IH6C3-GUai8'}, {'title': '4 Minuten Cringe ohne Kontext mit extra viel Bacon', 'href': '/watch?v=fV9qhquXq44'}, {'title': 'Was ein Feini- mit Lucas', 'href': '/watch?v=Hl5OYmqi4Hg'}, {'title': 'Hexameter | Latein Projekt Video', 'href': '/watch?v=0Nm_P7Rdtws'}, {'title': 'Shipment', 'href': '/watch?v=pkef6Gf0amw'}, {'title': '... So Sportsgames are actually funny | Wii Sports', 'href': '/watch?v=RRpFHW6D7OU'}, {'title': 'Ich bin ein Kreis holt mich hier raus!!!', 'href': '/watch?v=8f02fG1uPB8'}, {'title': 'Supremium-Verlosung | Erstes Video mit Stimme| BedWars Only Endstone challenge', 'href': '/watch?v=rc7ElJQO9pM'}, {'title': 'Noch mehr Cringe, als mit Bacon und Deni - mit Lucas', 'href': '/watch?v=1sqjyjHwPjE'}, {'title': 'How to Hammond', 'href': '/watch?v=yScRuNSu7Ug'}]
[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !p 0]
VLC media player 3.0.11 Vetinari (revision 3.0.11-0-gdc0c5ced72)
DuckBoss commented 4 years ago

That's super weird. I have v3.1.3 (on Ubuntu LTS Server) running fine right now with no issues. I tested that video you used, and it seems to be working fine. Refer to the images below (running v3.1.3) image image

I'll look into it, but I can't reproduce it on my end. Here are a couple things you can try: 1) Could you make sure that the volume of the youtube plugin is not 0? You can do this on v3.1.3 with the !volume command. 2) Can you make sure that the bot has the correct permissions to play audio in that mumble server/channel? 3) You could try updating the youtube-dl dependency (If a new version is available, I'm using v2020.6.16.1) 4) Could you check if audio output is working at all? Download the youtube video with a direct link as a sound board clip and try playing that sound board clip. You want to use the !sbdownload video_link file_name command and then do !sb file_name to download the youtube video and play it offline through the sound board plugin. If this works, then that narrows the issue down a little.

MacLemon commented 4 years ago

  1. !volume outputs 0.5 so it's reasonably up that I should hear something, or at least get the “sound emitting” indicator on the bot's icon.

  2. Yes, the bot has permission to play sounds. Soundboard works fine as well.

  3. I'm literally on the exact same version.

    $ youtube-dl --version
2020.06.16.1

  4. Soundboard works fine with existing sounds. Downloading YouTube Videos to the soundboard never worked for me in the sense it never produced playable soundboard files.

Here's the bot's log output of the corresponding interaction.

$ ls -la /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav
-rw-r--r--  1 JJMumbleBot  JJMumbleBot  61973114 Jan 24 14:44 /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav

$ rm /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav

$ ls -la /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav
ls: /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav: No such file or directory

[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !sbdownload https://www.youtube.com/watch?v=IH6C3-GUai8 Fahrstuhlmusik]
Removing cache dir /home/JJMumbleBot/.cache/youtube-dl ..
[youtube] IH6C3-GUai8: Downloading webpage
[youtube] Downloading just video IH6C3-GUai8 because of --no-playlist
[youtube] IH6C3-GUai8: Downloading webpage
[youtube] Downloading just video IH6C3-GUai8 because of --no-playlist
[download] Destination: /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav
[download] 100% of 59.10MiB in 00:01
[ffmpeg] Post-process file /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav exists, skipping

$ ls -la /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav
-rw-r--r--  1 JJMumbleBot  JJMumbleBot  61973114 Jan 24 14:44 /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav

[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !sb Fahrstuhlmusik]
VLC media player 3.0.11 Vetinari (revision 3.0.11-0-gdc0c5ced72)
TagLib: RIFF::File::read() -- Chunk 'B' has invalid ID
TagLib: RIFF::WAV::Properties::read() - 'fmt ' chunk not found or too short.

I've successfully used ffmpeg to create over 250 working soundboard .wav files, so I am pretty sure that ffmpeg is able to produce files that work with vlc and the bot in general.

To me it looks like the ffmpeg postprocessing step is failing, since it says file exsits, skipping. Maybe we're running into a name conflict here? youtube-dl downloading to filename.wav already and ffmpeg trying to use the same name for input and output?

$ file /home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav
/home/JJMumbleBot/media//sound_board/Fahrstuhlmusik.wav: WebM

At least supports my theory that the .wav conversion isn't taking place.

I'm also not sure why it's necessary to convert to .wav at all. vlc would be able to playback .webm, and many other compressed audio formats anyway. (And transcode on the fly into a format that's usable with murmur.) (Would also allow for a lot of storage space saving with soundboard files.)

I'm certain that playing YouTube videos did work in 3.0, not so sure about 3.1.0/3.1.1. It certainly never worked for me in 3.1.2/3.1.3 so far.

DuckBoss commented 4 years ago

The sound board !sbdownload issue:

As far as the soundboard not being able to download youtube files, I am encountering the same error now. It was working for me in version 3.0.0 and I haven't updated or touched that code since. I think when youtube made their infrastructure change, it may have also broken the !sbdownload command. I'll look into that separately since it seems that command isn't able to download any youtube links at the moment.

EDIT: For some reason restarting the bot seems to have fixed the issue for me (even though ffmpeg still skips the post processing in the logs, so I don't think it's a name conflict issue).
On a side note, my logs say I'm running v3.1.4, but that's just a development version from the github master branch which features no changes to sound board downloading, so we are running the same versions as far as the sound board is concerned. image image

Youtube plugin and VLC:

Although vlc supports multiple audio formats, I've been requiring .wav files because to get audio output in mumble, I need to transcode the audio stream into s16le pcm format for it to work. Currently, out of all the different muxers I've tried, only the wav muxer for vlc is able to transcode to s16le. If you know of a work around to this, let me know! The youtube plugin is a bit messy and I've been patching/upgrading it over time since v1.0 and a lot of things have changed, I might have to recreate it at some point.

As of right now, I still can't replicate the original issue where your audio playback isn't working (all youtube links including the one you tried are working for me). I'll review all the changes I made in v3.0+ and see if I can find the issue. The youtube/sound_board plugin issues have certainly come up since youtube's recent changes so I'll have to review this some more.

I'll update this comment if I find anything to add to it

EDIT: Updates since original comment post

I've tried uninstalling youtube-dl completely and reinstalling it, and I haven't been able to replicate any of the issues you've run into. Youtube songs are playing fine for me (using !yt or !link commands), and the sound board is also able to play downloaded youtube music (using !sbdownload command).

I'm running out of ideas, try the following: 1) Try using the direct link command and see if it outputs audio: !link https://youtu.be/IH6C3-GUai8 2) Can you try doing a fresh reinstall of youtube-dl (2020.6.16.1)?

MacLemon commented 4 years ago

Ad 2: I did upgrade all system packages and python packages. I uninstalled and reinstalled youtube-dl as requested.

$ pip freeze
appdirs==1.4.4
asn1crypto==1.3.0
Babel==2.8.0
backports.functools-lru-cache==1.6.1
beautifulsoup4==4.9.0
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
cheroot==8.3.0
Click==7.0
configparser==5.0.0
cryptography==2.6.1
evdev==0.8.1
Flask==1.1.2
gevent==20.6.2
greenlet==0.4.16
html5lib==1.0.1
idna==2.9
itsdangerous==0.24
jaraco.functools==3.0.1
Jinja2==2.10.1
lxml==4.5.2
MarkupSafe==1.1.1
more-itertools==8.4.0
mutagen==1.45.0
olefile==0.46
opuslib==3.0.1
packaging==20.4
Pillow==7.0.0
protobuf==3.12.2
psutil==5.7.0
pycparser==2.20
pymumble==1.3.1
pyOpenSSL==19.0.0
pyparsing==2.4.7
pyradios==0.0.21
PySocks==1.7.1
python-magic==0.4.18
pytz==2020.1
pyudev==0.22.0
requests==2.23.0
six==1.14.0
soupsieve==1.9
sqlite3==0.0.0
Tkinter==0.0.0
urllib3==1.25.9
webencodings==0.5.1
websockets==8.0.2
Werkzeug==1.0.1
WTForms==2.1
youtube-dl==2020.6.16.1
zope.event==4.4
zope.interface==5.1.0
$ python3.7 -V
Python 3.7.8

Ad 1: After 2 I restarted the bot. This is the only output of 3.1.3 when issuing !link https://youtu.be/IH6C3-GUai8

$ [JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !link https://youtu.be/IH6C3-GUai8]
VLC media player 3.0.11 Vetinari (revision 3.0.11-0-gdc0c5ced72)

$
$ [JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !1]
[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !sb 1]
[JJMumbleBot(3.1.3).General]:An audio plugin is using the audio thread with no interruption mode enabled. [Youtube]
[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !stop]

$ [JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !sb 1]
[JJMumbleBot(3.1.3).General]:An audio plugin is using the audio thread with no interruption mode enabled. [Youtube]
[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !stop]
[JJMumbleBot(3.1.3).Command]:Commands Received: [MacLemon -> !sb 1]
[JJMumbleBot(3.1.3).General]:An audio plugin is using the audio thread with no interruption mode enabled. [Youtube]

The Mumble window does show the cover art and claims it is playing, yet neither the console nor the “receiving audio indicator” show any sign of audio actually being sent to the server at all. I do not hear anything in my client.

I then tried to play something from the soundboard. The bot thinks it is playing from youtube though. I tried to stop the bot playing, which it did confirm. Retried playing from the soundboard and it still thinks it is playing and refuses to playback the soundboard. When I try to !stop it again it doesn't confirm anymore, yet still refuses to play !sb.

Interestingly upon issuing !stop the bot did correctly stop the vlc process, but for the duration of the video it keeps thinking that it's still playing.

Screenshot of Mumble of the described interaction timeline: Screen_Shot_2020-07-21_at_18_04_06

So there seem to be multiple correlated issues.

DuckBoss commented 4 years ago

Thank you for being very detailed in your issue reports, it helps a lot! I'm still not able to reproduce this issue in v3.1.3, but based on your last post I can see the problem that's occurring in the bot. When the bot encounters an issue related to an audio stream in vlc, the thread tends to get locked because the stream is active and vlc doesn't end the stream when it encounters an error processing a stream. This prevents other inputs from processing (or they process delayed after the thread dies). As a result, this issue also messes up the way the bot handles checking what plugins are using audio threads. I have a couple questions:

Downloading from YouTube to the Soundboard is working, but the required conversion step with ffmpeg is not taking place. (I guess because youtube-dl already saves with the .wav file extension when the file actually is in a different format like .webm and subsequently ffmpeg refuses to convert because it's refusing to overwrite the existing file which suggests to already be a converted target file, when it's actually not.) → Playback of the downloaded, but unconverted YouTube video via Soundboard fails to play.

  • This make sense, but in all my testing (windows/linux), the youtube downloading works regardless of the ffmpeg overwriting error, and the sound board is able to play all the downloaded youtube files perfectly. 
I'll look into this some more, I'm wondering if there's anything MacOS specific (with vlc, or ffmpeg, or youtube-dl) that could be causing issues with the bot, since that's the only platform I'm unable to test on (assuming your bot runs on MacOS).

I've been forcing youtube-dl to convert/save to `.wav` files to prevent file type errors when I play it in VLC, especially since I'm not sure how much the file types vary per video (if all videos download webm, or some other formats as well). This has been particularly stable when I've been working on implementing sound cloud support as well (unreleased).

A Proposed Solution:

I'm very close to releasing v4.0.0 of JJMumbleBot which completely reworks the audio system (as well as upgrading a lot of other things). It is significantly more optimized and runs a lot better. It is fundamentally different and is a lot more robust/flexible for audio plugins. To summarize, I believe the new v4.0.0 release will fix your issue.
I've created an audio interface system that acts similar to a media player and allows audio plugins to interact with it. With this change, I also reworked how the audio plugin interruption system works.

Some updates/changes specifically related to the audio system in v4.0.0 that might interest you:

Note: There are a lot of other updates, but this relates directly to the audio system

  • Support for SoundCloud links (soundcloud playlists don't work right now)
  • Centralized audio controls for whatever plugin is using the audio interface.
  • Example: Allows you to use !volume for all audio plugins instead of !sbvolume, !ytvolume, etc.
  • Adds support for queuing sound board clips (optional, default behaviour doesn't queue so it behaves the save as v3.0+)
  • Audio ducking system
  • Optionally enable audio ducking (!duckaudio) so the bot automatically lowers the volume when it detects people talking in the channel.

Example screenshot of audio interface debug output: image

After I release v4.0.0, I'll go back and try to retro-fix v3.1.3 if I'm able to replicate the issue on any of my systems. Although I do highly recommend upgrading to v4.0.0 when I release it as it will most likely fix your issue.

DuckBoss commented 4 years ago

Also, it seems that the youtube.lua file has been updated in the last 2 days, maybe try patching it again since the last time you patched the youtube.lua file was ~5 days ago. https://github.com/videolan/vlc/commits/master/share/lua/playlist

MacLemon commented 4 years ago

Downloading from YouTube to the Soundboard is working, but the required conversion step with ffmpeg is not taking place. (I guess because youtube-dl already saves with the .wav file extension when the file actually is in a different format like .webm and subsequently ffmpeg refuses to convert because it's refusing to overwrite the existing file which suggests to already be a converted target file, when it's actually not.) → Playback of the downloaded, but unconverted YouTube video via Soundboard fails to play.

This is not an assumption, it's a verified observation. I did check the files downloaded with file(1)and they were in webm format but with a .wav extension. ffmpeg did not convert them because it would have the identical filename for the source and destination files, so it refused to overwrite the file. As a result the conversion to wav format did not take place and soundboard is unable to play them.

Update vlc

I did try and update youtube.lua again, but it doesn't make a difference.

To summarize, I believe the new v4.0.0 release will fix your issue.

Since it completely changes the way audio works, there's not a lot worth putting any more time into finding the intricate detail fo this in the older release I guess. I personally don't mind going forward, waiting for the 4.0 release and switching to it.

If you're really keen on digging into the 3.1.3 issue, I could provide access to the bot, or connect an instance to a murmur of your choice, etc.

Sidenote: I do hope that 4.0 also does autoreconnect to a server if the connection is lost. (I probably should have filed a separate issue about that now that I think of it. I guess I haven't yet, but I don't want to mix issues here.)

Looking forward to 4.0, the stuff you've hinted at sounds neat!

DuckBoss commented 4 years ago

This is not an assumption, it's a verified observation. I did check the files downloaded with file(1)and they were in webm format but with a .wav extension. ffmpeg did not convert them because it would have the identical filename for the source and destination files, so it refused to overwrite the file. As a result the conversion to wav format did not take place and soundboard is unable to play them.

Thank you so much for confirming this, I should be able to fix this issue.

If you're really keen on digging into the 3.1.3 issue, I could provide access to the bot, or connect an instance to a murmur of your choice, etc.

I don't think this will be necessary, but I'll let you know if it comes down to it.

Sidenote: I do hope that 4.0 also does autoreconnect to a server if the connection is lost. (I probably should have filed a separate issue about that now that I think of it. I guess I haven't yet, but I don't want to mix issues here.)

This was just an oversight on my part, I believe the pymumble library I use for mumble connectivity has a built-in autoreconnect feature. I can get this added to v3.1.3 and v4.0+ after I work some things out.

MacLemon commented 4 years ago

Awesome, very much looking forward to this! \o/

DuckBoss commented 4 years ago

I've made the following changes in one of my recent commits [3c70e32] relating to the sound board plugin. Many of them include fixes and suggestions from your help.

  • Many major sound_board plugin improvements:
  • Updated sound_board plugin to use any vlc-supported file type.
  • Updated !sbdownload command to download youtube videos as 'webm' file types to save 50x-100x storage space.
  • Added !sbsearch command that uses fuzzy-searching techniques to find files most closely matching the search query.
  • Added config options to configure !sbsearch command.
  • Added new dependencies: fuzzywuzzy (fuzzy-searching library) and python-Levenshtein (increases fuzzy-searching performance up to 10x)

There are some other updates to the sound_board plugin as well, but I'll be including that in the release notes and wiki when v4.0.0 is ready for release. You can keep track of my progress here: #244

DuckBoss commented 4 years ago

Hello! If you have some time, could you try out the v4.0.0 Test Branch I have set up? I want to know if it fixes your issue from v3.0+.

As far as setting up the new version, it should be similar to v3.0. I recommend creating the config.ini file from the config_template.ini file in the cfg/ folder, and the bot should be able to set everything up based on your config.ini file when you launch it. You can also just copy the aliases_template.csv file and rename it to global_aliases.csv and place it in the cfg/ folder for the default aliases. I don't recommend trying to use any old configs with this new version for testing. I will be updating the wiki with the full documentation when I'm closer to releasing the new version.

Please let me know if you run into any problems, or have any questions.

DuckBoss commented 4 years ago

I'll be closing this thread due to inactivity, re-open it if your issue is unresolved and you have more information to share.

Gert-dev commented 3 years ago

I was testing out JJMumbleBot yesterday, and can confirm that this is still an issue with 5.1.0. My problem is the same as in the OP:

In other words, the bot does everything as expected, except no audio ever comes out. For comparison's sake, botamusique does produce audio on the same server - which also uses pymumble, but uses ffmpeg to stream downloaded files.

I enabled logging and took a look in the logs, but nothing out of the ordinary is printed, possibly because for the bot, everything is working as it should as far as it is concerned.

The bot is running on VLC 3.0.12 on Arch Linux ARM in my case.

DuckBoss commented 3 years ago

I haven't heard of this issue or run into it during any of my testing. I've tried to replicate this issue but I cannot. It might be a specific case with the vlc installation on an ARM system which would be out of the hands of the bot itself.

The bot is running on VLC 3.0.12 on Arch Linux ARM in my case.

Have you tried running the bot with the docker images from jasonjerome/jjmumblebot:release-5.1.0? I recently included Docker compatibility to allow a uniform experience for all users, regardless of their system setups.

For comparison's sake, botamusique does produce audio on the same server - which also uses pymumble, but uses ffmpeg to stream downloaded files.

I use VLC for youtube streaming to allow large playlists to play seamlessly without downloads, and Ffmpeg for the sound board plugin and anything that requires local audio to be played. Does the sound board audio work for you if the youtube streaming doesn't?

Gert-dev commented 3 years ago

Thanks for the quick response.

Have you tried running the bot with the docker images from jasonjerome/jjmumblebot:release-5.1.0?

That sounds like a good idea, but I don't think I'll be able to run them, since images are only provided for x86-64, at least according to Dockerhub (Docker images are architecture specific, IIRC, so an ARM32 image would need to be built and pushed).

I use VLC for youtube streaming to allow large playlists to play seamlessly without downloads, and Ffmpeg for the sound board plugin and anything that requires local audio to be played. Does the sound board audio work for you if the youtube streaming doesn't?

I tested this and, strange to say, it experiences the same issue, with the difference that the speaking icon for JJMumbleBot does light up in Mumble when it's playing back the sound, but nothing can be heard, not even at 100% volume. It does not light up for YouTube streams using !link.

I tried the exact same YouTube video I used for !sbdownload with botamusique, which also uses ffmpeg, and it does play correctly there. This seems to rule out issues with ffmpeg directly; perhaps it could be related to the parameters that are passed to it? botamusique seems to pass these parameters - though this may be a different problem from the one with vlc and YouTube streaming altogether.

(The streaming is also one of the reasons I became interested in trying out JJMumbleBot, because my ARM device is old and slow, the disk is slow, and downloads result in noticeable latency for playing.)

EDIT: Oops, my device is ARM32 (ARMv7), not ARM64; I corrected that in my post :smile:.

DuckBoss commented 3 years ago

That sounds like a good idea, but I don't think I'll be able to run them, since images are only provided for x86-64, at least according to Dockerhub (Docker images are architecture specific, IIRC, so an ARM32 image would need to be built and pushed).

That's a good point, I didn't realize this (I'm still learning Docker).

I tested this and, strange to say, it experiences the same issue, with the difference that the speaking icon for JJMumbleBot does light up in Mumble when it's playing back the sound, but nothing can be heard, not even at 100% volume. It does not light up for YouTube streams using !link.

I've encountered this issue once before on my Ubuntu private server, and it was caused by a hanging vlc/ffmpeg thread which had to be manually killed/terminated and I had to restart the bot. That is probably not your issue, but I'm just letting you know. I would also suggest checking file permissions for the bot, as that caused it once before for me as well.

You can also enable the ffmpeg/vlc debug messages in your config file by setting the AudioLibraryRunQuiet option to False:

; Enable/Disable Audio Library Console Messages
AudioLibraryRunQuiet = False

I tried the exact same YouTube video I used for !sbdownload with botamusique, which also uses ffmpeg, and it does play correctly there. This seems to rule out issues with ffmpeg directly; perhaps it could be related to the parameters that are passed to it? botamusique seems to pass these parameters - though this may be a different problem from the one with vlc and YouTube streaming altogether.

That is super weird. I'm not sure how to test and resolve this issue since I don't have an ARM32 setup. I'll look into the parameters for ffmpeg/vlc and see if there is some issue with that in newer versions of ffmpeg/vlc. My private Ubuntu Server that the bot is deployed in currently is using ffmpeg v4.2.4 and vlc v3.0.9.2 (newest for Ubuntu 20.04.2 LTS) so I'll have to look into this further before I can offer any advice for fixes.

Thanks for your detailed report, I'll see what I can do

Gert-dev commented 3 years ago

I've encountered this issue once before on my Ubuntu private server, and it was caused by a hanging vlc/ffmpeg thread which had to be manually killed/terminated and I had to restart the bot. That is probably not your issue, but I'm just letting you know.

Good to know, thanks! In my case the vlc process seems to (correctly) start and stop by itself.

You can also enable the ffmpeg/vlc debug messages in your config file by setting the AudioLibraryRunQuiet option to False:

Thanks for the tip. This made me see the following errors:

[024f5380] vlcpulse audio output error: PulseAudio server connection failure: Connection refused
[024df970] main interface error: no suitable interface module
[0247e298] main libvlc error: interface "globalhotkeys,none" initialization failed

This makes sense, since this is a CLI-only device with an SSH server on it; there is no X server nor PulseAudio. Ignoring why VLC desperately needs PulseAudio, even though it should just be running as pipe, I tried to install it anyway to see if that fixes the issue. Installing PulseAudio makes the error go away, but still not sound is produced, as before.

I don't know what the other two errors are about, but some searching around implies that they can be ignored.

My private Ubuntu Server that the bot is deployed in currently is using ffmpeg v4.2.4 and vlc v3.0.9.2 (newest for Ubuntu 20.04.2 LTS) so I'll have to look into this further before I can offer any advice for fixes.

I'm currently on VLC 3.0.12 and ffmpeg 4.3.1, for reference. I also checked the output logs for ffmpeg, but they only show debug logs on what it's doing, and no errors. Also strange that output seems to be produced by the bot in the case of ffmpeg (the icon of the bot lights up), but nothing comes out, which almost sounds like a volume issue, even though it's at maximum.

On a side note, are you using a raspberry pi by any chance?

Not quite, I'm using an old CuBox with an old Marvell Armada 510 ARM32 CPU.

DuckBoss commented 3 years ago

This makes sense, since this is a CLI-only device with an SSH server on it; there is no X server nor PulseAudio. Ignoring why VLC desperately needs PulseAudio, even though it should just be running as pipe, I tried to install it anyway to see if that fixes the issue. Installing PulseAudio makes the error go away, but still not sound is produced, as before. I don't know what the other two errors are about, but some searching around implies that they can be ignored.

I get those 3 errors on my development system too, they can be ignored.

I'm currently on VLC 3.0.12 and ffmpeg 4.3.1, for reference. I also checked the output logs for ffmpeg, but they only show debug logs on what it's doing, and no errors. Also strange that output seems to be produced by the bot in the case of ffmpeg (the icon of the bot lights up), but nothing comes out, which almost sounds like a volume issue, even though it's at maximum.

That really is an odd issue especially since ffmpeg makes the icon light up, but vlc doesn't. Usually when that happens, ffmpeg throws out hundreds of errors, so I am not sure what this is causing this.

botamusique seems to pass these parameters - though this may be a different problem from the one with vlc and YouTube streaming altogether.

I reviewed the botamusique github you mentioned to see their ffmpeg parameter implementation, but it looks like the only difference that may affect the bot would be the additional -nostdin parameter, which I added to the bot in the v5.1.1 release.

I would suggest updating to v5.1.1 Release to see if it makes any difference. This update doesn't require any config changes, and only changes 4 files (1 of them is a wiki change, so it can be ignored) so it should be a straight forward update.

Apart from that, I'm not sure what else I could do on my end. I don't have an Arch ARM32 setup to investigate this further, but I have a raspberry pi which also uses the ARM architecture so I'll conduct some testing on that device to see if I can find anything useful.

Gert-dev commented 3 years ago

I figured it out by a lucky guess: it's stereo audio that's breaking for some reason. If I disable it, both the sound board and the YouTube streamer work.

It's not immediately clear to me why, though, stereo audio works fine with botamusique. I tried fiddling with the sample rate (44.1 kHz instead of 48 kHz), but to no avail.

DuckBoss commented 3 years ago

Wow nice find! You responded as I was replying to your previous post. That is a weird issue since stereo audio is supported by the mumble server, but down-mixed to mono audio if the client doesn't support it (anything below v1.4.0 mumble snapshots).

I'll investigate this further, because that doesn't occur under normal circumstances.

Gert-dev commented 3 years ago

I updated to 5.1.1 to be sure, but it didn't fix it.

I'm also staring at the botamusique ffmpeg code right now to spot differences, because this is indeed really weird. I'm not an audio expert and this is not going to be the issue, but are you sure that halving the sample rate equates to mono? Botamusique sets the -ac parameter to 1 or 2 depending on if it's mono or stereo, whilst JJMumbleBot always sets -ac to 2 and halves the sample rate.

Strange, though, that it's the halved sample rate that is currently working, and the "normal" sample rate - that is equivalent to what botamusique does - isn't :smile:. Is there perhaps some buffer somewhere you allocate yourself that maybe isn't the right size or something (though I'd expect to see errors if that was happening).

EDIT: Also FYI, I was on latest Mumble 1.4.0 already to get access to stereo audio :smile:.

DuckBoss commented 3 years ago

I'm also staring at the botamusique ffmpeg code right now to spot differences, because this is indeed really weird. I'm not an audio expert and this is not going to be the issue, but are you sure that halving the sample rate equates to mono? Botamusique sets the -ac parameter to 1 or 2 depending on if it's mono or stereo, whilst JJMumbleBot always sets -ac to 2 and halves the sample rate.

This was what I had implemented early on in the JJMumbleBot project, but there were some major issues with the sampling when I did it this way. It might have been a bug with vlc at the time nearly 2 years ago, so I'll look at this again and see if I can fix the parameters so it works "the right way". Thanks for bringing this to my attention!

Halving the sampling rate doesn't make it mono, but it fixed a previous issue with vlc during sampling, and it didn't make a difference since mumble was down-mixing to mono anyways. As mentioned earlier, this might have been an issue with vlc early on in development that I never got back to.

Is there perhaps some buffer somewhere you allocate yourself that maybe isn't the right size or something (though I'd expect to see errors if that was happening).

Now that I think about it, I don't use any custom buffers, but I do read a fixed number of bytes from the audio stream which may be too much for a slower ARM processor. Normally I wouldn't consider this to be an issue, but it might be the problem in this case. Thanks for bringing this up, even if it's not the problem, I will make this value configurable by the user.

Specifically these lines: https://github.com/DuckBoss/JJMumbleBot/blob/afee3ede1c99c77d39118ad039201a0144722012/JJMumbleBot/lib/audio/audio_interface.py#L105 https://github.com/DuckBoss/JJMumbleBot/blob/afee3ede1c99c77d39118ad039201a0144722012/JJMumbleBot/lib/audio/audio_interface.py#L97

Gert-dev commented 3 years ago

I tried removing PulseAudio again, but removing it breaks both mono and stereo audio, so it appears to be necessary - even though no sound is ever output on the device itself. This happens for both ffmpeg and vlc.

Is there some option that can be passed to prevent this? Or maybe the bot itself does something specific with audio devices? I don't immediately see any options that may cause this being passed to either ffmpeg and VLC, and ffmpeg itself doesn't need this when botamusique uses it (the options appear to be virtually the same as for this bot).

Next, I tried fiddling with the buffer size and following botamusiques formula: 960 * channel_count (about 1920 for stereo), as well as making it lower than 1024, but to no avail. If I lower it, the bot usually doesn't even unmute.

What's also weird: the only difference between the options of stereo and mono is the 48000 Hz frequency, so I tried setting 24000 for stereo (i.e. enable stereo and set the settings to be exactly the same as mono), and it still didn't work, even though EnableStereoAudio = false works! This implies there is some other part of the code I'm missing that also depends on this setting, but searching only reveals bot_service_helper. I tried setting that to False, but it also has no effect.

DuckBoss commented 3 years ago

I tried removing PulseAudio again, but removing it breaks both mono and stereo audio, so it appears to be necessary - even though no sound is ever output on the device itself. This happens for both ffmpeg and vlc.

That's new to me since all the computers I've tested on display the PulseAudio error, but runs fine.

Or maybe the bot itself does something specific with audio devices? I don't immediately see any options that may cause this being passed to either ffmpeg and VLC, and ffmpeg itself doesn't need this when botamusique uses it (the options appear to be virtually the same as for this bot).

Hmm, the only process handling I do for ffmpeg/vlc is to launch it in a separate thread and set the launch parameters. Apart from that it's pretty much the ffmpeg/vlc processes working individually and piping the output to the bot. Maybe there is some issue with piping the data back to the bot?

Next, I tried fiddling with the buffer size and following botamusiques formula: 960 * channel_count (about 1920 for stereo), as well as making it lower than 1024, but to no avail. If I lower it, the bot usually doesn't even unmute.

The launch parameters for ffmpeg/vlc shouldn't matter as far as unmuting goes. The bot should have unmuted regardless of the data it is receiving from those processes. As seen here: https://github.com/DuckBoss/JJMumbleBot/blob/afee3ede1c99c77d39118ad039201a0144722012/JJMumbleBot/lib/audio/audio_interface.py#L99 And this is the unmute method from the reference above: https://github.com/DuckBoss/JJMumbleBot/blob/afee3ede1c99c77d39118ad039201a0144722012/JJMumbleBot/lib/utils/runtime_utils.py#L89

What's also weird: the only difference between the options of stereo and mono is the 48000 Hz frequency, so I tried setting 24000 for stereo (i.e. enable stereo and set the settings to be exactly the same as mono), and it still didn't work, even though EnableStereoAudio = false works! This implies there is some other part of the code I'm missing that also depends on this setting, but searching only reveals bot_service_helper. I tried setting that to False, but it also has no effect.

I have to look into this further, but I don't think there is anything else that refers to the usage of stereo audio except the launch parameters for ffmpeg/vlc.

Would you by any chance have another computer system to try the bot on? I know it's not ideal since you want it on your Arch ARM system, but it would be helpful for the sake of testing to see if it works fine on your x86 systems. Also, the v1.4 mumble servers/clients have been really buggy (as expected), but I think it might be a mumble issue unless the same audio issues are occurring even on v1.3 servers/clients.

Gert-dev commented 3 years ago

The launch parameters for ffmpeg/vlc shouldn't matter as far as unmuting goes. The bot should have unmuted regardless of the data it is receiving from those processes.

Strange. Perhaps if the buffer is too low, something else goes wrong, causing the playback to abort and the bot to mute itself really quickly again (the soundboard sound I tested is only a second or so). In any case, I think this can be ignored, since it's likely a symptom of something else.

I did some more testing here, but I think the unmuting bricking happens if you try to execute commands too quickly after the bot joins. If I wait a second or two, I can't reproduce this. Though, that would be an uncommon edge case, at best.

Would you by any chance have another computer system to try the bot on? I know it's not ideal since you want it on your Arch ARM system, but it would be helpful for the sake of testing to see if it works fine on your x86 systems.

Yes, I do. I've tested the same configuration on my own computer - which is an x86-64 desktop environment -, and it exposes the same issue, strangely enough. It's an Arch Linux (in case you want to try it out yourself: EndeavorOS might be a quick way to get it installed without having to set it up through the command line installer.)

I've also been fiddling with the audio settings a bit, and I think the sample rate thing "happens" to work because the amount of data pushed is the same. Halving the sample rate halves the amount of samples produced per second, and halving the channel count does the same. If I up the sample rate for mono to 48 kHz and leave the channels at 2, I get a lower-pitched version of the sound file, which makes sense; there is now twice as much data for the same audio, so reading the same amount of data will make it sound like it slows down.

Lowering the channels to 1 for mono and leaving the sample rate at 48 kHz produces the correct sound again, as expected. This is also the same as botamusique does:

    if audio_lib_type.value == AudioLibrary.FFMPEG.value:
        # ...
        params.extend(["-nostdin", "-i", uri, "-ss", f"{skipto}", "-acodec", "pcm_s16le", "-f", "s16le",
                       "-ab", "192k", "-ar", "48000"])
        if stereo:
            params.extend(["-ac", "2", "-threads", "8", "-"])
        else:
            params.extend(["-ac", "1", "-threads", "8", "-"])
    elif audio_lib_type.value == AudioLibrary.VLC.value:
        # ...
        if stereo:
            params.extend(["--sout",
                       "#transcode{acodec=s16le, channels=2, samplerate=48000, ab=192, threads=8}:std{access=file, mux=wav, dst=-}"])
        else:
            params.extend(["--sout",
                       "#transcode{acodec=s16le, channels=1, samplerate=48000, ab=192, threads=8}:std{access=file, mux=wav, dst=-}"])
        params.extend(["vlc://quit"])

But still, stereo refuses to work and only mono works, even though this code is now pretty much the same as botamusique does. I also tried using the exact same buffer size and stripping the -ab 192k, but to no avail.

DuckBoss commented 3 years ago

Yes, I do. I've tested the same configuration on my own computer - which is an x86-64 desktop environment -, and it exposes the same issue, strangely enough. It's an Arch Linux (in case you want to try it out yourself: EndeavorOS might be a quick way to get it installed without having to set it up through the command line installer.)

I might have some time this weekend to set it up as a VM and test it out.

I've also been fiddling with the audio settings a bit, and I think the sample rate thing "happens" to work because the amount of data pushed is the same. Halving the sample rate halves the amount of samples produced per second, and halving the channel count does the same. If I up the sample rate for mono to 48 kHz and leave the channels at 2, I get a lower-pitched version of the sound file, which makes sense; there is now twice as much data for the same audio, so reading the same amount of data will make it sound like it slows down.

Lowering the channels to 1 for mono and leaving the sample rate at 48 kHz produces the correct sound again, as expected. This is also the same as botamusique does:

Yup that sounds right! I won't have time to work on the bot until this weekend, so I'll fix the mono/stereo issue at that time.

But still, stereo refuses to work and only mono works, even though this code is now pretty much the same as botamusique does. I also tried using the exact same buffer size and stripping the -ab 192k, but to no avail.

Hmm, I'm not sure where to go from here, but I will try setting up that EndeaverOS VM (or a command-line Arch install if that doesn't work) this weekend and see if I can reproduce the issue. I'm really hoping it's just some minor issue with the bot, and not an obscure system issue.