Audio player disconnects from headphones after first loop

[riban-bw]

Describe the bug Playing an audio recording with loop play enabled will play once to both system and headphones output but the next loop only plays to system output.

To Reproduce Steps to reproduce the behavior:

1. Within "Admin" menu enable "RBPi Headphones"
2. Navigate to "Audio Recorder" view
3. Enable "Loop Play"
4. Start an audio file playing

Expected behaviour Audio plays to both main output and headphones.

Actual behaviour Audio plays to both main output and headphones for first loop then only main output for subsequent loops.

Setup:

• HARDWARE Raspberry Pi 3 Model B Plus Rev 1.3 Audio: HifiBerry DAC+ ADC PRO Display: ZynScreen 3.5 (v1) Wiring: MCP23017_ZynScreen I2C: MCP23017@0x20 Profile: z2_v1
• SYSTEM Raspbian GNU/Linux 10 (buster) Build Date: 2021-09-23 Memory: 21% (202M/923M) SD Card: 60% (16G/29G) Temperature: 51.0ºC Overclock: Disabled
• MIDI Profile: default.sh Tuning: 440 Hz Receive Mode: Stage (Omni On) ZS3 Sub-SnapShots: Off MIDI to Output: On Master Channel: off
• SOFTWARE zyncoder: testing (e236925) zynthian-ui: masterfx (37377b0) zynthian-sys: testing (a5e66e4) zynthian-data: testing (10e1f9f) zynthian-webconf: testing (729cc9b)

Additional context I see that after each loop the instance of MPlayer is destroyed and a new instance created which clears the jack routing. Maybe the audio player could remain active throughout the playback which might also improve playback performance.

[riban-bw]

On masterfx branch the disconnection occurs immediately after connection even if loop play is disabled.

[jofemodo]

AFAIK, nothing we can do regarding the "loop" flag. MPlayer works like this. It's a pitty. We should look for a better alternative. Something with time-stretching, tuning and good looping features. Something that we can integrate on the zynpad. A sampler/looper engine.

[riban-bw]

look for a better alternative

Yes we should but in the meantime we have an issue with audio connection to headphones not being re-established after first loop.

[jofemodo]

Of course, but MPlayer disconnect on every loop and this can't be changed easily. We could check every second (or less) while playing and re-make the connection but it wouldn't prevent the disconnection/connection lapsus. We should think of replacing the player.

[riban-bw]

The command to launch mplayer has an error. -ao jack noconnect should be -ao jack:noconnect. Because of this, mplayer is actually making the connection to system:playback. Zynthian autoconnect disconnects this and routes to the desired outputs but when loop occurs mplayer reinstantiates its jack client and reconnects to system:playback. Zynthian has not signal this has happened so cannot correct the routing.

I tried to put mplayer in a loop within the thread, calling autoroute but this wasn't robust. I started to look at starting mplayer paused then controlling its playback via commands but this is suboptimal so I stopped as this would be a plaster over a gaping wound.

Let's quietly ignore this issue and build a more robust audio player, maybe within an engine so it can be put in a chain. (Yes there is one already so maybe extend that.)

[riban-bw]

I will work on replacing mplayer with a player that works better with Zynthian and embed it in an engine so it appears on a mixer channel. A lot of this work is already done so just need to finish that off and implement a way to trigger recordings...

[jofemodo]

Sincerely, i wouldn't put too much effort on this. I would replace MPlayer by Sox, adding a volume control to the Mixer's strip and "ciao pescao". In the next iteration, after chainification, we will work in Audio&MIDI sampling/recording/playing/looping capabilities. We should integrate time-stretching and pitch-shifting out-the box. Every body is using this currently and young people consider this a "must have".

[riban-bw]

I tried lots of media players and each had issues so I wrote one. zynaudioplayer is a library (and Python wrapper) which can load a wav, ogg, etc. file, play / loop it, measure duration, etc. I think it is mostly working and integrated as an engine which workes well allowing effects to be added and mixer integration. It uses bank as the folder selection (Internal / USB) and preset as the file selection.

Stuff yet to do:

• ~Decode mp3~ have replaced mp3 with flac
• ~Change filenames~ rename and delete implemented
• Update controls when state changes, e.g. end of playback should change "Playing" to "Stopped" (@jofemodo I need help!)
• ~Set position~ Not ideal but can now set position within file
• Display song position
• Edit files (top & tail, normalise)

[jofemodo]

Hi @riban-bw !

Writing a library for playing audio files shouldn't be our business. I don't like the idea of reinventing wheels. Have you tried something more simple, like this?

# apt-get isntall vlc vlc-plugin-jack
# pip3 install python-vlc

The python code is pretty simple and we can play any file type, "loop" using set_time(), and the better, it's perfectly debugged and stable. Please, simply test it ;-)

import vlc
instance = vlc.Instance('--aout=jack')
# Here we could re-make jack connections as we want
p = instance.media_player_new()
m = instance.media_new("/zynthian/zynthian-data/audio/test.mp3")
p.set_media(m)
p.play() 
vlc.libvlc_audio_set_volume(p, 90)
p.pause()
p.play()
p.get_length()
p.set_time(289765)
p.set_time(189765)
p.pause

The full API docs are here:

https://www.olivieraubert.net/vlc/python-ctypes/doc/

Also interesting for catching/managing events from the library/player:

https://stackoverflow.com/questions/3595649/vlc-python-eventmanager-callback-type

Regards,

[jofemodo]

Add this to catch some events as example:

events = player.event_manager()

def SongFinished(event):
  print("THE END")

events.event_attach(vlc.EventType.MediaPlayerEndReached, SongFinished)

def pos_callback(event, player):
  sec = player.get_time() / 1000
  m, s = divmod(sec, 60)
  npos = 100000*sec/player.get_length()
  sys.stdout.write('\r%s %02d:%02d (%.2f%%)' % ('Position', m, s, npos))
  sys.stdout.flush()

events.event_attach(vlc.EventType.MediaPlayerPositionChanged, pos_callback, p)

[riban-bw]

I completely agree that we should not be reinventing the wheel. I wrote this library last year when I couldn't find a suitable replacement. Last week I spent a lot of time testing lots of off-the-shelf options but none did what we need (which is pretty simple). Often the options were bloated, providing too many features and obsfucating the stuff useful to us. Often they used excessive memory and/or processing. Many disconnected audio during looping and/or automatically (re)connected audio routing. I will check out VLC and also continue to develop (fix) my library - seeing as it is pretty much working now.

[riban-bw]

I tried vlc and it has these issues that I am yet to resolve:

• Player only appears in jack graph whilst playing
• Auto connects to system output
• After playback, can't get to play again (seems to play but does not reappear in jack graph and does not route)
• Hangs python and can't be closed with normal signals
• Uses quite a lot of memory
• Few examples of using with jack
• Documentation may be full but not very explanatory

I'm sure with enough time we could drill into the code and figure out (some of) these issues but I don't want to spend all my time trying to work out how to use someone's code which may lead to failure. I will fix the issues with my lib but if someone wants to get vlc working then great!

[jofemodo]

The jack connection problem is not difficult to solve:

import vlc
instance = vlc.Instance("--aout=jack --jack-name=zynvlc --jack-connect-regex=zynmixer:input_17")
m = instance.media_new("/zynthian/zynthian-data/audio/test.mp3")
p = instance.media_player_new()
p.set_media(m)
p.play()

After playback reach the end, you have to call this function again:

p.set_media(m)

But you can get looping by not leaving to reach the end, catching "MediaPlayerPositionChanged" event and using set_time(). I will make some tests with this.

Regarding the memory usage, i don't think 5% is too high, moreover when normally we are sitting in a pile of unused memory. This is more or less the same memory used by any synth engine (ZynAddSubFX, for instance)

I will do some researching with gstreamer too.

Regards!

[jofemodo]

BTW, i can't reproduce the "python hang up" issue. Could you point me a little bit with this?

[jofemodo]

Hi @riban-bw !

I just did some testing with gstreamer and i've to say that i like it. It's super modular and allows a high degree of control . Take a look to the code:

First install the needed packages:

# apt-get install python3-gst-1.0 gstreamer1.0-tools gstreamer1.0-fluendo-mp3 gstreamer1.0-plugins-good

And here comes the code:

#!/usr/bin/python3

import os
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst

Gst.init(None)

player = Gst.Pipeline.new("player")
source = Gst.ElementFactory.make("filesrc", "file-source")
decoder = Gst.ElementFactory.make("flump3dec", "mp3-decoder")
conv = Gst.ElementFactory.make("audioconvert", "converter")
sink = Gst.ElementFactory.make("jackaudiosink", "jack-output")
sink.set_property("port-pattern", "zynmixer:input_17")

player.add(source)
player.add(decoder)
player.add(conv)
player.add(sink)
source.link(decoder)
decoder.link(conv)
conv.link(sink)

def on_message(bus, message):
  global player
  t = message.type
  if t == Gst.MessageType.EOS:
    player.seek_simple(Gst.Format.TIME, Gst.SeekFlags.FLUSH, 0)
  elif t == Gst.MessageType.ERROR:
    player.set_state(Gst.State.NULL)
    err, debug = message.parse_error()
    print("Error: %s" % err, debug)

bus = player.get_bus()
bus.add_signal_watch()
bus.connect("message", on_message)

filepath = "/zynthian/zynthian-data/audio/test.mp3"
source.set_property("location", filepath)

player.set_state(Gst.State.PLAYING)
...
player.set_state(Gst.State.PAUSED)

[riban-bw]

I did try gstreamer and had issues (I think with pre-connecting, reconnecting on loop, etc.) I just tried again and restarting playback replays a fraction of audio before starting from start (may be how I triggered restart). I also got:

>>> source.set_property("location", "/zynthian/zynthian-my-data/capture/test-2.wav")
>>> player.set_state(Gst.State.PLAYING)
<enum GST_STATE_CHANGE_ASYNC of type Gst.StateChangeReturn>
>>> Segmentation fault

So - it is not necessarily true that all this code is fully debugged and robust! This was just a few minutes into testing. This seems to have been triggered by playing a wav file through a mp3 decoder. There is more work to do to get a seemless, play-anything behaviour.

The earlier lock-up with vlc may have been my test code looping but not resetting to start.

[riban-bw]

Okay - I have tried and tired and tried again so many off-the-shelf players and each has issues so I went back to my library and got zynaudioplayer working. It works really well now:

• Several instances may be started so you can use Zynthian like a DJ
• Handles all file formats (codec, bitrate, storage type, samplerate) that libsndfile supports*
• Soft mute when stopping to avoid clicks (I have not implemented during start to avoid clipping front of audio)
• Can loop at end of file (future enhancement to loop section)
• Recue to start if play to end else will play from position it was stopped (equivalent to pause)
• All features exposed in Python library
• Provides access to file metadata
• Can set playback gain (volume)
• Can get info without explicitly opening files, e.g. duration, metadata, etc. (does actually open file but only momentarily to read header)
• Samplerate converts files to jackd samplerate (can adjust quality of SRC which impacts resource usage)
• Handles multi-channel files - by default plays first two channels as stereo (or mono to both for a mono file) but you can select which pair of (consecutive) channels to playback or playback a mix of all channels (odd to left, even to right). This is implemented in the library but not yet exposed in the UI

*The current version of libsndfile in Debian (buster and bullseye) does not support mp3 but the latest version 1.1.0 does. I have compiled this and it works a treat so we should add this as a dpkg.

There are a few functions that I want to break out of the python class structure so that they may be called without instantiating an object, like _get_fileduration but that isn't urgent as our use will always instantiate an instance.

This is working so well that it is almost ready as a replacement for mplayer.

I would like help from @jofemodo on linking controls. I have got it working by implementing MIDI control within the player then assigning default MIDI controls in the engine control screen but there may be elements that we don't want default MIDI control, only UI control. Also need to update the UI from player events like stop playback, position, etc. HELP!

I have added controls in the chain context menu for controlling the recorder. This allows the recorder to be started/stopped from this context menu. It also allows individual chains to be primed in which case just these chains are recorded as stereo pairs to a multi-channel wav file. (Default is to record main output.)

[riban-bw]

Zynthian audio player is now implemented. I suggest merging into testing. Will close this issue.