[HOWTO] getting audio from JACK

nettings commented 4 years ago

I'm trying to get audio from JACK rather than a native ALSA device. Will describe my tests in this issue thread in the hope that they might be useful.

Platform is an RPi 4 with a Hifiberry DAC+ADC card and a Pi camera connected. I'm running a custom spin of Raspbian that is media-optimized, see https://github.com/nettings/medianet.

nettings commented 4 years ago

Setup 1: JACK server, ALSA jack plugin, mod-host, mpv plays files into jack graph. The mod-host is running some EQ plugin in this usecase. You don't need to use mod-host, but you do need to have some persistent jack client in the middle, so that mpv and the alsa jack plugin have something persistent to connect to, since both use ephemeral ports that appear and disappear all the time :(

I'm assuming you already have jack running.

install libasound2-plugins

create ~/.asoundrc, replacing the jack port names below with whatever is appropriate for your setup:

pcm.v4l {
    type jack
    capture_ports {
            0 effect_3:lv2_audio_out_1
            1 effect_3:lv2_audio_out_2
    }
}
pcm.v4ls {
type plug
slave {
    pcm "v4l"
    rate 48000
    format FLOAT_LE
    channels 2
}
}

v4l2rtspserver -W 1280 -H 720 -F 25 -A 48000 -C 2 -a FLOAT_LE /dev/video0,v4ls

Jack is using a fixed sample rate (48k in my case) and a fixed format of FLOAT_LE, but "real" ALSA devices allow the setting of a user-defined sample rate and format. Somehow even if these values are set in v4l2rtspserver, there is an error when opening the device. So we create a slave device v4ls that will always use the v4l device with those settings, while automatically converting to whatever settings v4l2rtspserver wants.
from another machine, tune in to the stream: mpv rtsp://your-host:8554/unicast
on the jack host, make sure you have some audio content going in, for example by playing some files: mpv --ao=jack --jack-port=effect_3:in.* *

This is not an optimal solution. Using a plug-type slave device adds load, latency and fragility. It should be possible to find settings that work directly with a jack-plugin device. But after hours of trying, I needed to hear some sound. :-D to be continued...

nettings commented 4 years ago

I notice I can get lower latency withmplayer -nocache -xy 300 -geometry 80%:20% rtsp://mn-door:8554/unicast. The stream is very choppy for a few seconds in the beginning, then settles down stably. With both players I experience minor audio jumps, but they sound reasonably discrete, no glitch noise.

nettings commented 4 years ago

Found that in order to be able to run v4l2rtspserver as a systemd service, it is necessary to define the alsa devices globally in /etc/asound.conf rather than in ~/.asoundrc.

nettings commented 3 years ago

Damn. It says so clearly in the README: v4l2rtspserver does not do float samples:

an Video4Linux device that support H264, HEVC, JPEG, VP8 or VP9 capture.
an ALSA device that support PCM S16_BE, S16_LE, S32_BE or S32_LE

That's why I can't get it to work without a plughw: device.

nettings commented 3 years ago

FWIW, an mpv player contiuously tells me

AV: 00:43:41 / 00:43:43 (100%) A-V: -0.255 ct:  0.612
Invalid audio PTS: 2621.941917 -> 2622.630708

while I'm playing a stream from the Pi cam with JACK audio via alsa-jack plugin.

mpromonet / v4l2rtspserver

[HOWTO] getting audio from JACK #195