judahpaul16 / gpt-home

ChatGPT at home! Basically a better Google Nest Hub or Amazon Alexa home assistant. Built on the Raspberry Pi using the OpenAI API.
https://hub.docker.com/r/judahpaul/gpt-home
GNU General Public License v3.0
435 stars 41 forks source link

Can't get microphone to pick up when using USB speakers and USB microphone #56

Closed hiro24 closed 3 months ago

hiro24 commented 3 months ago

I'm using a USB sound device to connect a couple small speakers (https://a.co/d/05CPIVYZ) and a USB microphone, mostly due to the long cord (https://a.co/d/00JrdQhD) and for the life of me I can't get the microphone to pick up.

I'm able to hear sound with aplay on both the host and in the container, and can record using arecord. I've had to modify my /etc/asound.conf to get to this point. This is currently what I have:

# Output device (hw:1)
pcm.!default {
    type plug
    slave.pcm {
        type hw
        card 1
    }
}

ctl.!default {
    type hw
    card 1
}

# Input device (hw:2)
pcm.input {
    type plug
    slave.pcm {
        type hw
        card 2
    }
}

ctl.input {
    type hw
    card 2
}

Here's the output of aplay -l

**** List of PLAYBACK Hardware Devices ****
card 0: Headphones [bcm2835 Headphones], device 0: bcm2835 Headphones [bcm2835 Headphones]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 1: Device [USB PnP Audio Device], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

And the output of arecord -l

**** List of CAPTURE Hardware Devices ****
card 1: Device [USB PnP Audio Device], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 2: Microphone [USB Lavalier Microphone], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

I've tried modifying common.py line: engine.setProperty('alsa_device', 'hw:Headphones,0') to a few different things. I'ved tried hw:Microphone,0 and plughw:2,0. Neither have worked. The logs repeat over and over

INFO:common:Could not understand audio, waiting for a new phrase...

I'm hitting up against a brick wall at this point. I'm not sure what to try next. Any ideas?

judahpaul16 commented 3 months ago

Try this asound.conf:

# The default PCM (Pulse Code Modulation) device configuration
pcm.!default {
    # 'asym' allows specifying different devices for playback and capture
    type asym
    playback.pcm {
        # 'plug' enables automatic format conversion if necessary
        type plug
        slave.pcm "hw:1,0"  # Sets the playback device to card 1, device 0 (USB PnP Audio Device)
    }
    capture.pcm {
        # 'plug' enables automatic format conversion if necessary
        type plug
        slave.pcm "hw:2,0"  # Sets the capture device to card 2, device 0 (USB Lavalier Microphone)
    }
}

# The default control interface configuration
ctl.!default {
    type hw  # Directly accesses hardware controls
    card 1   # Uses card 1 (USB PnP Audio Device) for control settings
}

then test it:

# Test recording with arecord using the configured default capture device
arecord -f cd test.wav

# Test playback with aplay using the configured default playback device
aplay test.wav

If this works try updating the engine property in common.py to engine.setProperty('alsa_device', 'plughw:2,0') and restarting the docker container:

docker restart gpt-home

If after all this it still doesn't work then send a snippet of event log right after restarting the main app with:

docker exec -it gpt-home supervisorctl restart app
hiro24 commented 3 months ago

I'm not able to use arecord with that command because the device is in use. I installed lsof in the container and tracked it to app.py. Using supervisorctl I stopped app. What I've noticed is that despite having rebooted the container, arecord -f cd seems to want to use the onboard microphone of the USB device that the speakers are hooked up to. This is probably why it's not working, because that's pretty covered up and super muffled. (and why I got the long cabled USB mic).

In the host if I run arecord -f cd test.wav it seems to use the USB microphone. In the container, it uses the USB soundcard w/ the onboard muffled mic. Which is odd, b/c they're both using the same asound.conf file.

hiro24 commented 3 months ago

Oh, and here's a snippet from events.log after a restart.

DEBUG:asyncio:Using selector: EpollSelector
DEBUG:common:Failed to initialize display, skipping...
 Reason: No Hardware I2C on (scl,sda)=(3, 2)
Valid I2C ports: ((1, 3, 2), (0, 1, 0), (10, 45, 44))
Traceback (most recent call last):
  File "/app/src/common.py", line 111, in initLCD
    i2c = busio.I2C(SCL, SDA)
          ^^^^^^^^^^^^^^^^^^^
  File "/env/lib/python3.11/site-packages/busio.py", line 35, in __init__
    self.init(scl, sda, frequency)
  File "/env/lib/python3.11/site-packages/busio.py", line 157, in init
    raise ValueError(
ValueError: No Hardware I2C on (scl,sda)=(3, 2)
Valid I2C ports: ((1, 3, 2), (0, 1, 0), (10, 45, 44))

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): www.google.com:80
DEBUG:urllib3.connectionpool:http://www.google.com:80 "GET / HTTP/1.1" 200 8520
DEBUG:common:Failed to initialize display, skipping...
 Reason: No Hardware I2C on (scl,sda)=(3, 2)
Valid I2C ports: ((1, 3, 2), (0, 1, 0), (10, 45, 44))
Traceback (most recent call last):
  File "/app/src/common.py", line 111, in initLCD
    i2c = busio.I2C(SCL, SDA)
          ^^^^^^^^^^^^^^^^^^^
  File "/env/lib/python3.11/site-packages/busio.py", line 35, in __init__
    self.init(scl, sda, frequency)
  File "/env/lib/python3.11/site-packages/busio.py", line 157, in init
    raise ValueError(
ValueError: No Hardware I2C on (scl,sda)=(3, 2)
Valid I2C ports: ((1, 3, 2), (0, 1, 0), (10, 45, 44))

INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
INFO:common:Timed out, waiting for phrase to start...
hiro24 commented 3 months ago

It looks like I may have got it going. I'm not entirely sure how. I did have pulseaudio installed and I removed it. Don't know if that was interfering. I also copied /etc/asound.conf to ~/.asoundrc of both the host user home directory and the container root directory. engine.setProperty('alsa_device') is set to plughw:2,0 atm.

judahpaul16 commented 3 months ago

It looks like I may have got it going. I'm not entirely sure how. I did have pulseaudio installed and I removed it. Don't know if that was interfering. I also copied /etc/asound.conf to ~/.asoundrc of both the host user home directory and the container root directory. engine.setProperty('alsa_device') is set to plughw:2,0 atm.

That is odd. Was pulseaudio installed in the docker container or on the host?

hiro24 commented 3 months ago

It was on the host. It was for something else on the pi I was messing with, and again not sure if it was the culprit.

judahpaul16 commented 3 months ago

Probably was pulseaudio. ~/.asoundrc wouldn't be an issue because that's a user specific alsa configuration and the docker container has no alsa configurations by default. asound.conf is mounted to the docker container so that if it changes on the host, it should reflect in the container.

# mount flag within `docker run` command
--mount type=bind,source=/etc/asound.conf,target=/etc/asound.conf

PulseAudio is a sound server that sits on top of ALSA (Advanced Linux Sound Architecture). It can manage ALSA devices and redirect sound streams, which can potentially override configurations set in asound.conf.