ESP32 Audio dev boards support

timotoots commented 1 year ago

Describe the problem you have/What new integration you would like Include support for ESP32 audio dev boards with integrated DAC.

Please describe your use case for this integration and alternatives you've tried: There are several existing ESP32 development sound boards that could be used as media players. For example Espressif Lyra-T, ESP32-A1S Audio Kit, Olimex ESP32-ADF.

Additional context I suspect they need some special commands on boot to make them work. Arduino-audio-tools project has integrated them like this: https://github.com/pschatzmann/arduino-audiokit/tree/main/src/audio_board

spudje commented 1 year ago

I would love to see the ESP32-A1S Audio Kit supported as well, in the same way as the ESPMuse devices are supported.

nagyrobi commented 1 year ago

Work in progress: https://github.com/esphome/esphome/pull/3552

nagyrobi commented 1 year ago

Raspiaudio Muse Luxe, AI-Thinker ESP32-A1S Audio Kit and Espressif LyraT v4.3 are the most popular and they all contain the ES8388 DAC.

This is a great DAC! Because it has a built-in mixer and multiple outputs too:

has two analog stereo inputs (one L/R pair with a switchable microphone preamp)
has implicitly an I2S DAC
has two analog outputs
the inputs can be mixed with the DAC signal to the outputs

ESP32 Audio Kit with the ESP32-A1S module is a wonderful board because:

it has small onboard microphones ready on one of the stereo input pair
it has a stereo jack line input on the other pair
has a 2x3W onboard amp on one of the output pair
has a stereo jack headphone on the other output pair
has a GPIO connected to the headphone jack for plug detection

This DAC could solve https://github.com/esphome/feature-requests/issues/1750 and https://github.com/esphome/feature-requests/issues/1751 all in one.

According to ES8388 documentation it's very versatile:

it has separate gain control on the inputs (independently on L/R channels, registers 39, 42)
DAC-only volume (registers 26, 27) and mute (register 29) on L/R
it has separate volume control for the sum (mixed) outputs (registers 46-49)

This would allow us to build a soundbar with ESPHome, where TV sound could be looped through the Line-In towards the speakers, and HA could play announcements mixed with the TV sound. Since the mixer passes audio through there's no delay in the process, which is beneficial to keep lipsync with the TV picture. ESP32 Audio Kit could do this out of the box.

Combined with an IR receiver we could even use the TV's own remote to adjust the audio volume straight here.

And I think the DAC can also digitize the input signal so microphone audio could be streamed to HA for assistant functionality (https://github.com/esphome/feature-requests/issues/1254).

nagyrobi commented 1 year ago

What would be nice to have is:

a select to choose the output (headphones or speakers - we could atomate based on headphones inserted status)
a select to choose the input (jack line-in or onboard mic)
a number to set the input gain volume
a number to set the master output volume
a switch for the master mute
the media player's own volume and mute would correspond to the DAC output volume
a binary_sensor if DAC is playing or not (we could automatically decrease or mute Line-in volume while announcement from HA using ESPHome automations)

Currently trying with this on ESP32-A1S Audio Kit v2.2, but only white noise yet:

external_components:
  - source: github://pr#3552
    components: [es8388]
    refresh: 1000s

es8388:

i2c:
  sda: GPIO33 
  scl: GPIO32
  frequency: 400kHz

media_player:
  - platform: i2s_audio
    name: "ESP32 Audio Kit"
    dac_type: external
    i2s_lrclk_pin: GPIO25
    i2s_dout_pin: GPIO26 
    i2s_bclk_pin: GPIO27
    mode: stereo
    mute_pin:
      number: GPIO21
      inverted: true

switch:
  - platform: gpio
    pin: GPIO21
    name: "AMP Switch / MUTE"
    restore_mode: ALWAYS_ON

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
      mode:
        input: true
    name: "Jack Status"

light:
  - platform: binary
    name: "Test LED 1"
    output: light_output_1
  - platform: binary
    name: "Test LED 2"
    output: light_output_2

output:
  - id: light_output_1
    platform: gpio
    pin: GPIO22
    inverted: true
  - id: light_output_2
    platform: gpio
    pin: GPIO19
    inverted: true

sehraf commented 1 year ago

FYI: some ESP32 AudioKits come with an AC101 https://github.com/Romkabouter/ESP32-Rhasspy-Satellite/blob/df4cd1f90fc39999fb6fe77663e7359b10aef47e/PlatformIO/src/devices/AudioKit.hpp#L67-L77

jazzvivi2 commented 1 year ago

it would be good to create the component for ac101, version 1 of these boards is still sold in many places

sehraf commented 1 year ago

it would be good to create the component for ac101, version 1 of these boards is still sold in many places

headphone output is working https://github.com/sehraf/esphome-components

EDIT: both output and input are working

sehraf commented 1 year ago

If anyone has a working voice assistant pipeline and an AudioKit with AC101, please give https://github.com/sehraf/esphome-components/blob/main/esp32-audiokit-voice-assistant.yaml a try (my pipeline isn't working, whisper crashes)

schmurtzm commented 6 months ago

If anyone has a working voice assistant pipeline and an AudioKit with AC101, please give sehraf/esphome-components@main/esp32-audiokit-voice-assistant.yaml a try (my pipeline isn't working, whisper crashes)

Thanks a lot for your work on it ! It works fine ! I made a post about it here, I duplicate the content here :

Hi, first you have to know that there are at least 7 versions of the A1S Audio Kit. The major difference is a version with AC101 codec and another one with ES8388 but as we can read in this topic there's also some other differences (like the caps on the wrong places sometimes).

About these versions :

Lots of info on the squeezelite-esp32 github repo
This video shows identical cards (same version number!) but with a different DAC: the old ones have an AC101 DAC (and a pigtail on the ESP) and the new ones an ES8388 DAC.
Phil Schatzmann's article on the subject
7 iterations of the boards cited here with their pins
Sometimes the microphone capacitors are soldered in the wrong place (in this topic)

On my side, I have an old A1S board with AC101 codec. Fortunately I've found this issue & this repo. It works well (Thanks Sehraf !), I took inspiration from Muse Luxe configuration to create my own ESP configuration forht ethe A1S with AC101 codec that I share with you :

substitutions:
  friendly_name: ESP32 AudioKit - voice assistant

  # Audiokit Buttons
  KEY1: "36"
  KEY2: "13" # may be in use for other purposes, see onboard config switch
  KEY3: "19" # also activates LED D4 if pressed
  KEY4: "23" # do not use on A1S V2.3 with initial pinout -> I2C ES8388
  KEY5: "18" # do not use on A1S V2.3 with initial pinout -> I2C ES8388
  KEY6: "5"  # do not use on A1S V2.3 with initial pinout -> I2S

  # Audiokit LEDs
  LED_D4: "22"
  LED_D5: ${KEY3}

esphome:
  name: esp32-audio-kit

esp32:
  board: esp-wrover-kit
  framework:
    type: arduino
  # amplifier is not required for headphones
  # on_boot:
  #   then:
  #     - output.turn_on: gpio_amp
external_components:
  source:
    type: git
    url: https://github.com/sehraf/esphome-components
    ref: main
  components: [ac101]
  refresh: 1s
logger:

api:
  encryption:
    key: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  services:
    - service: start_va
      then:
        - voice_assistant.start
    - service: stop_va
      then:
        - voice_assistant.stop

ota:
  password: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

wifi:
  ssid: "MySSID"
  password: "MyWifiPassword"

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "esp32-audio-kit Fallback Hotspot"
    password: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

captive_portal:

#status_led:
#  pin:
#    number: ${LED_D5}
#    inverted: true

web_server:
  port: 80

# AC101 (I2C)
i2c:
  sda: 33
  scl: 32
  scan: true

ac101:
  # address: 0x1a

media_player:
  - platform: i2s_audio
    id: AudioKit
    name: Media Player
    dac_type: external
    i2s_dout_pin: 25
    mode: stereo
    # on_play:
    #   output.turn_on: gpio_amp
    # on_idle:
    #   output.turn_off: gpio_amp

# I2S Audio
i2s_audio:
  i2s_lrclk_pin: 26
  i2s_bclk_pin: 27

microphone:
  - platform: i2s_audio
    adc_type: external
    i2s_din_pin: 35
    pdm: false
    id: mic_id

switch:
  - platform: gpio
    pin: GPIO21
    name: "AMP Switch"
    restore_mode: ALWAYS_ON
  - platform: template
    name: Use Wake Word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led

voice_assistant:
  id: va
  microphone: mic_id
  media_player: AudioKit
  use_wake_word: true
  on_listening:
    - light.turn_on:
        id: top_led
        brightness: 100%
        effect: pulse
  on_tts_start:
    - light.turn_on:
        id: top_led
        effect: none
  on_tts_end:
    - media_player.play_media: !lambda return x;
    - light.turn_on:
        id: top_led
        effect: pulse
  on_client_connected:
    - if:
        condition:
          - switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:
  on_client_disconnected:
    - if:
        condition:
          - switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          media_player.is_playing: AudioKit
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: top_led
        effect: none
    - delay: 1s
    - script.execute: reset_led
    - script.wait: reset_led
    - lambda: |-
        if (code == "wake-provider-missing" || code == "wake-engine-missing") {
          id(use_wake_word).turn_off();
        }

script:
  - id: reset_led
    then:
       - light.turn_off: top_led

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
      mode:
        input: true
    name: "Jack Status"

  - platform: gpio
    pin:
      number: GPIO036
      inverted: true
    name: "Key 1"
    filters:
      - delayed_off: 10ms

  - platform: gpio
    pin:
      number: GPIO013
      inverted: true
    name: "Key 2"
    filters:
      - delayed_off: 10ms

#  - platform: gpio
#    pin:
#      number: GPIO019
#      inverted: true
#    name: "Key 3"
#    filters:
#      - delayed_off: 10ms

  - platform: gpio
    pin:
      number: GPIO023
      inverted: true
      mode:
        input: true
        pullup: true
    name: "Key 4"
    filters:
      - delayed_off: 10ms

  - platform: gpio
    pin:
      number: GPIO018
      inverted: true
      mode:
        input: true
        pullup: true
    name: "Key 5"
    filters:
      - delayed_off: 10ms

  - platform: gpio
    pin: 
      number: GPIO005
      inverted: true
      mode:
        input: true
        pullup: true
    name: "Key 6"
    filters:
     - delayed_off: 10ms

    on_press:
      - voice_assistant.start:
    on_release:
      - voice_assistant.stop:
    on_click:
      - media_player.toggle: AudioKit  

light:
  - platform: binary
    name: "LED D4"
    id: top_led
    output: light_output_1
  - platform: binary
    name: "LED D5"
    output: light_output_2

output:
  - id: light_output_1
    platform: gpio
    pin: GPIO22
    inverted: true
  - id: light_output_2
    platform: gpio
    pin: GPIO19
    inverted: true

It's not perfect but it allows to start to play with Voice Assistant with this great board.

Some issues :

At boot it often enter in a crazy loop on wake word initialization (like this on this topic or this one). Just disable "Use Wake Word" toggle and enable it again to stop this loop. It seems to be related to the "voice_assistant" automations configured. A workaround could be to start Voice Assistant only after few seconds (as it seems related to the I2S microphone not initialized or something like that).
The jack is working however I didn't found how to detect the jack to disable the amp if necessary (pin 39 doesn't seem to work).
GPIO19 is muxed so I didn't found how to use the key3 and the LED D5 simultaneously (ESPHOME refuse to use the same GPIO two times)
The media player is working but require to disable "Use Wake Word" first. May be we could find a trick in HA to disable it automatically when something is played.
Volume is reset to 100% after each restart.

So it can be improved a lot, for example the keys could be mapped for Media Player. I'm sure that the issues could be fixed by someone with a great debugging level on ESPHome

I hope that it will help some of you, once configured the result is good, even if only one microphone is used, the wake word detection is not bad at all even at several meters or by whispering. The media player is working fine, the speakers and the audio jack output too. Even if Voice Assist is far from perfect, I can experiment with my A1S audio board now :wink:

juhanjuku commented 4 months ago

I have tested this board several occasions. My problem is that I can't play ACC+ radio with ESPHome Media Player. As solution I use Squeezelite where all streams are working. Can you confirm that your player works with other formats than MP3?

esphome / feature-requests

ESP32 Audio dev boards support #1882