Socket server: Unknown command in packed received: [some number]

[jmadotgg]

Bug report

Problem

I get the error "Unknown command in packet received: 31202" when running the ledstrip example with the python audioserver.py sample script. After that the connection is closing and the server as well as the python client keep retrying to establish a connectionn. I've looked into the socket server and the command16 variable is only supposed to have the values 3 or 4, is that right? Steps

1. Install python audioserver.py dependencies, setup correctly with esp32 ip.
2. compile ledstrip example with correct ssid and password in secrets.h
3. Upload example and run python script

Example Is something here not configured correctly?

#!/usr/bin/python
##+--------------------------------------------------------------------------
##
## audioserver - (c) 2023 Dave Plummer.  All Rights Reserved.
##
## File:        audioserver.py - Pushes Audio FFT data to NightDriverStrip
##
## Description:
##
##    Samples the system audio, splits it into bands using an FFT, and then
##    sends is over WiFi to a NightDriverStrip instance
##
## History:     Feb-20-2023     davepl      Created
##
##---------------------------------------------------------------------------

import pyaudio
import numpy as np
import socket
import struct
import time
import sys

# NightDriver ESP32 wifi address - update to your ESP32 WiFi

client = '192.168.178.72'        

# Set up audio input stream. 512@24000 gives a nice framerate.  And 512
# is what I run on the ESP32 if connected via hardware mic, so at least it matches

chunk_size   = 512
sample_rate  = 44100
max_freq     = 20000
num_bands    = 12

# Correction I apply to get a mostly linear response across the bands.  

if num_bands == 16:
    band_scalars = [ 0.35, 0.20, 0.125, 0.1, 0.5, 1.2, 1.7, 2.0, 2.1, 2.75, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0 ]
else:
    if num_bands == 12:
        band_scalars = [ 1.0, 1.0, 1.0, 1.0, 0.01, 0.01, 0.01, 0.1, 0.1, 0.1, 0.1, 1.0 ]

# Open the audio stream.  I'm reading from the mic here because I could not find a system independent
# way to read from the default output device, which would normally require a loopback instance be 
# installed and for simplicity sake, I don't want that.

p = pyaudio.PyAudio()

#
# Set up FFT parameters:
#

fft_size = 2**12  # Choose the size of the FFT (power of 2 for efficiency)

# Calculate the frequencies corresponding to each FFT bin.  

freqs = np.fft.rfftfreq(fft_size, d=1.0/sample_rate)  

# Divide the frequency range into frequency bands of equal logrithmic width
# `20` is the minimum frequency of human hearing, `max_freq` is the maximum frequency of interest, 
# and `num_bands` is the desired number of frequency bands.  

bands = np.logspace(np.log10(20), np.log10(max_freq), num_bands+1).astype(int)  

# Compute the width of each frequency band.  This returns the detla between band (n, n+1)) for each band

band_widths = np.diff(bands)  # Take the difference between adjacent frequency band limits

# The socket we will open to the ESP32 to send our band data to

print("Connect to " + client + "...")

sock = None

stream = p.open(format=pyaudio.paFloat32, channels=1, rate=sample_rate, input=True, frames_per_buffer=chunk_size)

# Loop to continuously sample audio and compute spectrum analyzer bands
while True:

    # Connect to the socket we will be sending to if its not already connected
    if sock == None:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        address = (client, 49152)
        sock.connect(address)
        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
        sock.setblocking(True);

    # Read the raw audio data.  We ignore overflow exceptions from not accepting every bit, it's ok if
    # miss a few in betweeen samples
    audio_data = np.frombuffer(stream.read(chunk_size, exception_on_overflow=False), dtype=np.float32)

    # Compute the FFT to put the samples into frequency
    fft_data = np.abs(np.fft.rfft(audio_data, n=fft_size))

    # Compute band values
    band_values = []
    for i in range(len(bands)-1):                                       # BUGBUG RANGE stops one before
        band_start = np.searchsorted(freqs, bands[i])
        band_stop = np.searchsorted(freqs, bands[i+1])
        band_value = np.median(fft_data[band_start:band_stop])
        band_values.append(band_value)

    band_values = np.multiply(band_values, band_scalars)

    # Scale band values to [0, 1]
    band_values = np.clip(band_values, 0.000001, None)                  # Avoid div by zero
    max_band_value = np.max(band_values) + 0.000001;                    # Avoid zero maximumum
    if (max_band_value > 1):
        scaled_values = np.divide(band_values, max_band_value)
    else:
        scaled_values = band_values

    # Convert scaled values to ASCII bargraph
    bargraphs = []
    for scaled_value in scaled_values:
        asterisks = "*" * int(round(scaled_value * 8))
        bargraph = f"{asterisks:<8}"
        bargraphs.append(bargraph)

    # Print ASCII bargraphs
    #breakpoint()
    print(bargraphs)
    sys.stdout.write('*')

    # Compose and send the PEAKDATA packet to be sent to the ESP32 NightDriverStrip instance

    packed_data = struct.pack('f' * len(scaled_values), *scaled_values)
    command = 4
    length32 = 4 * num_bands
    seconds = int(time.time())
    micros  = time.perf_counter() - seconds

    header1 = (command).to_bytes(2, byteorder='little')             # Offset 0, command16
    header2 = (num_bands).to_bytes(2, byteorder='little')           # Offset 2, num_bands
    header3 = (length32).to_bytes(4, byteorder='little')            # Offset 4, length32 
    header4 = (seconds).to_bytes(8, byteorder='little')             # Offset 8, seconds
    header5 = struct.pack('d', micros)                              # Offset 16, micros

    complete_packet = header1 + header2 + header3 + header4 + header5 + packed_data

    try:
        sock.send(complete_packet)
    except socket.error as e:
        #breakpoint()
        print("Socket error!");
        sock.close()
        sock = None

    time.sleep(0.015);

sock.close()

The globals.h section for the ledstrip demo:

#elif LEDSTRIP

    // The LED strips I use for Christmas lights under my eaves

    #ifndef PROJECT_NAME
    #define PROJECT_NAME            "Ledstrip"
    #endif

    #ifndef ENABLE_WEBSERVER
    #define ENABLE_WEBSERVER            0   // Turn on the internal webserver
    #endif

    #define ENABLE_WIFI                 1   // Connect to WiFi
    #define INCOMING_WIFI_ENABLED       1   // Accepting incoming color data and commands

    #define WAIT_FOR_WIFI               1   // Hold in setup until we have WiFi - for strips without effects
    #define TIME_BEFORE_LOCAL           5   // How many seconds before the lamp times out and shows local content
    #define COLORDATA_SERVER_ENABLED    1   // Also provides a response packet
    #define NUM_CHANNELS    1
    #define MATRIX_WIDTH    (1*144)     // My maximum run, and about all you can do at 30fps
    #define MATRIX_HEIGHT   1
    #define NUM_LEDS        (MATRIX_WIDTH * MATRIX_HEIGHT)
    #define ENABLE_REMOTE   0                     // IR Remote Control
    #define ENABLE_AUDIO    0                     // Listen for audio from the microphone and process it

    #ifndef LED_PIN0
        #define LED_PIN0        5
    #endif

    #define DEFAULT_EFFECT_INTERVAL     (1000*20)

    #define RING_SIZE_0 1
    #define RING_SIZE_1 2
    #define RING_SIZE_2 4
    #define RING_SIZE_3 8
    #define RING_SIZE_4 16

Notes I have connected to led strips to port 5 of my esp32.

Thank you very much for your help! :)

[jmadotgg]

I cannot seem to figure out the reason why it does not work.

[jmadotgg]

I cannot seem to get this to work. When I do not set ENABLE_AUDIO to 1 it does not process the peak_data, right? Hence the example audioserver.py won't work? But if I do set it to to 1 it tries to read from a non existing mic. What is the correct way to continue further? Thank you in advance.

[robertlipe]

31202 is 0x79E2, so it's not like it's wrong wndian or off one byte.

I do lots of builds without enable audio set and no mic attached. Mine never spawn that adc thread or socketswrvwr. Threads or block on them.

Just debug it methodically. Does a scope show a solid sine wave when you whistle to it? Do you have the types of boards that are actually supported (I think the list is long in M1 stuff and short on "solder and $1 mic to pins X and Y". Does your audio gear and the config.h mess March? (E.g., not soldering an i2s mix to code reading analong inputs...) Does something resembling an audio wave appear in that big buffer if you graph it with gplot or Numbers or whatever? Then proceed upstream: is the stream of audio bytes. Basically sane but perhaps missing a synchronization primitive or printing an wrong thing in the debug message or something?

There's just not a lot of collected knowledge that I know of about audio debugging here.

On Sun, Nov 17, 2024, 3:02 PM Julius @.***> wrote:

I cannot seem to get this to work. When I do not set ENABLE_AUDIO to 1 it does not process the peak_data, right? Hence the example audioserver.py won't work? But if I do set it to to 1 it tries to read from a non existing mic. What is the correct way to continue further? Thank you in advance.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/666#issuecomment-2481552457, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD3YDA25ENWTA5SG2JB32BD77RAVCNFSM6AAAAABRPGVV5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBRGU2TENBVG4 . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>