Unable to run Spectrum Demo

[Gun-neR]

Hello. New to the LMS-ESP32, and relitivly new to micropython... but not having fun with this combo so far :(

After much futzing about, I think I have finaly managed to get communication with the BlogExamples spectrum demo, but on the Mindstorms App (RIS) it keeps tossing out this error:

Error specerr Command failed: 'module' object has no attribute 'spectrogram'

leading me to wonder if something has changed with the ULAB or something? since this was published... but that is just a random guess on my part :P

[antonvh]

Sorry to hear that it's been a bit hard. Have you tried flashing it with the latest ulab firmware here? https://firmware.antonsmindstorms.com/

[Gun-neR]

Hello. Just recieved a brand new 3 pack of the pro. Shows as running: MicroPython v1.19.1 on 2022-10-12; ESP32 module (lvgl,ulab,spiram) with ESP32

[antonvh]

OK. Tonight I have time to check what is going on. Could you provide me with the link/exact steps you followed so I can reproduce the error? Do you have a display connected to the ESP32?

[Gun-neR]

No display, just running Thonny off of Windows PC to LMS-ESP32. The HUB is suposed to act as the display with it's LED matrix... thus show the interaction between the two devices.

Installed spectrum_lego.py to the ESP

https://github.com/antonvh/LMS-uart-esp/tree/main/Projects/BlogExamples/lms-esp32

from ulab import numpy as np
from ulab import scipy as spy
import time
from uartremote import *

SCK_PIN = 33
SD_PIN = 32
WS_PIN = 27

I2S_ID = 0

alpha = 0.997

WAV_SAMPLE_SIZE_IN_BITS = 16

FORMAT = I2S.MONO
SAMPLE_RATE_IN_HZ = 5000 # up to 2500Hz, enough for voice
BUFFER_LENGTH_IN_BYTES = 4192

audio_in = I2S(I2S_ID,
             sck = Pin(SCK_PIN),
             ws = Pin(WS_PIN),
             sd = Pin(SD_PIN),
             mode = I2S.RX,
             bits = 16,
             format = FORMAT,
             rate = SAMPLE_RATE_IN_HZ,
             ibuf = BUFFER_LENGTH_IN_BYTES)

def hamming(N):
    # hamming filter for removing sharp artefacts of finite sampling signal
    n = np.linspace(0,N,num=N)
    w = 0.54-0.46*np.cos(2*3.1415*n/N)
    return w

# initialize 
#neop=neopixel.NeoPixel(Pin(21),64)

def level(l):
    for i in range(8):
        if i<l:
            neop[i]=(5*i,5*(7-i),0)
        else:
            neop[i]=(0,0,0)
        neop.write()

def led_xy(x,y,col):
    iy=y%8
    neop[iy*8+x]=col

def led_power(q):
    m=3000 #max(q)
    f=m/8
    for i,qi in enumerate(q):
        l=qi
        if l>m:
            l=m
        iy=int(l/f)
        for ii in range(iy):
            led_xy(i,ii,(30,0,0))
        for ii in range(iy,8):
            led_xy(i,ii,(0,0,0))
    neop.write()

mic_samples = bytearray(256)
mic_samples_mv = memoryview(mic_samples) # efficient pointer to original array

# we use efficient ulab functions (written in C) 
m_sq=np.zeros(5)
m_sq_prev=np.zeros(5)
alpha=0.1

def spec():
    global m_sq,m_sq_prev
    num_bytes_read_from_mic = audio_in.readinto(mic_samples_mv)
    # interpret raw buffer to signed int16 array
    q=np.frombuffer(mic_samples,dtype=np.int16)
    # multiply each array elemelt with hamming windows
    q=q*hamming(128)
    q=q-np.mean(q) # get rid of DC component
    # perform FFT and take abolute value of complex numbers
    z=spy.signal.spectrogram(q)
    # reshape only positive frequency elements in n bins
    zs=z[:60].reshape((5,12))
    # calculate sum of each bin
    sq=np.sum(zs,axis=1)
    #m_sq=np.maximum(sq,m_sq) # take maximum per array element
    m_sq=alpha*sq+(1-alpha)*m_sq_prev
    m_sq=np.maximum(sq,m_sq)
    m_sq_prev=m_sq
    return tuple(list(sq.tolist()+m_sq.tolist()))
    #return tuple(sq)

def add_commands(ur):
    ur.add_command(spec,'10f')

ur=UartRemote()
add_commands(ur)
ur.loop()

Installed plot_freq.py into available spot on the HUB - adjusted the port for A as that is what the ESP is plugged into.

https://github.com/antonvh/LMS-uart-esp/tree/main/Projects/BlogExamples/SPIKE

from projects.uartremote import *
import hub
import time

ur=UartRemote('A')

# import module spectrum_lego on remote side
# this makes the function 'spec' available to call from this side
ur.add_module('spectrum_lego')

m=5000

def show_display(f):
# displays power spectrum as array of 5, with next 5 values the dynamic range value
# LEDS for power spectrum power = 8, dyn range power = 9
    for i in range(5):
        ff=max(f[i],m)
        n=int(f[i]/ff*6)-1 # skip 0
        nm=int(f[i+5]/ff*6)-1
        for j in range(n):
            hub.display.pixel(i,5-j,8) # reverse display (5-j)
        hub.display.pixel(i,5-nm,9)

# clear display
hub.display.show(' ')

t=time.ticks_ms()
i=0
f=[0]*10
while True:
    try:
        show_display(f)
        ack,f=ur.call('spec')
        hub.display.show(' ')
        time.sleep_ms(10)# add small delay for stability
        i+=1
        if i==60:
            # show ms per frame
            #print(time.ticks_diff(time.ticks_ms(),t)/60.)
            t=time.ticks_ms()
            i=0
    except KeyboardInterrupt:
        print('interrupted!')
        break
    except:
        print("Error",ack,f)

I can tell it is at least communicating, as the errors change if I mess around with the ESP code at the line where that z variable is (Line 98 on my setup):

# perform FFT and take abolute value of complex numbers
    z=spy.signal.spectrogram(q)

I must say, that between the bleeping issues getting the uartremote.mpy into the HUB without it always wanting to update the firmware?!?! Keep trying it the same way and eventualy it works. Go fig.

And I find the instructions for this LMS-ESP board are spread all over withing your site, github and YouTube... Thus this has been a bit more complicated then I hoped... and I am not new to microcontrollers.

I just need a functioning interaction between this ESP and the HUB to help me better understand the needed code as I read the documentation (I am more a hands on learner) and thought this particular demo would be the best as it didn't require any additional hardware... as the mic was part of the kit I ordered anyway. But perhaps it wasn't the right choice of demo afterall :)

[Gun-neR]

Oh, I also have main.py on the ESP... still not sure if that is nessesary or not but don't want to mess with it yet again until I get the missing attribute 'spectrogram' figured out.

from uartremote import *
ur=UartRemote()         # initialize uartremote on default uart and default uart pins
ur.loop()  

[Gun-neR]

Sorry it took so long... As stated, I find the info about this cool board spread all around your website and GitHub and thus hard to keep track of what info is where... but that is just me :)

Anyhow, I got the links updated in posts above... and this was the video that got me started down this rabbit hole :P

https://antonsmindstorms.com/2022/02/13/hook-up-any-i2c-sensor-hobby-servo-and-more-to-your-lego-robot-with-the-esp32-board/#a-spectrum-analyzer-with-a-digital-mems-microphone

[Gun-neR]

BTW, Thanks for looking into this for me. Meanwhile I just noticed that you are working on a PyBricks option for the LMS-ESP32... This makes me happy as I am somewhat unimpressed with LEGO's software and their recent EOL of the Mindstorms lineup... as I just picked up two of the RI 51515 sets.

Meanwhile, I just ordered some of your RGB strips... looking forward to lighting up my Lego robots with programable LEDs :D

[antonvh]

Sorry about the docs. They are so much work! Still, it's incredible how far you got with the chaotic info! I ran some tests and concluded you did everything right. If you see the message about the missing spectrogram, that means that communication between spike and esp works!

The problem is that the 'spectrogram' method is absent from ulab.scipy.signal in the ESP firmware. It was there in the past.

This is something for @ste7anste7an to fix. Can you comment, @ste7anste7an ?

[ste7anste7an]

@Gun-neR You are right, the spectrogram is removed from the ulab.scipy.signal. Fortunately it is moved to ulab.utils.spectrogram. Therefore, I suggest you add

import ulab

at the beginning of the lms-esp32/spectrum_lego.py file and replace: spy.signal.spectrogram(q) with ulab.utils.spectrogram(q). Coming days I can not test it myself. I will test it this weekend.

Sorry that the documentation is spread all over github and youtube. We started putting all the docs together in a single spot: https://docs.antonsmindstorms.com/en/latest/ . Eventually this should become the place where you will find all the documentation. We are still working on it.

[Gun-neR]

Hello @ste7anste7an Thank you for fast result... It Verks Now!!! :)

I found your other tutorials and am currently going through them... I didn't realize I could access the LEGO HUB directly with Thonny... nice to know :)

[Gun-neR]

Update... well... it works for about 5-20 seconds, then the ESP side seems to lock up, requiring a reset, and stop/start on the LEGO side. But as that was not the topic of this issue, I will leave it closed and carry on with my overall experimenting with this fun little LMS-ESP32 board. Thanks to all!

[antonvh]

I have the same: the ESP locks up on my side too. We'll have to investigate this. Maybe a memory leak?

[antonvh]

About the lock-up: I tried putting everything from spectrum_lego into main.py. The program ran a bit longer but still crashed eventually.

[ste7anste7an]

It works very stable on my hub and LMS-ESP32. These are the steps I took:

• I updated the green mindstorms hub to the latest Mindstorms firmware (I was running pybricks).
• I installed the uartremote library on the lego hub using this file: https://github.com/antonvh/UartRemote/blob/master/MicroPython/SPIKE/install_uartremote.py)
• I took an LMS-ESP32 with the latest firmware
• I connected the LMS-ESP32 to the Lego hub on Port B
• I used 2 seperate Tonny sessions: one for the Lego hub and one for LMS-ESP32
• I first loaded the LMS-ESP spectrum_lego.py file in thonny, and copy and pasted that in the REPL of the LMS-ESP32
• then, I loaded the lego SPIKE example plot_freq.py in thonny and copy and past the whole code in the REPL of the Lego hub thonny instantance.
• now, it runs for almost 10 minutes while I type this message

Hope this helps for you guys to get it work in a stable way.

[ste7anste7an]

2nd test.

• Loaded the main.py and the altered (changed z=ulab.utils.spectrogram(q))spectrum_lego.py` on the LMS-ESP32
• reset the LMS-ESP32 (the main.py loads at boot)
• opened Mindstroms app and loaded the plot_freq.py in the Mindstorm app
• started the app. Works for approx 20 seconds

Then I changed line 35 to: time.sleep_ms(50)

Now it seems to work more stable (it runs now for a few minutes). I think the serial connection gets congested with the lower delay values.

[ste7anste7an]

Now it still crashes after a few minutes.

3rd test:

• loaded the Mindtorms app in one of the programs (in my case '0') .
• Started the program from the hub by selecting 0 on the hub and start it.

It crashed eventually. When I restart the program on the Lego hub, I do not need to restart the LMS-ESP32, indicating that that side is OK.

4th test

• Running the program on the hub without having the Mindstorms app open (just standalone).
• chrashes eventually (only hub side)

5 th test:

• No changes on LMS-ESP32 side (when pressing reset button code gets executed)
• on Mindstorms I loaded the plot_freq.py code (with 10ms delay in line 35 using Thonny by selecting the python code and pasting it in the Lego hub repl.
• now it keeps working for over 30 minutes

Conclusion:

• The LMS-ESP32 code is fine and stable
• The Mindstorms app software causes some problem. But which problem?

[Gun-neR]

@ste7anste7an Thanks for the info.

Based on this, I simply added...

time.sleep(.05)

... just in front of...

return tuple(list(sq.tolist()+m_sq.tolist()))

... in spectrum_lego.py on the LMS-ESP32

And then left it listening to YouTube music, as my voice was getting hoarse :P And so far it hasn't quit (about an hour and still going as I type this).

Since I don't plan on making a dedicated LEGO VU type meter for my music system... hah!... it should be just fine, however long it lasts, for my needs.

[Gun-neR]

Update... @ste7anste7an

Wow, I had forgot about the running program, watched a movie and didn't notice until now (about three hours have since passed) that the Spectrum Demo is still running!!

Now, I have no idea how/if a 50ms delay in the ESP processing affects the actual spectrogram... as I never had it running long enough prior to really see how it works with different sounds :P But it still seems to flash lots of lines of LEDs just fine and I can sorta see different results with high and low frequency's, so I guess all is still fine.

I might experiment with less delay time, down to the 10ms you tried, and see if there is a better value to use.

[Gun-neR]

FYI. 10ms died in seconds. 20ms was unknown timeframe but wasn't running many hours later. But 30ms is still going strong at 10+ hours.

[antonvh]

Now the challenge is to detect in the code when this happens and reset the connection... I suspect the LEGO python virtual machine or the Soft UART buffer management. However, 30ms (+/- 30Hz) is acceptable for almost all applications.

[Gun-neR]

...and still going just short of a solid day.

I am way too noobie with Python/Micropython/Mindstorms, but I do recall how Arduino SoftwareSerial had limitations when pushing data "too-much-too-fast". I'll leave the rest of this particular issue to the experts, whilst I next try to tie my LEGO/ESP32 into my MQTT server :D

[ste7anste7an]

You might have a look at this code: https://github.com/ste7anste7an/lego_mqtt It is far from a complete example , but gives you some hints in using MQTT on the LMS-ESP32 board.

[Gun-neR]

FYI. After running straight for last 4+days with that 30ms delay in the ESP code, it finally stopped sometime in last 8 hours.

This help in determining if a memory leak or something?

[antonvh]

Thanks a lot for the test! We have a buffer congestion issue. I'll try to stabilize that...