antonvh / LMS-uart-esp

Documentation and example code for extending MINDSTORMS hubs with i2c and uart devices using the LMS-uart-convertor board and LMS-esp-i2c-wifi board.
https://antonsmindstorms.com/product/uart-breakout-board-for-spike-and-ev3-openmv-compatible/
BSD 3-Clause "New" or "Revised" License
20 stars 2 forks source link

Unable to run Spectrum Demo #15

Closed Gun-neR closed 1 year ago

Gun-neR commented 1 year ago

Hello. New to the LMS-ESP32, and relitivly new to micropython... but not having fun with this combo so far :(

After much futzing about, I think I have finaly managed to get communication with the BlogExamples spectrum demo, but on the Mindstorms App (RIS) it keeps tossing out this error:

Error specerr Command failed: 'module' object has no attribute 'spectrogram'

leading me to wonder if something has changed with the ULAB or something? since this was published... but that is just a random guess on my part :P

antonvh commented 1 year ago

Sorry to hear that it's been a bit hard. Have you tried flashing it with the latest ulab firmware here? https://firmware.antonsmindstorms.com/

Gun-neR commented 1 year ago

Hello. Just recieved a brand new 3 pack of the pro. Shows as running: MicroPython v1.19.1 on 2022-10-12; ESP32 module (lvgl,ulab,spiram) with ESP32

antonvh commented 1 year ago

OK. Tonight I have time to check what is going on. Could you provide me with the link/exact steps you followed so I can reproduce the error? Do you have a display connected to the ESP32?

Gun-neR commented 1 year ago

No display, just running Thonny off of Windows PC to LMS-ESP32. The HUB is suposed to act as the display with it's LED matrix... thus show the interaction between the two devices.

Installed spectrum_lego.py to the ESP

https://github.com/antonvh/LMS-uart-esp/tree/main/Projects/BlogExamples/lms-esp32

from ulab import numpy as np
from ulab import scipy as spy
import time
from uartremote import *

SCK_PIN = 33
SD_PIN = 32
WS_PIN = 27

I2S_ID = 0

alpha = 0.997

WAV_SAMPLE_SIZE_IN_BITS = 16

FORMAT = I2S.MONO
SAMPLE_RATE_IN_HZ = 5000 # up to 2500Hz, enough for voice
BUFFER_LENGTH_IN_BYTES = 4192

audio_in = I2S(I2S_ID,
             sck = Pin(SCK_PIN),
             ws = Pin(WS_PIN),
             sd = Pin(SD_PIN),
             mode = I2S.RX,
             bits = 16,
             format = FORMAT,
             rate = SAMPLE_RATE_IN_HZ,
             ibuf = BUFFER_LENGTH_IN_BYTES)

def hamming(N):
    # hamming filter for removing sharp artefacts of finite sampling signal
    n = np.linspace(0,N,num=N)
    w = 0.54-0.46*np.cos(2*3.1415*n/N)
    return w

# initialize 
#neop=neopixel.NeoPixel(Pin(21),64)

def level(l):
    for i in range(8):
        if i<l:
            neop[i]=(5*i,5*(7-i),0)
        else:
            neop[i]=(0,0,0)
        neop.write()

def led_xy(x,y,col):
    iy=y%8
    neop[iy*8+x]=col

def led_power(q):
    m=3000 #max(q)
    f=m/8
    for i,qi in enumerate(q):
        l=qi
        if l>m:
            l=m
        iy=int(l/f)
        for ii in range(iy):
            led_xy(i,ii,(30,0,0))
        for ii in range(iy,8):
            led_xy(i,ii,(0,0,0))
    neop.write()

mic_samples = bytearray(256)
mic_samples_mv = memoryview(mic_samples) # efficient pointer to original array

# we use efficient ulab functions (written in C) 
m_sq=np.zeros(5)
m_sq_prev=np.zeros(5)
alpha=0.1

def spec():
    global m_sq,m_sq_prev
    num_bytes_read_from_mic = audio_in.readinto(mic_samples_mv)
    # interpret raw buffer to signed int16 array
    q=np.frombuffer(mic_samples,dtype=np.int16)
    # multiply each array elemelt with hamming windows
    q=q*hamming(128)
    q=q-np.mean(q) # get rid of DC component
    # perform FFT and take abolute value of complex numbers
    z=spy.signal.spectrogram(q)
    # reshape only positive frequency elements in n bins
    zs=z[:60].reshape((5,12))
    # calculate sum of each bin
    sq=np.sum(zs,axis=1)
    #m_sq=np.maximum(sq,m_sq) # take maximum per array element
    m_sq=alpha*sq+(1-alpha)*m_sq_prev
    m_sq=np.maximum(sq,m_sq)
    m_sq_prev=m_sq
    return tuple(list(sq.tolist()+m_sq.tolist()))
    #return tuple(sq)

def add_commands(ur):
    ur.add_command(spec,'10f')

ur=UartRemote()
add_commands(ur)
ur.loop()

Installed plot_freq.py into available spot on the HUB - adjusted the port for A as that is what the ESP is plugged into.

https://github.com/antonvh/LMS-uart-esp/tree/main/Projects/BlogExamples/SPIKE

from projects.uartremote import *
import hub
import time

ur=UartRemote('A')

# import module spectrum_lego on remote side
# this makes the function 'spec' available to call from this side
ur.add_module('spectrum_lego')

m=5000

def show_display(f):
# displays power spectrum as array of 5, with next 5 values the dynamic range value
# LEDS for power spectrum power = 8, dyn range power = 9
    for i in range(5):
        ff=max(f[i],m)
        n=int(f[i]/ff*6)-1 # skip 0
        nm=int(f[i+5]/ff*6)-1
        for j in range(n):
            hub.display.pixel(i,5-j,8) # reverse display (5-j)
        hub.display.pixel(i,5-nm,9)

# clear display
hub.display.show(' ')

t=time.ticks_ms()
i=0
f=[0]*10
while True:
    try:
        show_display(f)
        ack,f=ur.call('spec')
        hub.display.show(' ')
        time.sleep_ms(10)# add small delay for stability
        i+=1
        if i==60:
            # show ms per frame
            #print(time.ticks_diff(time.ticks_ms(),t)/60.)
            t=time.ticks_ms()
            i=0
    except KeyboardInterrupt:
        print('interrupted!')
        break
    except:
        print("Error",ack,f)

I can tell it is at least communicating, as the errors change if I mess around with the ESP code at the line where that z variable is (Line 98 on my setup):

# perform FFT and take abolute value of complex numbers
    z=spy.signal.spectrogram(q)

I must say, that between the bleeping issues getting the uartremote.mpy into the HUB without it always wanting to update the firmware?!?! Keep trying it the same way and eventualy it works. Go fig.

And I find the instructions for this LMS-ESP board are spread all over withing your site, github and YouTube... Thus this has been a bit more complicated then I hoped... and I am not new to microcontrollers.

I just need a functioning interaction between this ESP and the HUB to help me better understand the needed code as I read the documentation (I am more a hands on learner) and thought this particular demo would be the best as it didn't require any additional hardware... as the mic was part of the kit I ordered anyway. But perhaps it wasn't the right choice of demo afterall :)

Gun-neR commented 1 year ago

Oh, I also have main.py on the ESP... still not sure if that is nessesary or not but don't want to mess with it yet again until I get the missing attribute 'spectrogram' figured out.

from uartremote import *
ur=UartRemote()         # initialize uartremote on default uart and default uart pins
ur.loop()  
Gun-neR commented 1 year ago

Sorry it took so long... As stated, I find the info about this cool board spread all around your website and GitHub and thus hard to keep track of what info is where... but that is just me :)

Anyhow, I got the links updated in posts above... and this was the video that got me started down this rabbit hole :P

https://antonsmindstorms.com/2022/02/13/hook-up-any-i2c-sensor-hobby-servo-and-more-to-your-lego-robot-with-the-esp32-board/#a-spectrum-analyzer-with-a-digital-mems-microphone

Gun-neR commented 1 year ago

BTW, Thanks for looking into this for me. Meanwhile I just noticed that you are working on a PyBricks option for the LMS-ESP32... This makes me happy as I am somewhat unimpressed with LEGO's software and their recent EOL of the Mindstorms lineup... as I just picked up two of the RI 51515 sets.

Meanwhile, I just ordered some of your RGB strips... looking forward to lighting up my Lego robots with programable LEDs :D

antonvh commented 1 year ago

Sorry about the docs. They are so much work! Still, it's incredible how far you got with the chaotic info! I ran some tests and concluded you did everything right. If you see the message about the missing spectrogram, that means that communication between spike and esp works!

The problem is that the 'spectrogram' method is absent from ulab.scipy.signal in the ESP firmware. It was there in the past.

This is something for @ste7anste7an to fix. Can you comment, @ste7anste7an ?

ste7anste7an commented 1 year ago

@Gun-neR You are right, the spectrogram is removed from the ulab.scipy.signal. Fortunately it is moved to ulab.utils.spectrogram. Therefore, I suggest you add

import ulab

at the beginning of the lms-esp32/spectrum_lego.py file and replace: spy.signal.spectrogram(q) with ulab.utils.spectrogram(q). Coming days I can not test it myself. I will test it this weekend.

Sorry that the documentation is spread all over github and youtube. We started putting all the docs together in a single spot: https://docs.antonsmindstorms.com/en/latest/ . Eventually this should become the place where you will find all the documentation. We are still working on it.

Gun-neR commented 1 year ago

Hello @ste7anste7an Thank you for fast result... It Verks Now!!! :)

2023-03-06 16 01 19

I found your other tutorials and am currently going through them... I didn't realize I could access the LEGO HUB directly with Thonny... nice to know :)

Gun-neR commented 1 year ago

Update... well... it works for about 5-20 seconds, then the ESP side seems to lock up, requiring a reset, and stop/start on the LEGO side. But as that was not the topic of this issue, I will leave it closed and carry on with my overall experimenting with this fun little LMS-ESP32 board. Thanks to all!

antonvh commented 1 year ago

I have the same: the ESP locks up on my side too. We'll have to investigate this. Maybe a memory leak?

antonvh commented 1 year ago

About the lock-up: I tried putting everything from spectrum_lego into main.py. The program ran a bit longer but still crashed eventually.

ste7anste7an commented 1 year ago

It works very stable on my hub and LMS-ESP32. These are the steps I took:

Hope this helps for you guys to get it work in a stable way.

ste7anste7an commented 1 year ago

2nd test.

Then I changed line 35 to: time.sleep_ms(50)

Now it seems to work more stable (it runs now for a few minutes). I think the serial connection gets congested with the lower delay values.

ste7anste7an commented 1 year ago

Now it still crashes after a few minutes.

3rd test:

It crashed eventually. When I restart the program on the Lego hub, I do not need to restart the LMS-ESP32, indicating that that side is OK.

4th test

5 th test:

Conclusion:

Gun-neR commented 1 year ago

@ste7anste7an Thanks for the info.

Based on this, I simply added...

time.sleep(.05)

... just in front of...

return tuple(list(sq.tolist()+m_sq.tolist()))

... in spectrum_lego.py on the LMS-ESP32

And then left it listening to YouTube music, as my voice was getting hoarse :P And so far it hasn't quit (about an hour and still going as I type this).

Since I don't plan on making a dedicated LEGO VU type meter for my music system... hah!... it should be just fine, however long it lasts, for my needs.

Gun-neR commented 1 year ago

Update... @ste7anste7an

Wow, I had forgot about the running program, watched a movie and didn't notice until now (about three hours have since passed) that the Spectrum Demo is still running!!

Now, I have no idea how/if a 50ms delay in the ESP processing affects the actual spectrogram... as I never had it running long enough prior to really see how it works with different sounds :P But it still seems to flash lots of lines of LEDs just fine and I can sorta see different results with high and low frequency's, so I guess all is still fine.

I might experiment with less delay time, down to the 10ms you tried, and see if there is a better value to use.

Gun-neR commented 1 year ago

FYI. 10ms died in seconds. 20ms was unknown timeframe but wasn't running many hours later. But 30ms is still going strong at 10+ hours.

antonvh commented 1 year ago

Now the challenge is to detect in the code when this happens and reset the connection... I suspect the LEGO python virtual machine or the Soft UART buffer management. However, 30ms (+/- 30Hz) is acceptable for almost all applications.

Gun-neR commented 1 year ago

...and still going just short of a solid day.

I am way too noobie with Python/Micropython/Mindstorms, but I do recall how Arduino SoftwareSerial had limitations when pushing data "too-much-too-fast". I'll leave the rest of this particular issue to the experts, whilst I next try to tie my LEGO/ESP32 into my MQTT server :D

ste7anste7an commented 1 year ago

You might have a look at this code: https://github.com/ste7anste7an/lego_mqtt It is far from a complete example , but gives you some hints in using MQTT on the LMS-ESP32 board.

Gun-neR commented 1 year ago

FYI. After running straight for last 4+days with that 30ms delay in the ESP code, it finally stopped sometime in last 8 hours.

This help in determining if a memory leak or something?

antonvh commented 1 year ago

Thanks a lot for the test! We have a buffer congestion issue. I'll try to stabilize that...