OpenInterpreter / 01

The #1 open-source voice interface for desktop, mobile, and ESP32 chips.
https://01.openinterpreter.com/
GNU Affero General Public License v3.0
4.92k stars 517 forks source link

Server and device is connected but don't recieve M5Echo data #184

Open hpsaturn opened 6 months ago

hpsaturn commented 6 months ago

Summary

Server and device is connected and running ok without exceptions, also with Ollama or with OPENAI API key, but not happening anything, the server don't receive M5Echo data, only reaches the first ping and the socket is connected, but I only have:

Cannot call "receive" once a disconnect message has been received.

To Reproduce

Steps to reproduce the behavior:

  1. flashed lastest firmware: lastest (builded with PlatformIO)
  2. export OPENAI_API_KEY=sk...
  3. Server running with: poetry run 01 --server --expose
  4. provisioning the device using the captive portal
  5. connection ok (the server received the ping request from the device)
  6. push the button on the M5 device and talk
  7. not happen anything

Expected behavior Something in the speaker from the IA.

Screenshots screenshot20240329_005125

Device output:

12:00:26.867 > Captive Test, V0.5.0 compiled Mar 29 2024 01:40:54 by CD_FER
12:00:26.871 > ESP32-PICO-D4-1
E (413) wifi_netif: esp_wifi_internal_reg_rxcb for if=1 failed with 12289
12:00:26.887 > E (413) wifi_init_default: esp_wifi_register_if_rxcb for if=0x3ffb4f04 failed with 259
12:00:27.011 > 
12:00:27.011 > Startup TiM5Atom initializing...OK
12:01:04.874 > Wifi scan complete: 19 . WIFI_SCAN_RUNNING: -1Done scanning wifi
12:01:04.882 > Served HTML Page
12:01:31.302 > .Connected to Wi-Fi
12:01:31.306 > Served Post connection HTML Page
12:02:08.581 > Connecting to 01OS at dxxx-80-xxx-226-148.ngrok-free.app:80/ping
12:02:08.665 > Started the ping request
12:02:08.765 > Got status code: 200
12:02:08.765 > Setting up websocket to 01OS d192-xx-xxx-226-148.ngrok-free.app:80E (102297) I2S: i2s_driver_uninstall(2048): I2S port 0 has not installed
12:02:08.777 > 
12:02:08.777 > connected to WiFi
12:02:08.781 > InitI2sSpeakerOrMic 1
12:02:08.781 > Content length is: 4
12:02:08.785 > 
12:02:08.785 > Body:
12:02:08.785 > pongWebsocket connection flow completed
12:02:08.789 >  
12:02:08.789 > Connected to 01 websocket!
12:02:08.789 >  
12:02:08.789 > Served success HTML Page
12:02:08.845 > [WSc] Connected to url: /
12:02:19.277 > Recording...
12:02:19.277 > InitI2sSpeakerOrMic 0
12:02:19.281 > Recording ready.
12:02:19.361 > Reading chunk at 0...
12:02:19.373 > Read 1024 bytes in chunk.
12:02:19.377 > Reading chunk at 1024...
12:02:19.409 > Read 1024 bytes in chunk.
12:02:19.409 > Reading chunk at 2048...
12:02:19.437 > Read 1024 bytes in chunk.
12:02:19.441 > Reading chunk at 3072...
12:02:19.473 > Read 1024 bytes in chunk.
12:02:19.473 > Reading chunk at 4096...
12:02:19.501 > Read 1024 bytes in chunk.
12:02:19.505 > Reading chunk at 5120...
12:02:19.537 > Read 1024 bytes in chunk.
12:02:19.537 > Reading chunk at 6144...
12:02:19.565 > Read 1024 bytes in chunk.
12:02:19.569 > Reading chunk at 7168...
12:02:19.601 > Read 1024 bytes in chunk.
12:02:19.601 > Reading chunk at 8192...
12:02:19.629 > Read 1024 bytes in chunk.
12:02:19.633 > Reading chunk at 9216...
12:02:19.665 > Read 1024 bytes in chunk.
12:02:19.665 > [microphone] flushing and sending 10240 bytes of data
12:02:19.725 > Reading chunk at 0...
12:02:19.733 > Read 1024 bytes in chunk.
12:02:19.737 > Reading chunk at 1024...
12:02:19.769 > Read 1024 bytes in chunk.
12:02:19.769 > Reading chunk at 2048...
12:02:19.797 > Read 1024 bytes in chunk.
12:02:19.801 > Reading chunk at 3072...
12:02:19.833 > Read 1024 bytes in chunk.
12:02:19.833 > Reading chunk at 4096...
12:02:19.861 > Read 1024 bytes in chunk.
12:02:19.865 > Reading chunk at 5120...
12:02:19.897 > Read 1024 bytes in chunk.
12:02:19.897 > Reading chunk at 6144...
12:02:19.929 > Read 1024 bytes in chunk.
12:02:19.933 > Reading chunk at 7168...
12:02:19.961 > Read 1024 bytes in chunk.
12:02:19.961 > Reading chunk at 8192...
12:02:19.993 > Read 1024 bytes in chunk.
12:02:19.997 > Reading chunk at 9216...
12:02:20.025 > Read 1024 bytes in chunk.
12:02:20.025 > [microphone] flushing and sending 10240 bytes of data
12:02:20.089 > Reading chunk at 0...
12:02:20.101 > Read 1024 bytes in chunk.
12:02:20.105 > Reading chunk at 1024...
12:02:20.133 > Read 1024 bytes in chunk.
12:02:20.133 > Reading chunk at 2048...
12:02:20.165 > Read 1024 bytes in chunk.
12:02:20.169 > Reading chunk at 3072...
12:02:20.197 > Read 1024 bytes in chunk.
12:02:20.197 > Reading chunk at 4096...
12:02:20.229 > Read 1024 bytes in chunk.
12:02:20.233 > Reading chunk at 5120...
12:02:20.261 > Read 1024 bytes in chunk.
12:02:20.261 > Reading chunk at 6144...
12:02:20.293 > Read 1024 bytes in chunk.
12:02:20.297 > Reading chunk at 7168...
12:02:20.321 > Stopped recording.
12:02:20.325 > Read 1024 bytes in chunk.
12:02:20.325 > Reading chunk at 8192...
12:02:20.329 > [microphone] flushing and sending 8192 bytes of data
12:02:20.357 > Read 1024 bytes in chunk.

Server output

poetry run 01 --server --expose

○              

Starting...                                                                                                                                                                   

Exposing server to the internet...                                                                                                                                            

INFO:     Started server process [3356134]
INFO:     Waiting for application startup.

Ready.                                                                                                                                                                        

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:10001 (Press CTRL+C to quit)

Your server is being hosted at the following URL: https://dxx-80-xxx-236-148.ngrok-free.app
INFO:     80.xx.xx.148:0 - "GET /ping HTTP/1.1" 200 OK
INFO:     ('80.xx.xx.148', 0) - "WebSocket /" [accepted]
INFO:     connection open
Cannot call "receive" once a disconnect message has been received.

Desktop (please complete the following information):

Additional context

Also I don't have a success test running poetry run 01, I have ALSA errors, but I think that this has not relation with this issue, because here I'm running the server.. I guess. Right?

tyfiero commented 6 months ago

Huh... we have been seeing this error: "Cannot call "receive" once a disconnect message has been received." frequently, which I believe is due to the device disconnecting before it receives the data. We will look into this issue and update you here with what we find

hpsaturn commented 6 months ago

is weird, because is clear that the device is connected, because when you stop the server, the firmware send many messages of disconnection, not before to this. But, I understand you, maybe is a internal disconnection?

hpsaturn commented 6 months ago

I built the firmware using CORE_DEBUG_LEVEL=4 to catch more logs, and I found maybe some weird:

21:30:07.066 > [ 53498][D][WiFiClient.cpp:536] connected(): Disconnected: RES: 0, ERR: 128

Also we have a I2S error:

21:30:07.062 > E (53780) I2S: i2s_driver_uninstall(2048): I2S port 0 has not installed

The complete output is here:

21:29:46.470 > [ 32911][D][WiFiGeneric.cpp:1039] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
21:29:46.478 > [ 32911][W][WiFiGeneric.cpp:1061] _eventCallback(): Reason: 202 - AUTH_FAIL
21:29:46.486 > [ 32915][D][WiFiGeneric.cpp:1081] _eventCallback(): WiFi Reconnect Running
21:29:46.542 > [ 32985][D][WiFiGeneric.cpp:1039] _eventCallback(): Arduino Event: 4 - STA_CONNECTED
21:29:46.566 > [ 33007][D][WiFiGeneric.cpp:1039] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
21:29:46.574 > [ 33007][D][WiFiGeneric.cpp:1102] _eventCallback(): STA IP: 192.168.178.131, MASK: 255.255.255.0, GW: 192.168.178.1
21:29:47.458 > .Connected to Wi-Fi
21:29:47.462 > Served Post connection HTML Page
21:30:06.890 > Connecting to 01OS at xxxx-80-xxx-236-148.ngrok-free.app:80/ping
21:30:06.950 > Started the ping request
21:30:07.050 > Got status code: 200
21:30:07.054 > Setting up websocket to 01OS xxxx-80-xxx-236-148.ngrok-free.app:80
21:30:07.058 > connected to WiFi
21:30:07.062 > E (53780) I2S: i2s_driver_uninstall(2048): I2S port 0 has not installed
21:30:07.066 > [ 53498][D][WiFiClient.cpp:536] connected(): Disconnected: RES: 0, ERR: 128
21:30:07.074 > InitI2sSpeakerOrMic 1
21:30:07.074 > Content length is: 4
21:30:07.078 > 
21:30:07.078 > Body:
21:30:07.078 > pongWebsocket connection flow completed
21:30:07.082 >  
21:30:07.082 > Connected to 01 websocket!
21:30:07.086 >  
21:30:07.086 > Served success HTML Page
21:30:07.138 > [WSc] Connected to url: /