search

xoseperez / espurna

Home automation firmware for ESP8266-based devices
http://tinkerman.cat
GNU General Public License v3.0
2.99k stars 637 forks source link

Problem in websocket with DHT [WEBSOCKET] #9 error(1002): Invalid UTF-8 in text frame #2163

Closed karmacoma92 closed 4 years ago

karmacoma92 commented 4 years ago

Before creating a new issue please check that you have:

Fulfilling this template will help developers and contributors to address the issue. Try to be as specific and extensive as possible. If the information provided is not enough the issue will likely be closed.

You can now remove this line and the above ones. Text in italic is meant to be replaced by your own words. If any of the sections below are not relevant to the issue (for instance, the screenshots) then you can delete them.

Bug description [WEBSOCKET] #9 error(1002): Invalid UTF-8 in text frame

I'm testing now with 2.6.3. I do not have set WIFI_... erase.config/ERASE.CONFIG does not help, -DPIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK221 doesn't help either. I could only change the password, no wifi, nothing. Relays doesn't seem to have the issue but I haven't tested any more

Steps to reproduce You have to run esptool.py erase_flash and then flash the latest build 1.14.2-dev afterwards, or
flash directly latest 1.14.2-dev build on top of previous version from 23/01/20

Expected behavior This is the expected behavior [184195] [WEBSOCKET] #7 connected, ip: 192.168.20.222, url: /ws [184363] [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002): Invalid UTF-8 in text frameevi� � [WEBSOCKET] #7 error(1002) [184396] [WEBSOCKET] #7 disconnected

Screenshots N/A

Device information

[000175] [MAIN] ESPURNA 1.14.2.PA5-dev (200219) [000175] [MAIN] xose.perez@gmail.com [000176] [MAIN] http://tinkerman.cat

[000177] [MAIN] CPU chip ID: 0x8E1600 [000180] [MAIN] CPU frequency: 80 MHz [000184] [MAIN] SDK version: 2.2.2-dev(38a443e) [000188] [MAIN] Core version: 2.6.3 [000191] [MAIN] Core revision: 3d128e5c [000194] [MAIN] Build time: 1582320145 [000198] [000199] [MAIN] Flash chip ID: 0x146085 [000202] [MAIN] Flash speed: 40000000 Hz [000206] [MAIN] Flash mode: DOUT [000208] [000210] [MAIN] Flash size (CHIP) : 1048576 bytes / 256 sectors ( 0 to 255) [000217] [MAIN] Flash size (SDK) : 1048576 bytes / 256 sectors ( 0 to 255) [000224] [MAIN] Reserved : 4096 bytes / 1 sectors ( 0 to 0) [000231] [MAIN] Firmware size : 569392 bytes / 140 sectors ( 1 to 140) [000238] [MAIN] Max OTA size : 450560 bytes / 110 sectors ( 141 to 250) [000245] [MAIN] EEPROM size : 4096 bytes / 1 sectors ( 251 to 251) [000253] [MAIN] Reserved : 16384 bytes / 4 sectors ( 252 to 255) [000260] [000261] [MAIN] EEPROM sectors: 251, 250 [000264] [MAIN] EEPROM current: 251 [000267] [000268] [MAIN] EEPROM: 4096 bytes initially | 1288 bytes used (31%) | 2808 bytes free (68%) [000277] [MAIN] Heap : 40984 bytes initially | 5280 bytes used (12%) | 35704 bytes free (87%) [000285] [MAIN] Stack : 4096 bytes initially | 1232 bytes used (30%) | 2864 bytes free (69%) [000293] [000294] [MAIN] Boot version: 31 [000297] [MAIN] Boot mode: 1 [000300] [MAIN] Last reset reason: External System [000304] [MAIN] Last reset info: Fatal exception:0 flag:6 (EXT_SYS_RST) epc1:0x00000000 epc2:0x00000000 epc3:0x00000000 excvaddr:0x00000000 depc:0x00000000 [000317] [000319] [MAIN] Board: GENERIC_ESP01S_DHT11_10 [000322] [MAIN] Support: ALEXA API BROKER BUTTON DEBUG_SERIAL DEBUG_TELNET DEBUG_WEB DOMOTICZ HOMEASSISTANT LED MDNS_SERVER MQTT NTP SCHEDULER SENSOR TELNET TERMINAL THINGSPEAK WEB [000338] [MAIN] OTA: ARDUINO ASYNCTCP WEB [000342] [MAIN] Sensors: DHTXX [000345] [MAIN] WebUI image: SENSOR [000348] [000349] [MAIN] Power: 2475 mV [000351] [MAIN] WiFi Sleep Mode: MODEM [000355]

---8<-------

Additional context Add any other context about the problem here.

firmware.zip

tcpdump.zip

mcspr commented 4 years ago

Thanks!

Looking at the last message, before the UTF-8 error:

{"pwrUnits":0,"eneUnits":0,"tmpUnits":0,"tmpCorrection":-9.5,"humCorrection":22,"snsRead":6,"snsReport":10,"snsSave":0,"magnitudesConfig":{"index":[0,0],"type":[1,2],
"units":[>>>"°C"<<<,"%"],
"description":["DHT11 @ GPIO2","DHT11 @ GPIO2"],"size":2}}

Can you try changing this unit to just "C"? https://github.com/xoseperez/espurna/blob/8f7d77b2a1247063adc3bf8db04cb38d0426d2a0/code/espurna/sensor.ino#L85

karmacoma92 commented 4 years ago

Hi, you made it work !! thanks! Oscar

mcspr commented 4 years ago

Cool. What I don't get is why .ino file gets corrupted like that when translated into .cpp. Can you check what locale and file code/espurna/sensor.ino output is on that Ubuntu system?

karmacoma92 commented 4 years ago

Hi

here is the output

espurna$ file code/espurna/sensor.ino code/espurna/sensor.ino: C source, UTF-8 Unicode text

espurna$ locale LANG=es_ES.UTF-8 LANGUAGE= LC_CTYPE="es_ES.UTF-8" LC_NUMERIC=en_GB.UTF-8 LC_TIME=en_GB.UTF-8 LC_COLLATE="es_ES.UTF-8" LC_MONETARY=en_GB.UTF-8 LC_MESSAGES="es_ES.UTF-8" LC_PAPER=en_GB.UTF-8 LC_NAME=en_GB.UTF-8 LC_ADDRESS=en_GB.UTF-8 LC_TELEPHONE=en_GB.UTF-8 LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=en_GB.UTF-8 LC_ALL=

as far as I understand there is no problem with that, am I wrong? Thanks

mcspr commented 4 years ago

Just wondering. For some reason °C isn't stored properly in the firmware file. For example, these are .elf contents of the section storing PROGMEM strings:

Yours:

(this is xxd output)
000861e0: 6d00 b567 2f6d b300 6b57 6800 4a00 6b57  m..g/m..kWh.J.kW
000861f0: 0057 0056 0041 0068 5061 0025 00b0 4600  .W.V.A.hPa.%..F.
00086200: b043 0000 5dc3 2740 51c3 2740 48c3 2740  .C..].'@Q.'@H.'@

Mine (built generic env with -DDHT_SUPPORT=1)

00086c30: 7070 6d00 c2b5 672f 6dc2 b300 6b57 6800  ppm...g/m...kWh.
00086c40: 4a00 6b57 0057 0056 0041 0068 5061 0025  J.kW.W.V.A.hPa.%
00086c50: 00c2 b046 00c2 b043 0000 0000 b5cd 2740  ...F...C......'@

°C is stored as C2 B0 43 for me but as B0 43 for you. nm even shows the size difference of the magnitude_celcius in that 1 byte:

$ xtensa-lx106-elf-nm -S issue2616.elf | grep magnitude_cel
4027c200 00000003 t _ZL17magnitude_celsius
$ xtensa-lx106-elf-nm -S mytestfirmware.elf | grep magnitude_cel
4027cc55 00000004 t _ZL17magnitude_celsius

edit: Why I was asking about the locale, this is what happens when converting stuff with pre-unicode era encoding:

>>> b'\xb0\x43'.decode('latin1')
'°C'
>>> b'\xb0\x43'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
>>> b'\xc2\xb0\x43'.decode('utf-8')
'°C'
mcspr commented 4 years ago

IDE & version PlatformIO Core 4.2 and PIO Home 3.1

Based on my previous comment about decoding / encoding being wrong, I tried installing PIO with python2 and check what exactly happens with .ino conversion:

$ ~/pio2/bin/pio run -e itead-sonoff-basic-dht
...
Converting espurna.ino
Unicode decode error has occurred, please remove invalid (non-ASCII or non-UTF8) characters from /home/max/Projects/espurna/code/espurna/espurna.ino.cpp file
^Z
[1]+  Stopped                 ~/pio2/bin/pio run -e itead-sonoff-basic-dht
$ file espurna/espurna.ino.cpp
espurna/espurna.ino.cpp: C source, ISO-8859 text

Same thing with python3 version:

$ file espurna/espurna.ino.cpp
espurna/espurna.ino.cpp: C source, UTF-8 Unicode text

Since python2 is EOL, I'm inclined to leave things as they are.

$ ~/pio2/bin/pip install -U
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
karmacoma92 commented 4 years ago

hi,

do you think there might be something wrong in my FW file?

Oscar

mcspr commented 4 years ago

Yes. To fix the issue completely you would need to change PIO installation from python 2 to python 3. There is a workaround though, but I'd rather incentivise py3 migration.

karmacoma92 commented 4 years ago

The same happens with generic-esp01s-ds18b20-10-ota, I believe it does make sense since it is ºC...

I know I'm having the one installed with pip2 pip -V pip 20.0.2 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)

how would you go from pip2 to pip3? Thanks Oscar

edit: add missing 'you'

mcspr commented 4 years ago

Use pip3 / python3 -mpip and follow https://docs.platformio.org/en/latest/installation.html? I would assume Ubuntu comes with recent py3 version already installed, pip3 might be missing but is probably in some additional apt-get'able package

karmacoma92 commented 4 years ago

how about this:

`[268971] [MQTT] Received domoticz/out => { "Battery" : 255, "RSSI" : 12, "description" : "", "dtype" : "General", "hwid" : "5", "id" : "00000001", "idx" : 63, [268982] [DOMOTICZ] Error parsing data [268991] [MQTT] Received domoticz/out => "name" : "Cloud Cover", "nvalue" : 0, "stype" : "Percentage", "svalue1" : "47.00", "unit" : 1 }

[268998] [DOMOTICZ] Error parsing data [282142] [SENSOR] DHT22 @ GPIO14 - temperature: 21.6C [282143] [SENSOR] DHT22 @ GPIO14 - humidity: 42% [292409] [MQTT] Received domoticz/out => { "Battery" : 255`

Could it be something related to previous problem? thanks Oscar

mcspr commented 4 years ago

Unlikely. Can you open a separate issue about that? But, based on what I see, Received domoticz/out shows 2 different parts of the JSON object, which were received in two different messages. You can also check what some other MQTT client or do a tcpdump to see raw mqtt packets.

karmacoma92 commented 4 years ago

Ok, I will do that thanks once more Oscar

mcspr commented 4 years ago

Closing via https://github.com/platformio/platformio-core/issues/3393

karmacoma92 commented 4 years ago

Hi, I'm afraid it doesn't work for me, I must say I haven't got the time to upgrade python to version 3, is it having something to do with that?

Thanks Oscar

karmacoma92 commented 4 years ago

I'm afraid I must be doing something wrong

pio upgrade --dev Please wait while upgrading PlatformIO ... PlatformIO has been successfully upgraded to 4.2.2b1 Release notes: https://docs.platformio.org/en/latest/history.html /PlatformIO/Projects/espurna/code$ pio --version PlatformIO, version 4.2.1

Oscar

karmacoma92 commented 4 years ago

Hi I have fixed the issue was happening:

pio --version PlatformIO, version 4.2.2b1

platformio --version PlatformIO, version 4.2.2b1

Apparently now it's working, I'll keep on testing tomorrow thanks! Oscar