search

JuiceRescue / juicepassproxy

Proxy UDP requests to/from Juicebox EV chargers to MQTT discoverable by Home Assistant
Apache License 2.0
86 stars 12 forks source link

v07 UnicodeDecodeError invalid continuation byte #111

Open pbjfarm opened 2 weeks ago

pbjfarm commented 2 weeks ago

My JPP docker setup stopped working a couple weeks ago. It seems to be related to a character in the JuiceBox ID that the box is sending...although I haven't changed anything.

This appears to be different than the other issue mentioning UnicodeDecodeError - as that issue was using what appeared to be an encrypted version of the protocol...mine is still v07.

juicepassproxy  | 2024-11-09 18:58:10  DEBUG     [juicebox_mitm] Starting JuiceboxMITM Loop
juicepassproxy  | 2024-11-09 18:58:22  DEBUG     [juicebox_mqtthandler] From JuiceBox: b'0\xc300081509281447541218021146:v07,s0,u6623,V2479,L20710648,S1,T25,M24,m32,t30,i50,e0,f6002,X0,Y0!UF0:\n'
juicepassproxy  | 2024-11-09 18:58:22  ERROR     [__main__] A JuicePass Proxy task failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 1: invalid continuation byte
juicepassproxy  | 2024-11-09 18:58:30  DEBUG     [__main__] jpp_task_list: [<Task finished name='mqtt_handler' coro=<JuiceboxMQTTHandler.start() done, defined at /juicepassproxy/juicebox_mqtthandler.py:320> result=None>, <Task finished name='mitm_handler' coro=<JuiceboxMITM.start() done, defined at /juicepassproxy/juicebox_mitm.py:49> exception=UnicodeDecodeError('utf-8', b'0\xc300081509281447541218021146:v07,s0,u6623,V2479,L20710648,S1,T25,M24,m32,t30,i50,e0,f6002,X0,Y0!UF0:\n', 1, 2, 'invalid continuation byte')>]

Startup just loops until finally erroring out.

ivanfmartinez commented 2 weeks ago

@pbjfarm I think that with my branch the container will not stop, can you test ?

https://github.com/JuiceRescue/juicepassproxy/pull/69

If is just sometimes that it send wrong messages we can just skip, but if your device keeps sending that, maybe we need to get a way to ignore that unexpected character

pbjfarm commented 2 weeks ago

@ivanfmartinez I have your branch running now. It's producing the same error, but is continuing with the Starting JuiceboxMITM Loop...and it did finally exit with:

juicepassproxy  | 2024-11-10 21:33:21  ERROR     [aiorun] Unhandled exception; stopping loop: 'Task exception was never retrieved'

The weird thing is that the problem ID in the message doesn't quite match my JuiceBox ID.

Sent:  b'0\xc300081509281447541218021146
Mine:       0100081509281447541218021146

I'm not sure if it's gotten corrupted on the JuiceBox, or what? I'm not really sure how to change it either...

ivanfmartinez commented 2 weeks ago

@pbjfarm its TCP/IP which does have checkings, the value will not be corrupted on TCP/IP part, it should be corrupted before going out juicebox device.

It appear to me something hardware related.

Even if I change the \xc3 to 1 on the message you sent on first comment of the issue, the checksum still does not match. Appear that something else on the message are wrong.

To try to understand if there is also another exception that I can catch probably I need a complete log to check. If you dont want to send log here you can send to my email (my github username at gmail) or telegram with same username .

All messages are corrupted or some are correct ?

pbjfarm commented 2 weeks ago

@ivanfmartinez I thought this was UDP...but it does seem like my messages (or at least the ID) are corrupted somehow.

I've now managed to brick my box somehow (it's not showing the local WiFi SSID on reboot), so I won't be able to provide logs until I can figure out what's going on.

ivanfmartinez commented 2 weeks ago

@pbjfarm JB used UDP and UDP is part of TCP/IP.

If the ids are broken in all messages, them is something on the device.

Maybe FalconFour can help with more ideas or maybe someone else on Discord check main page for link https://github.com/JuiceRescue

carrel-gr commented 2 weeks ago

Hi. Not trying to be critical at all. But I'd like to clarify about TCP and UDP. UDP isn't part of TCP/IP. IP is a layer 3 protocol. Both UDP and TCP are layer 4 protocols that run on top of IP. You use one OR the other. TCP establishes a stateful connection to a remote peer and provides reliable stream transport so that higher layer protocols can send data without worrying about issues such as resending data if packets are lost. TCP takes care of that. UDP is a best effort datagram protocol. It is connectionless. It sends a packet and it's done. When using UDP, the higher layer protocol must handle issues such as resending lost packets/data.

While UDP doesn't guarantee packet delivery, it does support a checksum to try to detect any packet corruption. However, this checksum is optional and the implementation can set it to zero to disable it.

UDP seems a logical choice for the JB as it sends a complete message in a single datagram, and even if a packet is lost, the JB is going to resend another soon anyway. TCP would add unnecessary overhead, especially on a busy server. As for the UDP checksum, I have no idea if the JB uses it or not. We could tell from a packet capture. Since the Enel X protocol provides its own checksum, there is a good chance they are not using the UDP checksum.

philipkocanda commented 1 week ago

This just started happening for me one day ago. Here's the error:

juicepassproxy  | 2024-11-15 23:12:07  INFO      [ha_mqtt_discoverable.sensors] Setting Send Command to JuiceBox to Send Command to JuiceBox using hmd/text/JuiceBox/Send-Command-to-JuiceBox/state
juicepassproxy  | 2024-11-15 23:12:12  ERROR     [juicebox_mitm] Not a valid juicebox message |b'0817081001070462465218264504:v08\xc2\xe1_\x01\x00\x01Z&\x17\xa5Wzp\x8c,\x00\xa7\xee/Tf\t\xf4\xe3\xed\xa4\xb6\x88I\x9en\x84a\x83\\\xb7\xc4\xa9RD\xe8%\x0cX\xfe\x93\x96\xa7\xa5\xf3B\x94)\xf7>\xef\x16\x9f\x8d\xa4\xa9\x84\x17sha\x86\xc1\xa9\x8ff\x02/F\x12|C\x0eR\xd5\xf1!\xeeYTE\xdfS/J<\xf5\xcfO\xc2\xa9\xaa\xf5\x98\xff\x85\x82\x9f4\x01\xbeJ\r\xac\xc7\xf0\xb6\xb9\x06,ce.\x9e\x1c\xf0%\x17m\xaf\xe4`\xc5'| 'utf-8' codec can't decode byte 0xc2 in position 32: unexpected end of data
juicepassproxy  | Traceback (most recent call last):
juicepassproxy  |   File "/juicepassproxy/juicebox_message.py", line 144, in juicebox_message_from_bytes
juicepassproxy  |     string = data.decode("utf-8")
juicepassproxy  |              ^^^^^^^^^^^^^^^^^^^^
juicepassproxy  | UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 32: invalid continuation byte
juicepassproxy  |
juicepassproxy  | During handling of the above exception, another exception occurred:
juicepassproxy  |
juicepassproxy  | Traceback (most recent call last):
juicepassproxy  |   File "/juicepassproxy/juicebox_mitm.py", line 156, in _message_decode
juicepassproxy  |     decoded_message = juicebox_message_from_bytes(data)
juicepassproxy  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
juicepassproxy  |   File "/juicepassproxy/juicebox_message.py", line 148, in juicebox_message_from_bytes
juicepassproxy  |     return JuiceboxEncryptedMessage().from_bytes(data)
juicepassproxy  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
juicepassproxy  |   File "/juicepassproxy/juicebox_message.py", line 389, in from_bytes
juicepassproxy  |     string = data[0:33].decode("utf-8")
juicepassproxy  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^
juicepassproxy  | UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 32: unexpected end of data
juicepassproxy  | 2024-11-15 23:12:12  ERROR     [__main__] A JuicePass Proxy task failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 32: invalid continuation byte
juicepassproxy  | Traceback (most recent call last):
juicepassproxy  |   File "/juicepassproxy/juicepassproxy.py", line 581, in main
juicepassproxy  |     await asyncio.gather(
juicepassproxy  |   File "/juicepassproxy/juicebox_mitm.py", line 62, in start
juicepassproxy  |     await self._connect()
juicepassproxy  |   File "/juicepassproxy/juicebox_mitm.py", line 105, in _connect
juicepassproxy  |     self._mitm_loop_task = await self._mitm_loop()
juicepassproxy  |                            ^^^^^^^^^^^^^^^^^^^^^^^
juicepassproxy  |   File "/juicepassproxy/juicebox_mitm.py", line 136, in _mitm_loop
juicepassproxy  |     await self._main_mitm_handler(data, remote_addr)
juicepassproxy  |   File "/juicepassproxy/juicebox_mitm.py", line 213, in _main_mitm_handler
juicepassproxy  |     data = await self._local_mitm_handler(data, decoded_message)
juicepassproxy  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
juicepassproxy  |   File "/juicepassproxy/juicebox_mqtthandler.py", line 649, in local_mitm_handler
juicepassproxy  |     message = await self._basic_message_parse(data)
juicepassproxy  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
juicepassproxy  |   File "/juicepassproxy/juicebox_mqtthandler.py", line 472, in _basic_message_parse
juicepassproxy  |     parts = re.split(r",|!|:", data.decode("utf-8"))
juicepassproxy  |                                ^^^^^^^^^^^^^^^^^^^^
juicepassproxy  | UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 32: invalid continuation byte
juicepassproxy  | 2024-11-15 23:12:25  ERROR     [__main__] Restarting JuicePass Proxy Loop (9)
juicepassproxy  | 2024-11-15 23:12:25  INFO      [juicebox_mqtthandler] max_current: 32
philipkocanda commented 1 week ago

I tried rebooting the juicebox, and I found something interesting.

After issuing the reboot command through the telnet shell (nc -v 10.0.1.230 2000), as soon as the juicebox came online again I briefly saw the following result from the list command:

list
! # Type  Info
# 0 UDPC  juicenet-udp-prod4-exw.enelx.com:9004 (30660)
# 1 HTTP  https://juicenet-prod4.enelx.com/v1/unit/0817081001070462465218264504/keys:443 (20379)
> [2024-11-15 | 22:22:27: Closed: 1]

Note: That HTTP thing wasn't there before and went away after a few seconds.

Checking the response when visiting that URL, I saw this:

{"key":"ef3c41c925ff10174684512b40beace03dd22aeeac5aa2fb97a8af49fcb4d38c","exp_date":"2025-11-14T14:27:30.802Z","err_msg":null}
philipkocanda commented 1 week ago

Found another bit of interesting information. I was using the DNS override method, and I had the juicenet-udp-prod4-exw.enelx.com and directory-api.emotorwerks.com hosts set to the IP of the machine hosting juicepassproxy. I decided to temporarily revert this change to let the juicebox phone home again. This worked and my car started charging again. Without any ability for me to control it though, as it appeared in the Enel X app as "Charger occupied: another user is charging." - whatever that means 🙃

What I saw after rebooting the juicebox again and running list is this:

❯ nc -v 10.0.1.230 2000
Connection to 10.0.1.230 port 2000 [tcp/callbook] succeeded!
[2024-11-15 | 23:09:22: Ready]
> list
list
! # Type  Info
# 0 UDPC  jbv1.emotorwerks.com:8042 (6470)
# 1 HTTP  https://directory-api.emotorwerks.com/v1/profiles/juicebox/0817081001070462465218264504/:443 (35085)
> [2024-11-15 | 23:09:24: Closed: 1]
[2024-11-15 | 23:09:24: Closed: 0]
[2024-11-15 | 23:09:25: Opening: juicenet-udp-prod4-exw.enelx.com:9004]
[2024-11-15 | 23:09:25: Opened: 0]
[2024-11-15 | 23:09:26: Opening: juicenet-prod4.enelx.com]
[2024-11-15 | 23:09:27: Opened: 1]
[2024-11-15 | 23:09:31: Closed: 1]
[2024-11-15 | 23:09:33: Opened: 1]
[2024-11-15 | 23:09:33: Closed: 1]
[2024-11-15 | 23:10:04: Opened: 1]
[2024-11-15 | 23:10:04: Closed: 1]
[2024-11-15 | 23:10:24: Opened: 1]
[2024-11-15 | 23:10:24: Closed: 1]

This time there's another URL listed (https://directory-api.emotorwerks.com/v1/profiles/juicebox/0817081001070462465218264504), which returned this little gem:

{
  "control_api_endpoint" : "https://juicenet-prod4.enelx.com",
  "device_https_endpoint" : "https://juicenet-prod4.enelx.com",
  "device_ocpp_endpoint" : "https://juicenet-prod4.enelx.com",
  "device_protocol" : "Both",
  "device_udp_endpoint" : "juicenet-udp-prod4-exw.enelx.com:9004",
  "encrypt" : "false"
}

Note the presence of the OCPP protocol (Open Charge Point Protocol). Could this be the way to MITM this box and gain permanent control over it?

philipkocanda commented 1 week ago

One more error, this time a different one (Unsupported encrypted message version):

2024-11-16 01:32:43  INFO      [ha_mqtt_discoverable.sensors] Setting Send Command to JuiceBox to Send Command to JuiceBox using hmd/text/JuiceBox/Send-Command-to-JuiceBox/state
2024-11-16 01:32:43  INFO      [telnetlib3.client] Connected to <Peer 10.0.1.230 2000>
2024-11-16 01:32:45  INFO      [juicebox_udpcupdater] JuiceboxUDPCUpdater Check Starting
2024-11-16 01:32:46  INFO      [juicebox_udpcupdater] UDPC IP correct
2024-11-16 01:32:48  ERROR     [juicebox_mitm] Not a valid juicebox message |b'0817081001070462465218264504:v08(\x1d?\x01\x00\x00\r\xadT\x93\xeaL*wt\xe3T\xdb\xe6\x96CU\x8c&\xf9K\x8a\x83\xc0\xcdu\xd6\xf5\x1e*\t\x81P]\x01\xeeGz\x8b\x9cX\xeb\xb6\xdd\xae\xe3m\xfd\xa4\xc97c\x86@\x1cEH\xc8e2\xd4#\xa9\xc2c\xd7a\xc0\x81\xd9{\xac\x04\xb4{\xf49V\xcb#4X\x1dxo\xb6\x8f-\xbf\x07O\xba?w\xd3\xd7\x10\xa40T1\xaa\x97X\x12.x\xca`\xd1\xc3\x8e\xc89\xae\n\x98\xe2#B\xc0\x94v'| Unsupported encrypted message version: 'b'0817081001070462465218264504:v08(\x1d?\x01\x00\x00\r\xadT\x93\xeaL*wt\xe3T\xdb\xe6\x96CU\x8c&\xf9K\x8a\x83\xc0\xcdu\xd6\xf5\x1e*\t\x81P]\x01\xeeGz\x8b\x9cX\xeb\xb6\xdd\xae\xe3m\xfd\xa4\xc97c\x86@\x1cEH\xc8e2\xd4#\xa9\xc2c\xd7a\xc0\x81\xd9{\xac\x04\xb4{\xf49V\xcb#4X\x1dxo\xb6\x8f-\xbf\x07O\xba?w\xd3\xd7\x10\xa40T1\xaa\x97X\x12.x\xca`\xd1\xc3\x8e\xc89\xae\n\x98\xe2#B\xc0\x94v''
Traceback (most recent call last):
  File "/juicepassproxy/juicebox_message.py", line 144, in juicebox_message_from_bytes
    string = data.decode("utf-8")
             ^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 39: invalid start byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/juicepassproxy/juicebox_mitm.py", line 156, in _message_decode
    decoded_message = juicebox_message_from_bytes(data)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/juicepassproxy/juicebox_message.py", line 148, in juicebox_message_from_bytes
    return JuiceboxEncryptedMessage().from_bytes(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/juicepassproxy/juicebox_message.py", line 398, in from_bytes
    raise JuiceboxInvalidMessageFormat(f"Unsupported encrypted message version: '{data}'")
juicebox_exceptions.JuiceboxInvalidMessageFormat: Unsupported encrypted message version: 'b'0817081001070462465218264504:v08(\x1d?\x01\x00\x00\r\xadT\x93\xeaL*wt\xe3T\xdb\xe6\x96CU\x8c&\xf9K\x8a\x83\xc0\xcdu\xd6\xf5\x1e*\t\x81P]\x01\xeeGz\x8b\x9cX\xeb\xb6\xdd\xae\xe3m\xfd\xa4\xc97c\x86@\x1cEH\xc8e2\xd4#\xa9\xc2c\xd7a\xc0\x81\xd9{\xac\x04\xb4{\xf49V\xcb#4X\x1dxo\xb6\x8f-\xbf\x07O\xba?w\xd3\xd7\x10\xa40T1\xaa\x97X\x12.x\xca`\xd1\xc3\x8e\xc89\xae\n\x98\xe2#B\xc0\x94v''
2024-11-16 01:32:49  ERROR     [__main__] A JuicePass Proxy task failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 39: invalid start byte
Traceback (most recent call last):
  File "/juicepassproxy/juicepassproxy.py", line 581, in main
ivanfmartinez commented 1 week ago

The enel X servers are enabling encryption on the devices.

You have to block access to the enel x servers using something

I have tried to access the standard endpoint for some of the devices that are using encryption and all return encrypt false, probably the Juicebox send any header that makes the server know that device can support encryption and then are returning true or other thing to enable the encryption.

ivanfmartinez commented 1 week ago

@philipkocanda the line numbers from your exceptions does not match the lines from latest version in my branch. Latest version has better messages when encryption is found.

Your device is sending v08 protocol version, can you add information about your device at, help understand if its a different version or just something that directory server changed :

https://github.com/JuiceRescue/JuiceboxRescueWiki/wiki/Firmware-Versions

https://github.com/JuiceRescue/JuiceboxRescueWiki/wiki/Hardware-Versions