search

nopnop2002 / esp-idf-can2mqtt

CANbus to mqtt bridge using esp32
MIT License
42 stars 8 forks source link

inconsistent MQTT Payload #5

Open ericgrand opened 11 months ago

ericgrand commented 11 months ago

Hello! Thanks for your code, it is very much appreciated! At first, I'm not a programmer at all, barely able to use pre made code! I succed in flashing the ESP, configuring the CAN settings and while monitoring, the CAN message are properly read. THe CAN ID I need is properly sent to my mosquito server, but the Payload is not consistent... There is 2 errors when the ESP 32 start, related to table build...I suspect it may be linked. Could you direct me to find the most probable mistake I have made?

Thanks in advance,

Eric mqttexplorer serial monitor error ESP

nopnop2002 commented 11 months ago

Can you share your can2mqtt.csv/mqtt2can.csv?

ericgrand commented 11 months ago

Sure! mqtt2can.csv can2mqtt.csv

nopnop2002 commented 11 months ago

I used your can2mqtt.csv/mqtt2can.csv.

$ cat mqtt2can.csv
#The file can2mqtt.csv has three columns.
#In the first column you need to specify the CAN Frame type.
#The CAN frame type is either S(Standard frame) or E(Extended frame).
#In the second column you have to specify the CAN-ID as a __hexdecimal number__.
#In the last column you have to specify the MQTT-Topic.
#Each CAN-ID and each MQTT-Topic is allowed to appear only once in the whole file.

S,0C0,/BMS/SOC
#E,201,/can/ext/201
#S,203,/can/std/203
#E,203,/can/ext/203

$ cat mqtt2can.csv
#The file can2mqtt.csv has three columns.
#In the first column you need to specify the CAN Frame type.
#The CAN frame type is either S(Standard frame) or E(Extended frame).
#In the second column you have to specify the CAN-ID as a __hexdecimal number__.
#In the last column you have to specify the MQTT-Topic.
#Each CAN-ID and each MQTT-Topic is allowed to appear only once in the whole file.

S,0C0,/BMS/SOC
#E,201,/can/ext/201
#S,203,/can/std/203
#E,203,/can/ext/203

This is my logging. There is no error.

I (6891) MAIN: Mount SPIFFS filesystem
I (6891) MAIN: build_table file=/spiffs/can2mqtt.csv
I (6901) MAIN: build_table _ntopic=1
I (6901) main: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC] topic_len=8
I (6901) MAIN: build_table file=/spiffs/mqtt2can.csv
I (6931) MAIN: build_table _ntopic=1
I (6931) main: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC] topic_len=8

Can you share all your logging?

ericgrand commented 11 months ago

Ok, that mean's I should miss something...I'm using Visual Studio Code with ESP-IDF extension. I don't understand why it is build in your case wit a topic length of 8, which is correct, and in my case it build it with a topic length of 9... Here is a copy of the terminal log after a flash: `ho 8 tail 4 room 4
load:0x40080404,len:3876 entry 0x4008064c I (31) boot: ESP-IDF v5.1.2 2nd stage bootloader I (31) boot: compile time Nov 25 2023 16:45:18 I (31) boot: Multicore bootloader I (35) boot: chip revision: v3.1 I (39) boot.esp32: SPI Speed : 40MHz
I (44) boot.esp32: SPI Mode : DIO I (48) boot.esp32: SPI Flash Size : 4MB I (53) boot: Enabling RNG early entropy source... I (58) boot: Partition Table: I (62) boot: ## Label Usage Type ST Offset Length I (69) boot: 0 nvs WiFi data 01 02 00009000 00006000 I (77) boot: 1 phy_init RF data 01 01 0000f000 00001000 I (84) boot: 2 factory factory app 00 00 00010000 00100000 I (92) boot: 3 storage Unknown data 01 82 00110000 000f0000 I (99) boot: End of partition table I (103) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=26ea4h (159396) map I (145) esp_image: segment 1: paddr=00036ecc vaddr=3ffb0000 size=03b64h ( 15204) load I (148) esp_image: segment 2: paddr=0003aa38 vaddr=40080000 size=055e0h ( 21984) load I (156) esp_image: segment 3: paddr=00040020 vaddr=400d0020 size=a9460h (693344) map I (302) esp_image: segment 4: paddr=000e9488 vaddr=400855e0 size=1065ch ( 67164) load I (330) boot: Loaded app from partition at offset 0x10000 I (330) boot: Disabling RNG early entropy source... I (342) cpu_start: Multicore app I (342) cpu_start: Pro cpu up. I (342) cpu_start: Starting app cpu, entry point is 0x40081390 0x40081390: call_start_cpu1 at C:/Users/Eric/esp/esp-idf/components/esp_system/port/cpu_start.c:157

I (0) cpu_start: App cpu up. I (360) cpu_start: Pro cpu start user code I (360) cpu_start: cpu freq: 160000000 Hz I (360) cpu_start: Application information: I (365) cpu_start: Project name: can2mqtt I (370) cpu_start: App version: 42a799e-dirty I (375) cpu_start: Compile time: Nov 25 2023 16:44:08 I (382) cpu_start: ELF file SHA256: 07cde329a520bd42... I (388) cpu_start: ESP-IDF: v5.1.2 I (392) cpu_start: Min chip rev: v0.0 I (397) cpu_start: Max chip rev: v3.99 I (402) cpu_start: Chip rev: v3.1 I (407) heap_init: Initializing. RAM available for dynamic allocation: I (414) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM I (420) heap_init: At 3FFB89C0 len 00027640 (157 KiB): DRAM I (426) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM I (432) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM I (439) heap_init: At 40095C3C len 0000A3C4 (40 KiB): IRAM I (447) spi_flash: detected chip: generic I (450) spi_flash: flash io: dio I (455) app_start: Starting scheduler on CPU0 I (459) app_start: Starting scheduler on CPU1 I (459) main_task: Started on CPU0 I (469) main_task: Calling app_main() I (519) wifi:wifi driver task: 3ffc096c, prio:23, stack:6656, core=0 I (529) wifi:wifi firmware version: 91b9630 I (529) wifi:wifi certification version: v7.0 I (529) wifi:config NVS flash: enabled I (529) wifi:config nano formating: disabled I (539) wifi:Init data frame dynamic rx buffer num: 32 I (539) wifi:Init static rx mgmt buffer num: 5 I (549) wifi:Init management short buffer num: 32 I (549) wifi:Init dynamic tx buffer num: 32 I (559) wifi:Init static rx buffer size: 1600 I (559) wifi:Init static rx buffer num: 10 I (559) wifi:Init dynamic rx buffer num: 32 I (569) wifi_init: rx ba win: 6 I (569) wifi_init: tcpip mbox: 32 I (579) wifi_init: udp mbox: 6 I (579) wifi_init: tcp mbox: 6 I (579) wifi_init: tcp tx win: 5744 I (589) wifi_init: tcp rx win: 5744 I (589) wifi_init: tcp mss: 1440 I (599) wifi_init: WiFi IRAM OP enabled I (599) wifi_init: WiFi RX IRAM OP enabled I (609) phy_init: phy_version 4780,16b31a7,Sep 22 2023,20:42:16 I (689) wifi:mode : sta (08:d1:f9:34:58:80) I (689) wifi:enable tsf I (689) MAIN: wifi_init_sta finished. I (709) wifi:new:<6,0>, old:<1,0>, ap:<255,255>, sta:<6,0>, prof:1 I (709) wifi:state: init -> auth (b0) I (709) wifi:state: auth -> assoc (0) I (719) wifi:state: assoc -> run (10) I (749) wifi:idx:0 (ifx:0, a0:04:60:80:0b:66), tid:0, ssn:0, winSize:64 I (769) wifi:connected with Maison_Fonsorbes_2.4, aid = 21, channel 6, BW20, bssid = a0:04:60:80:0b:66 I (769) wifi:security: WPA2-PSK, phy: bgn, rssi: -71 I (779) wifi:pm start, type: 1

I (889) wifi:AP's beacon interval = 204800 us, DTIM period = 2 I (4279) esp_netif_handlers: sta ip: 10.0.0.207, mask: 255.255.255.0, gw: 10.0.0.1 I (4279) MAIN: got ip:10.0.0.207 I (4279) MAIN: connected to ap SSID:XXXXXXXXXXXXXXXXX password:XXXXXXXXXXX I (4289) MAIN: Bitrate is 500 Kbit/s I (4289) MAIN: CTX_GPIO=21 I (4299) MAIN: CRX_GPIO=22 I (4299) gpio: GPIO[22]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0 I (4309) gpio: GPIO[21]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0 I (4319) MAIN: Driver installed I (4319) MAIN: Driver started I (4319) MAIN: Initializing SPIFFS file system I (4439) MAIN: Partition size: total: 896321, used: 1506 I (4449) MAIN: d_name=can2mqtt.csv d_ino=0 d_type=1 I (4449) MAIN: d_name=mqtt2can.csv d_ino=0 d_type=1 I (4499) MAIN: Mount SPIFFS filesystem I (4499) MAIN: build_table file=/spiffs/can2mqtt.csv I (4499) MAIN: build_table _ntopic=2 ] (4499) MAIN: This line is invalid [ ] topic_len=9: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC I (4509) MAIN: build_table file=/spiffs/mqtt2can.csv I (4519) MAIN: build_table _ntopic=2 ] (4519) MAIN: This line is invalid [ ] topic_len=9: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC I (4539) PUB: Start Subscribe Broker:10.0.0.25 I (4539) PUB: client_id=[pub-08d1f9345880] I (4539) PUB: CONFIG_MQTT_BROKER=[10.0.0.25] I (4549) convert_mdns_host: from=[10.0.0.25] I (4549) PUB: ip=[10.0.0.25] I (4559) PUB: uri=[mqtt://10.0.0.25] I (4559) SUB: Start Subscribe Broker:10.0.0.25 I (4559) PUB: Other event id:7 I (4559) TWAI: task start ] topic_len=9_rx: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC ] topic_len=9_sub: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC I (4589) SUB: client_id=[sub-08d1f9345880] I (4599) SUB: CONFIG_MQTT_BROKER=[10.0.0.25] I (4599) convert_mdns_host: from=[10.0.0.25] I (4609) SUB: ip=[10.0.0.25] I (4609) wifi:I (4609) SUB: uri=[mqtt://10.0.0.25]

idx:1 (ifx:0, a0:04:60:80:0b:66), tid:6, ssn:0, winSize:64 I (4609) SUB: Other event id:7 I (4559) main_task: Returned from app_main() I (4639) PUB: MQTT_EVENT_CONNECTED I (4649) PUB: Connect to MQTT Server I (4649) SUB: MQTT_EVENT_CONNECTED I (4649) SUB: Connect to MQTT Server ] (4659) SUB: subscribe[0] topic=[/BMS/SOC I (4679) SUB: MQTT_EVENT_SUBSCRIBED, msg_id=51570`
nopnop2002 commented 11 months ago

The cause is that the CSV file cannot be read correctly.

In your log, the topic length (topic_len) is 9.

] topic_len=9_rx: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC
] topic_len=9_sub: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC

Your topic is [/BMS/SOC]. The correct topic length (topic_len) is 8.

I (6941) main: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC] topic_len=8
I (6971) main: topics=[0] frame=0 canid=0xc0 topic=[/BMS/SOC] topic_len=8

There is a possibility that the line feed code of the CSV file is LF+CR.

You can check the line feed code of the CSV file with this. The line feed code is only 0x0a.

$ od -cx can2mqtt.csv
0000000   #   T   h   e       f   i   l   e       c   a   n   2   m   q
           5423    6568    6620    6c69    2065    6163    326e    716d
0000020   t   t   .   c   s   v       h   a   s       t   h   r   e   e
           7474    632e    7673    6820    7361    7420    7268    6565
0000040       c   o   l   u   m   n   s   .      \n   #   I   n       t
           6320    6c6f    6d75    736e    202e    230a    6e49    7420
0000060   h   e       f   i   r   s   t       c   o   l   u   m   n
           6568    6620    7269    7473    6320    6c6f    6d75    206e
0000100   y   o   u       n   e   e   d       t   o       s   p   e   c
           6f79    2075    656e    6465    7420    206f    7073    6365
0000120   i   f   y       t   h   e       C   A   N       F   r   a   m
           6669    2079    6874    2065    4143    204e    7246    6d61
0000140   e       t   y   p   e   .  \n   #   T   h   e       C   A   N
           2065    7974    6570    0a2e    5423    6568    4320    4e41
0000160       f   r   a   m   e       t   y   p   e       i   s       e
           6620    6172    656d    7420    7079    2065    7369    6520
0000200   i   t   h   e   r       S   (   S   t   a   n   d   a   r   d
           7469    6568    2072    2853    7453    6e61    6164    6472
0000220       f   r   a   m   e   )       o   r       E   (   E   x   t
           6620    6172    656d    2029    726f    4520    4528    7478
0000240   e   n   d   e   d       f   r   a   m   e   )   .  \n   #   I
           6e65    6564    2064    7266    6d61    2965    0a2e    4923
0000260   n       t   h   e       s   e   c   o   n   d       c   o   l
           206e    6874    2065    6573    6f63    646e    6320    6c6f
0000300   u   m   n       y   o   u       h   a   v   e       t   o
           6d75    206e    6f79    2075    6168    6576    7420    206f
0000320   s   p   e   c   i   f   y       t   h   e       C   A   N   -
           7073    6365    6669    2079    6874    2065    4143    2d4e
0000340   I   D       a   s       a       _   _   h   e   x   d   e   c
           4449    6120    2073    2061    5f5f    6568    6478    6365
0000360   i   m   a   l       n   u   m   b   e   r   _   _   .      \n
           6d69    6c61    6e20    6d75    6562    5f72    2e5f    0a20
0000400   #   I   n       t   h   e       l   a   s   t       c   o   l
           4923    206e    6874    2065    616c    7473    6320    6c6f
0000420   u   m   n       y   o   u       h   a   v   e       t   o
           6d75    206e    6f79    2075    6168    6576    7420    206f
0000440   s   p   e   c   i   f   y       t   h   e       M   Q   T   T
           7073    6365    6669    2079    6874    2065    514d    5454
0000460   -   T   o   p   i   c   .  \n   #   E   a   c   h       C   A
           542d    706f    6369    0a2e    4523    6361    2068    4143
0000500   N   -   I   D       a   n   d       e   a   c   h       M   Q
           2d4e    4449    6120    646e    6520    6361    2068    514d
0000520   T   T   -   T   o   p   i   c       i   s       a   l   l   o
           5454    542d    706f    6369    6920    2073    6c61    6f6c
0000540   w   e   d       t   o       a   p   p   e   a   r       o   n
           6577    2064    6f74    6120    7070    6165    2072    6e6f
0000560   l   y       o   n   c   e       i   n       t   h   e       w
           796c    6f20    636e    2065    6e69    7420    6568    7720
0000600   h   o   l   e       f   i   l   e   .  \n  \n   S   ,   0   C
           6f68    656c    6620    6c69    2e65    0a0a    2c53    4330
0000620   0   ,   /   B   M   S   /   S   O   C  \n   #   E   ,   1   0
           2c30    422f    534d    532f    434f    230a    2c45    3031
0000640   1   ,   /   c   a   n   /   e   x   t   /   1   0   1  \n   #
           2c31    632f    6e61    652f    7478    312f    3130    230a
0000660   S   ,   1   0   3   ,   /   c   a   n   /   s   t   d   /   1
           2c53    3031    2c33    632f    6e61    732f    6474    312f
0000700   0   3  \n   #   E   ,   1   0   3   ,   /   c   a   n   /   e
           3330    230a    2c45    3031    2c33    632f    6e61    652f
0000720   x   t   /   1   0   3  \n
           7478    312f    3330    000a
0000727
ericgrand commented 11 months ago

Thanks for your feedback, I will investigate!

ericgrand commented 11 months ago

So progress made, thanks to your feedback. CSV files were CRLF, I switched to LF only and I have no more errors during the boot phase! Nevertheless, the payload received by the broker is still not readable. Here is the serial monitor capture of MQTT message sent by the ESP32, which seems correct, and below a screen capture of the message received by the broker...Is there any specific parameter for caracter standard? I'm really sorry, it may be a dumb question, but I'm very new to all these processes...

Edit: As a matter of test, I have sent the MQTT messages to the Mosquitto test server, publically available. The payload is still not readable... It seems it has to do with the encoding when the message is sent...

Standard ID: 0x0c0      DLC: 8  Data: 0x43 0x01 0x00 0x00 0x00 0x06 0x00 0x00 
I (4156) TWAI: publish[0] frame=0 canid=0xc0 topic=[/BMS/SOC] topic_len=8
I (4166) TWAI: mqttBuf.data[i]=0x43
I (4166) TWAI: mqttBuf.data[i]=0x1
I (4176) TWAI: mqttBuf.data[i]=0x0
I (4176) TWAI: mqttBuf.data[i]=0x0
I (4186) TWAI: mqttBuf.data[i]=0x0
I (4186) TWAI: mqttBuf.data[i]=0x6
I (4186) TWAI: mqttBuf.data[i]=0x0
I (4196) TWAI: mqttBuf.data[i]=0x0
I (4196) PUB: TOPIC=[/BMS/SOC] LEN=8
I (4206) PUB: DATA=0x43
I (4206) PUB: DATA=0x1
I (4206) PUB: DATA=0x0
I (4216) PUB: DATA=0x0
I (4216) PUB: DATA=0x0
I (4216) PUB: DATA=0x6
I (4226) PUB: DATA=0x0
I (4226) PUB: DATA=0x0
I (4226) PUB: EventBits=0x1
I (4246) PUB: MQTT_EVENT_PUBLISHED, msg_id=19261
I (4296) SUB: MQTT_EVENT_DATA
I (4296) SUB: type=200
I (4296) SUB: TOPIC=[/BMS/SOC]
I (4296) SUB: DATA=0x43
I (4296) SUB: DATA=0x1
I (4296) SUB: DATA=0x0
I (4306) SUB: DATA=0x0
I (4306) SUB: DATA=0x0
I (4306) SUB: DATA=0x6
I (4316) SUB: DATA=0x0
I (4316) SUB: DATA=0x0
I (4316) SUB: subscribe[index].frame=0
I (4336) TWAI: tx_msg.identifier=[0xc0] tx_msg.extd=0
I (4336) TWAI: twai_transmit success

mqttexplorer2

nopnop2002 commented 11 months ago

the payload received by the broker is still not readable.

What is this tool you are using?

This tool cannot display data correctly. Please use other mqtt tools.

mqttexplorer2

ericgrand commented 11 months ago

It is MQTT explorer. Working fine to access MQTT broker mqttexplorer3

nopnop2002 commented 11 months ago

Select data type to unsigned int. 104162359-c52b3180-53f4-11eb-8e33-5c1a36ccff1b

ericgrand commented 11 months ago

I don't have such option in my version... I have the same issue with the plugin used in my Jeedom (equivalent to Home Assistant software). I can't change the data type. I think I have to dig a little bit more to understand the payload format...

Thanks for your kind support!