btsimonh commented 5 years ago

hi shmuelzon, any thoughts on platform.io support? s

shmuelzon commented 5 years ago

Hey,

After reading up on PlatformIO for a whole 5 minutes now, I must say: I don't get it... :) This project is specifically targeted for the ESP-IDF. If it were Arduino-based or something, then it would, maybe, make sense to use PlatformIO so it could be easily compiled for other platforms. But, in this case, as far as I can tell, it will allow obtaining the toolchain and SDK easily and a wrapper around make. I also have some added logic in my Makefile so I'm not sure if these will play nicely together.

Anyways, if all it means is adding a platformio.ini file to the repository, I'm not against that, but I probably won't do that myself as I'm not using PlatformIO myself.

Thanks

btsimonh commented 5 years ago

:). platform.io is useful; it means almost anyone can download the project and platform.io does the rest (installs all the tools required automagically), and it plays fairly nice with a couple of IDEs. However, it does like a certain style of project; I've converted a few, but it's seldom just the addition of the platform.io file :(. It was worth asking in case you had some experience; at some point I'll have a crack at it and see how good/bad/ugly it is, and let you know. The painful thing is it normally wants a directory restructure - not to a bad structure but to a different one; and that then does not look nice in a pull request ....

shmuelzon commented 5 years ago

The painful thing is it normally wants a directory restructure

That is a shame since it will probably break support with ESP-IDF and I don't want that. Might be 'fixable' with symlinks or something, but we'll cross that bridge when we get there.

Once ESP-IDF v3.2 is officially out, I'll evaluate the stability of it and probably move to the built-in MQTT and HTTP client implementations which will remove two dependencies and perhaps make PlatformIO porting simpler.

btsimonh commented 5 years ago

ok, got a build; but according to platformio it's oversize: Memory Usage -> http://bit.ly/pio-memory-usageError: The program size (1113826 bytes) is greater than maximum allowed (1048576 bytes) how big do you expect it to be? (this used the platformio esp32@1.4.0; yet to confirm what that means in terms of the expressif sdk version :). ) an SPIFFS creation not done yet; but progressing....

shmuelzon commented 5 years ago

Nice!

The project has a custom partition table. From what I saw in the PlatformIO's documentation, you need to add that to the ini file. There are also some other settings in the sdkconfig.defaults that should be taken into account.

As for the ESP-IDF version, could you use the commit ID from the README file?

btsimonh commented 5 years ago

seems the default partition table is this: https://github.com/espressif/esp-idf/blob/master/components/partition_table/partitions_singleapp.csv

not sure how platformio identifies which partition represents where it has to fit it's code... but after setting your partition table as the one, it seems to have stopped complaining.

for the moment, I added the following to sdkconfig.h:

define CONFIG_ESP_MQTT_ENABLED 1

define CONFIG_ESP_MQTT_TASK_STACK_SIZE 4096

define CONFIG_ESP_MQTT_TASK_STACK_PRIORITY 5

define CONFIG_ESP_MQTT_EVENT_QUEUE_SIZE 64

define CONFIG_LWIP_SO_RCVBUF 1

it seems to already know about sdkconfig.defaullts - should I throw the above in there instead?

btsimonh commented 5 years ago

p.s. I tried building with the platformio esp ppackage at 1.5.0, and there was some build error building the sdk libs. more investigation needed as to how we could specify a specific commit of the esp sdk; I assume we'd need to modify their 'package'.....

shmuelzon commented 5 years ago

not sure how platformio identifies which partition represents where it has to fit it's code...

It probably does the same as ESP-IDF by searching for the app partition type.

it seems to already know about sdkconfig.defaullts - should I throw the above in there instead?

The configuration in sdkconfig.defaults is just for overriding the default configuration. The MQTT ones that you added are the default settings. Why do you need to manually add them?

I assume we'd need to modify their 'package'.....

That would be a problem, no? Even if we stick with the official ESP-IDF release, I had many WiFi and BLE disconnections as well as all sorts of crashes with them. I'm hoping v3.2 will be better.

btsimonh commented 5 years ago

ok, my first attempt https://github.com/btsimonh/esp32-ble2mqtt/tree/platformio/src still no spiffs creation, but you can see the simple mods to your .py for getting the gatt details using platformio, and the folder restructure to put everything under 'src' like platform.io likes :(. I assume the mkspiffs stuff needs to build for local machine, not target; then is used to make the spiffs partition with the config data in it. It may be that platformio already supports it as a binary - need to investigate.

Need to go do 'real' work now; may be a few days/weeks before I return to this, but it's a start...

btsimonh commented 5 years ago

oh; and the version define is hardcoded to '1'; seems you can inline script in python to do such things, but I'm no python expert, and don't know how platformio deals with git on multiple platforms, so that may wait until we have a working build :).

shmuelzon commented 5 years ago

Is the directory restructure really needed? It seems there are two variables (src_dir and lib_dir) that you can use to set as main and components, respectively, and keep the original directory structure and compatibility with ESP-IDF. Also, your sdkconfig.h doesn't include the changes I've defined in sdkconfig.defaults. Specifically, CONFIG_BT_ACL_CONNECTIONS to allow more BT connections and disabling CONFIG_SW_COEXIST_ENABLE for added stability.

btsimonh commented 5 years ago

probably not; will check those out. have found that the mkspiffs may not be needed - running pio run -t buildfs in a 'terminal' inside vscode built a spiffs.bin and then pio run -t uploadfs (with button held from power on) put the spiffs in the right place in the device. Then the previously uploaded binary runs!.

p.s. without spiffs present, the program resets continually. Maybe it should format the spiffs and create a default config?

shmuelzon commented 5 years ago

A default configuration file doesn't have much use because if you don't have the WiFi and MQTT credentials, you can't do much or be upgraded remotely.

A have a partial implementation of enabling an open access point and loading an application and/or SPIFFS image via web browser. But the HTTP server code is also part of the newer ESP-IDF so it will also have to wait...

btsimonh commented 5 years ago

ok, have a serial log; seems not to be happy with my devices. Any advice appreciated before I dive into the code myself and try to work it out. :). Also, I should modify the code to log the flags from the SDK to see what we ended up with (ref you previous note).

Executing task in folder esp32-ble2mqtt: C:\Python27\Scripts\platformio.exe device monitor <

--- Miniterm on COM11 115200,8,N,1 --- --- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H --- %=m[0m ␛[0;32mI (41) boot: SPI Flash Size : 4MB␛[0m ␛[0;32mI (45) boot: Partition Table:␛[0m ␛[0;32mI (49) boot: ## Label Usage Type ST Offset Length␛[0m ␛[0;32mI (56) boot: 0 nvs WiFi data 01 02 00009000 00004000␛[0m ␛[0;32mI (63) boot: 1 otadata OTA data 01 00 0000d000 00002000␛[0m ␛[0;32mI (71) boot: 2 phy_init RF data 01 01 0000f000 00001000␛[0m ␛[0;32mI (78) boot: 3 ota_0 OTA app 00 10 00010000 00180000␛[0m ␛[0;32mI (86) boot: 4 ota_1 OTA app 00 11 00190000 00180000␛[0m ␛[0;32mI (93) boot: 5 storage Unknown data 01 82 00310000 00080000␛[0m ␛[0;32mI (101) boot: End of partition table␛[0m ␛[0;32mI (105) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x2d994 (186772) map␛[0m ␛[0;32mI (180) esp_image: segment 1: paddr=0x0003d9bc vaddr=0x3ffc0000 size=0x02654 ( 9812) load␛[0m ␛[0;32mI (184) esp_image: segment 2: paddr=0x00040018 vaddr=0x400d0018 size=0xc8204 (819716) map␛[0m ␛[0;32mI (476) esp_image: segment 3: paddr=0x00108224 vaddr=0x3ffc2654 size=0x03464 ( 13412) load␛[0m ␛[0;32mI (481) esp_image: segment 4: paddr=0x0010b690 vaddr=0x3ffc5ab8 size=0x00000 ( 0) load␛[0m ␛[0;32mI (483) esp_image: segment 5: paddr=0x0010b698 vaddr=0x40080000 size=0x00400 ( 1024) load␛[0m ␛[0;32mI (492) esp_image: segment 6: paddr=0x0010baa0 vaddr=0x40080400 size=0x14494 ( 83092) load␛[0m ␛[0;32mI (534) esp_image: segment 7: paddr=0x0011ff3c vaddr=0x400c0000 size=0x00000 ( 0) load␛[0m ␛[0;32mI (534) esp_image: segment 8: paddr=0x0011ff44 vaddr=0x50000000 size=0x00000 ( 0) load␛[0m ␛[0;32mI (553) boot: Loaded app from partition at offset 0x10000␛[0m ␛[0;32mI (554) boot: Disabling RNG early entropy source...␛[0m ␛[0;32mI (555) cpu_start: Pro cpu up.␛[0m ␛[0;32mI (558) cpu_start: Starting app cpu, entry point is 0x40081320␛[0m ␛[0;32mI (0) cpu_start: App cpu up.␛[0m ␛[0;32mI (569) heap_init: Initializing. RAM available for dynamic allocation:␛[0m ␛[0;32mI (576) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM␛[0m ␛[0;32mI (582) heap_init: At 3FFD4548 len 0000BAB8 (46 KiB): DRAM␛[0m ␛[0;32mI (588) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM␛[0m ␛[0;32mI (594) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM␛[0m ␛[0;32mI (601) heap_init: At 40094894 len 0000B76C (45 KiB): IRAM␛[0m ␛[0;32mI (607) cpu_start: Pro cpu start user code␛[0m ␛[0;32mI (289) cpu_start: Starting scheduler on PRO CPU.␛[0m ␛[0;32mI (0) cpu_start: Starting scheduler on APP CPU.␛[0m ␛[0;32mI (331) BLE2MQTT: Version: 1␛[0m ␛[0;32mI (331) Config: Initializing configuration␛[0m ␛[0;32mI (361) Config: version: 81b6cb3d19699ba1f316cf4c097174be␛[0m ␛[0;32mI (361) OTA: Initializing OTA␛[0m I (371) wifi: wifi driver task: 3ffde20c, prio:23, stack:3584, core=0 I (371) wifi: wifi firmware version: 54ef8ed I (371) wifi: config NVS flash: enabled I (381) wifi: config nano formating: disabled ␛[0;32mI (381) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE␛[0m ␛[0;32mI (391) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE␛[0m I (411) wifi: Init dynamic tx buffer num: 32 I (411) wifi: Init data frame dynamic rx buffer num: 32 I (411) wifi: Init management frame dynamic rx buffer num: 32 I (421) wifi: Init static rx buffer size: 1600 I (421) wifi: Init static rx buffer num: 10 I (421) wifi: Init dynamic rx buffer num: 32 ␛[0;32mI (431) BTDM_INIT: BT controller compile version [8353b1b] ␛[0m ␛[0;32mI (431) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE␛[0m ␛[0;32mI (501) phy: phy_version: 3960, 5211945, Jul 18 2018, 10:40:07, 0, 0␛[0m ␛[0;32mI (771) WiFi: Connecting to SSID MYSSID␛[0m I (781) wifi: mode : sta (30:ae:a4:e8:9e:98) I (901) wifi: n:11 0, o:1 0, ap:255 255, sta:11 0, prof:1 I (1751) wifi: state: init -> auth (b0) I (1761) wifi: state: auth -> assoc (0) I (1761) wifi: state: assoc -> run (10) I (1781) wifi: connected with MYSSID, channel 11 I (1781) wifi: pm start, type: 1

␛[0;32mI (1781) WiFi: Connected␛[0m ␛[0;32mI (2361) event: sta ip: 192.168.1.176, mask: 255.255.255.0, gw: 192.168.1.1␛[0m ␛[0;32mI (2361) BLE2MQTT: Connected to WiFi, connecting to MQTT␛[0m ␛[0;32mI (2361) MQTT: Connecting MQTT client␛[0m ␛[0;32mI (2361) esp_mqtt: esp_mqtt_start: create task␛[0m ␛[0;32mI (2371) esp_mqtt: esp_mqtt_process: begin connection attempt␛[0m ␛[0;32mI (3521) esp_mqtt: esp_mqtt_process: connection attempt successful␛[0m ␛[0;32mI (3521) MQTT: MQTT client connected␛[0m ␛[0;32mI (3521) BLE2MQTT: Connected to MQTT, scanning for BLE devices␛[0m ␛[0;32mI (6281) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:db, connecting␛[0m ␛[0;31mE (15441) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (17781) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (17921) BLE2MQTT: Connected to device: 00:1a:22:09:2c:db, scanning␛[0m ␛[0;31mE (17951) BT_APPL: service change write ccc failed␛[0m ␛[0;32mI (19351) BLE2MQTT: Publishing: 00:1a:22:09:2c:db/GenericAccess/DeviceName = CC-RT-M-BLE␛[0m ␛[0;32mI (19401) BLE2MQTT: Publishing: 00:1a:22:09:2c:db/GenericAccess/Appearance = 0␛[0m ␛[0;32mI (19531) BLE2MQTT: Publishing: 00:1a:22:09:2c:db/GenericAccess/PeripheralPrivacyFlag = false␛[0m ␛[0;32mI (19781) BLE2MQTT: Publishing: 00:1a:22:09:2c:db/GenericAccess/PeripheralPreferredConnectionParameters = 0,0,0,0␛[0m ␛[0;32mI (19901) BLE2MQTT: Publishing: 00:1a:22:09:2c:db/GenericAttribute/ServiceChanged = 0,0␛[0m ␛[0;32mI (19981) BLE2MQTT: Publishing: 00:1a:22:09:2c:db/DeviceInformation/ManufacturerNameString = eq-3␛[0m ␛[0;32mI (20461) BLE2MQTT: Publishing: 00:1a:22:09:2c:db/DeviceInformation/ModelNumberString = CC-RT-M-BLE␛[0m ␛[0;32mI (21391) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;32mI (24731) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:9a, connecting␛[0m ␛[0;33mW (25871) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (25881) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (25881) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;33mW (26461) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0008␛[0m ␛[0;31mE (26471) BLE: Failed reading characteristic, status = 0x85␛[0m ␛[0;32mI (26471) BLE: Connection closed, reason = 0x8␛[0m ␛[0;32mI (26471) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:db␛[0m ␛[0;31mE (26821) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (28771) BLE2MQTT: Connected to device: 00:1a:22:09:2c:9a, scanning␛[0m ␛[0;31mE (28801) BT_APPL: service change write ccc failed␛[0m ␛[0;32mI (29891) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (33921) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;32mI (35321) BLE2MQTT: Discovered BLE device: 00:1a:22:08:5b:1e, connecting␛[0m ␛[0;33mW (38031) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0013␛[0m ␛[0;32mI (38041) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (38041) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;31mE (54761) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (56091) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (58931) BLE2MQTT: Connected to device: 00:1a:22:08:5b:1e, scanning␛[0m ␛[0;32mI (63971) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (68261) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0013␛[0m ␛[0;32mI (68261) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (68261) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (76101) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (78581) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:db, connecting␛[0m ␛[0;32mI (79941) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (84491) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0013␛[0m ␛[0;32mI (84491) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (84491) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;33mW (88821) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0008␛[0m ␛[0;32mI (88831) BLE: Connection closed, reason = 0x8␛[0m ␛[0;32mI (88831) BLE2MQTT: Disconnected from device: 00:1a:22:08:5b:1e␛[0m ␛[0;33mW (93691) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=771 reason=0x003e␛[0m ␛[0;31mE (93701) BLE: Open failed, status = 0x85␛[0m ␛[0;32mI (94611) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:db, connecting␛[0m ␛[0;31mE (96761) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (98291) BLE2MQTT: Connected to device: 00:1a:22:09:2c:db, scanning␛[0m ␛[0;32mI (110561) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (114221) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (118781) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (118791) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (118791) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;33mW (118841) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=515 reason=0x0016␛[0m ␛[0;32mI (119261) BLE: Connection closed, reason = 0x16␛[0m ␛[0;32mI (119261) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:9a␛[0m ␛[0;32mI (120791) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (134431) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (138931) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (138931) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (138931) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (139281) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (146581) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (151091) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (151101) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (151101) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (151621) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (162901) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;32mI (166911) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:9a, connecting␛[0m ␛[0;33mW (167321) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (167331) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (167331) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;31mE (172011) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (173261) BLE2MQTT: Connected to device: 00:1a:22:09:2c:9a, scanning␛[0m ␛[0;32mI (177351) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (180961) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (185501) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (185511) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (185511) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (186191) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;33mW (187691) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0016␛[0m ␛[0;32mI (187691) BLE: Connection closed, reason = 0x16␛[0m ␛[0;32mI (187691) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:db␛[0m ␛[0;32mI (209471) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;32mI (212761) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:db, connecting␛[0m ␛[0;33mW (213721) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (213721) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (213721) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;31mE (213851) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (215181) BLE2MQTT: Connected to device: 00:1a:22:09:2c:db, scanning␛[0m ␛[0;32mI (219731) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (239371) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (243901) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (243901) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (243901) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (244561) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (251501) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (256001) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (256011) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (256011) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;33mW (262981) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=515 reason=0x0016␛[0m ␛[0;32mI (262981) BLE: Connection closed, reason = 0x16␛[0m ␛[0;32mI (262981) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:9a␛[0m ␛[0;32mI (266051) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (269701) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;32mI (270951) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:9a, connecting␛[0m ␛[0;33mW (274201) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (274201) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (274201) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;31mE (277051) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (278611) BLE2MQTT: Connected to device: 00:1a:22:09:2c:9a, scanning␛[0m ␛[0;32mI (284241) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (297941) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (302491) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (302491) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (302491) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (302961) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;33mW (304821) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0016␛[0m ␛[0;32mI (304831) BLE: Connection closed, reason = 0x16␛[0m ␛[0;32mI (304831) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:db␛[0m ␛[0;32mI (308851) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:db, connecting␛[0m ␛[0;31mE (309941) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m ␛[0;32mI (310261) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;32mI (311421) BLE2MQTT: Connected to device: 00:1a:22:09:2c:db, scanning␛[0m ␛[0;33mW (314651) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (314661) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (314661) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (318671) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (322271) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (326771) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (326781) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (326781) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (327341) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m ␛[0;32mI (336431) BLE2MQTT: Connected to device: 4c:65:a8:da:f6:07, scanning␛[0m ␛[0;33mW (340951) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0013␛[0m ␛[0;32mI (340951) BLE: Connection closed, reason = 0x13␛[0m ␛[0;32mI (340951) BLE2MQTT: Disconnected from device: 4c:65:a8:da:f6:07␛[0m ␛[0;32mI (341481) BLE2MQTT: Discovered BLE device: 4c:65:a8:da:f6:07, connecting␛[0m

btsimonh commented 5 years ago

ohhhh... no root topic? so it's literally publishing to '00:1a:22:09:2c:db/GenericAccess/DeviceName'? or did i miss something?

I'll need a root topic; my mqtt server is quite 'busy', so can't subscribe to the root! So an MQTT prefix in config would be required.

The good news is that I am seeing mqtt data on my server, even if it's not talking to the devices quite right :).

Ref webserver, ap mode, etc. I'd love to have this functionality inside tasmota as the 'application' framework. He has a form of plugin structure which bears some investigation - so you would not need to maintain the MQTT, webserver, config, or OTA aspects, and have a team of hundreds contributing to these bits. One issue is license though; he's GPL, which for me is always less attractive than MIT.

shmuelzon commented 5 years ago

Well, first off, if you compile the same code via ESP-IDF does it work as expected? Do you know which ESP-IDF version PlatformIO actually uses? Might be interesting to compile "regularly" against the same version and test with that. I've never seen those errors before:

BT_L2CAP: slave connection parameters update failed, the parameters are out of range BT_APPL: service change write ccc failed

Maybe related to the BT version in the ESP-IDF but for that we need to know which version is used.

As for the prefix, I know which devices I'm interested in so I can just listen by their MAC address but I can see how a prefix would be useful. I'll try to take a look at that soon.

btsimonh commented 5 years ago

working on prefix now :). Will look at logging versions, etc. probably easier than trying to work out what we have.
Don't have a raw ESP-IDF env :( Maybe post a binary I can upload?

btsimonh commented 5 years ago

ok, pushed some prefix addition on top (check last commit) - BUT I've screwed something up; it crashes shortly after mqtt connect; so please check my code. Have to do 'real' work now :(. Will try to go back to default folder structure tonight, and scrub that branch in favor of something more amenable...

shmuelzon commented 5 years ago

I looked at your latest commit, a few comments:

Please try to keep the same coding style as the rest of the code
Please move the "prefix" config entry to be under the mqtt/publish section of the JSON. You can then use config_mqtt_topics_get("prefix", "") to implement config_mqtt_prefix_get()
I don't think mqtt.c is the correct location to implement this. I understand why you would want to add this in a general location but as this is an application level thing it should be in ble2mqtt.c. You might be able to modify ble_topic_suffix() to add the prefix there (and probably rename the function). Alternatively, maybe add an _mqtt_publish() function in the same file but that will result with many string copies

btsimonh commented 5 years ago

platform.io is killing me!

btsimonh commented 5 years ago

ok, now building without changing folder structure. https://github.com/btsimonh/esp32-ble2mqtt/tree/btsimonh-platformio2

btsimonh commented 5 years ago

Added prefix. The platform.io built binary is yet to prove itself.

shmuelzon commented 5 years ago

The platform.io built binary is yet to prove itself.

Still the same errors as above?

btsimonh commented 5 years ago

yes, and more. https://github.com/platformio/platform-espressif32/releases shows at pio 1.4.0 I was using esp-IDF 3.1. The specific error 'BT_L2CAP: slave connection parameters update failed, the parameters are out of range' seems related to timing; maybe I'm not running the right clock rates somewhere? Or just that device is out of spec, or expressif are over-aggressive with the spec. But I'm also getting 'BT_APPL: service change write ccc failed' and also a crash: 'Guru Meditation Error: Core 0 panic'ed (LoadStoreError). Exception was unhandled.'

I tried to build with pio 1.5.0 -> esp-idf 3.1.1, but I get build errors when it tries to build the sdk:

Compiling .pioenvs\esp32dev\nghttp\nghttp2\lib\nghttp2_http.o
CC:\Users\Simon\.platformio\packages\framework-espidf\components\nghttp\nghttp2\lib\nghttp2_helper.c: In function 'nghttp2_put_uint16be':ompiling .pioenvs\esp32dev\nghttp\nghttp2\lib\nghttp2_map.o

C:\Users\Simon\.platformio\packages\framework-espidf\components\nghttp\nghttp2\lib\nghttp2_helper.c:33:16: error: implicit declaration of function 'htons' [-Werror=implicit-function-declaration]
uint16_t x = htons(n);

I could probably exclude these files with sdkconfig. looking at https://github.com/espressif/esp-idf/releases it would seem that I'm wasting my time with 3.1 :).

I'm also looking at pio, and how I could ask for a specific commit of the esp-idf;

could you post the sdkconfig you end up with for comparison?

shmuelzon commented 5 years ago

See attached my sdkconfig.h. sdkconfig.h.txt

btsimonh commented 5 years ago

ok, built now with pio esp platform version 1.5.0 with build flag -Wno-implicit-function-declaration log.1.5.0.txt looking better :). so this should be using a vanilla 3.1.1 sdk (20 days old)..... where abouts in the versions is your specific SDK commit? they broke something after that? did you retry with the official 3.1.1? will go compare the sdkconfig now....

btsimonh commented 5 years ago

ok, had one of 'those' errors, and a possible scan start/stop bug? - I'll monitor it for a while.

␛[0;32mI (503849) BLE2MQTT: Discovered BLE device: 00:1a:22:09:2c:db, connecting␛[0m
␛[0;31mE (503859) BT_BTM: BTM_BleScan scan not active
␛[0m
␛[0;33mW (503859) BT_APPL:  bta_dm_ble_scan stop scan failed, status=0x6
␛[0m
␛[0;33mW (503869) BT_L2CAP: L2CAP - LE - cannot start new connection at conn st: 1␛[0m
␛[0;31mE (503869) BLE: Scanning stopping failed, status: 0x1␛[0m
␛[0;31mE (519099) BT_HCI: btu_hcif_hdl_command_complete opcode 0x200c status 0xc␛[0m
␛[0;31mE (519249) BT_BTM: BTM_BleScan scan already active␛[0m
␛[0;33mW (519249) BT_APPL:  bta_dm_ble_scan start scan failed. status=0x6
␛[0m
␛[0;31mE (519259) BLE: Scanning starting failed, status: 0x1␛[0m
␛[0;31mE (519279) BT_L2CAP: slave connection parameters update failed, the parameters are out of range␛[0m
␛[0;33mW (520069) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=259 reason=0x0016␛[0m

btsimonh commented 5 years ago

ok, my sdkconfig seems to be based on the 'default' pio one for 3.1.1 It has a lot of extra fields, mainly _TRACE type things (setting debug levels?). sdkconfig.h.txt I can't see anything specifically scarily different.

shmuelzon commented 5 years ago

The 139d49894 tag I use is somewhere between 3.0 and 3.1. I've tried the 3.1 release but still saw issues with it.

Those aren't the errors I've seen. My problems were frequent WiFi and BLE disconnections along with sporadic crashes. I think they relate to WiFi/BLE coexistence and the crashes might be related to the MQTT implementation. Since 3.2 includes a different MQTT implementation, I'm hoping it will be more stable.

As for the configuration changes, the only main difference I see is CONFIG_SW_COEXIST_ENABLE.

btsimonh commented 5 years ago

more like this ?:

I (2195779) wifi: bcn_timout,ap_probe_send_start
␛[0;31mE (2196229) esp_mqtt: lwmqtt_unsubscribe_one: -6␛[0m
␛[0;33mW (2197429) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0016␛[0m
␛[0;31mE (2198229) esp_mqtt: lwmqtt_unsubscribe_one: -6␛[0m
I (2198279) wifi: ap_probe_send over, resett wifi status to disassoc
I (2198279) wifi: state: run -> init (100)
I (2198289) wifi: pm stop, total sleep time: 1258123681 us / 2196404204 us

I (2198289) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
␛[0;31mE (2198299) esp_mqtt: lwmqtt_keep_alive: -6␛[0m
␛[0;32mI (2198299) WiFi: Disconnected␛[0m
␛[0;32mI (2198299) esp_mqtt: esp_mqtt_process: exit task␛[0m
␛[0;32mI (2198299) BLE2MQTT: Disonnected from WiFi, stopping MQTT␛[0m
␛[0;32mI (2198319) MQTT: Disconnecting MQTT client␛[0m
␛[0;32mI (2198309) MQTT: MQTT client disconnected␛[0m
␛[0;33mW (2198299) esp_mqtt: esp_mqtt_unsubscribe: not connected␛[0m
␛[0;32mI (2198339) BLE2MQTT: Disonnected from MQTT, stopping BLE␛[0m
␛[0;32mI (2198339) MQTT: Connecting MQTT client␛[0m
␛[0;32mI (2198339) esp_mqtt: esp_mqtt_start: create task␛[0m
␛[0;32mI (2198349) esp_mqtt: esp_mqtt_process: begin connection attempt␛[0m
␛[0;32mI (2198349) BLE: Connection closed, reason = 0x16␛[0m
␛[0;32mI (2198359) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:9a␛[0m
␛[0;33mW (2198389) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=515 reason=0x0016␛[0m
␛[0;31mE (2198389) BT_BTM: BTM_BleScan scan not active
␛[0m
␛[0;33mW (2198389) BT_APPL:  bta_dm_ble_scan stop scan failed, status=0x6
␛[0m
␛[0;32mI (2198399) BLE: Connection closed, reason = 0x0␛[0m
␛[0;32mI (2198399) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:db␛[0m
␛[0;31mE (2198419) BLE: Scanning stopping failed, status: 0x1␛[0m
I (2198449) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
I (2198449) wifi: state: init -> auth (b0)
I (2198459) wifi: state: auth -> assoc (0)
I (2198459) wifi: state: assoc -> run (10)
I (2198479) wifi: connected with MYSSID, channel 11
I (2198479) wifi: pm start, type: 1

␛[0;32mI (2198489) WiFi: Connected␛[0m
␛[0;32mI (2199409) event: sta ip: 192.168.1.176, mask: 255.255.255.0, gw: 192.168.1.1␛[0m
␛[0;32mI (2199409) BLE2MQTT: Connected to WiFi, connecting to MQTT␛[0m

btsimonh commented 5 years ago

and maybe memory issue:

I (2330949) wifi: bcn_timout,ap_probe_send_start
W (2330949) wifi: alloc eb len=76 type=2 fail, heap:11968

W (2330949) wifi: m f probe req l=0

␛[0;31mE (2331809) esp_mqtt: lwmqtt_unsubscribe_one: -6␛[0m
W (2331949) wifi: alloc eb len=76 type=2 fail, heap:11996

W (2331949) wifi: m f probe req l=0

␛[0;33mW (2332649) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=515 reason=0x0016␛[0m
␛[0;31mE (2332649) BT_BTC: btc_gattc_cback transfer failed
␛[0m
␛[0;31mE (2332649) BT_BTC: btc_gattc_cback transfer failed
␛[0m
W (2332949) wifi: alloc eb len=76 type=2 fail, heap:14796

W (2332949) wifi: m f probe req l=0

I (2333449) wifi: ap_probe_send over, resett wifi status to disassoc
I (2333449) wifi: state: run -> init (100)
I (2333449) wifi: pm stop, total sleep time: 51734830 us / 109767671 us

I (2333459) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
␛[0;32mI (2333469) WiFi: Disconnected␛[0m
␛[

btsimonh commented 5 years ago

yep, and now died, last messages: - will reboot and monitor today

␛[0;31mE (2356089) MQTT: Failed subscribing to mqtt2ble/test/00:1a:22:09:2c:9a/Connected␛[0m
␛[0;33mW (2356109) BT_APPL: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0016␛[0m
␛[0;32mI (2356109) BLE: Connection closed, reason = 0x0␛[0m
␛[0;32mI (2356119) BLE2MQTT: Disconnected from device: 00:1a:22:09:2c:9a␛[0m

btsimonh commented 5 years ago

ok, i've updated esp_mqtt, fixed a port number issue with the update (changed from int to char *). I was getting regular disconnects from MQTT, which then causes all sorts of race-hell in the BLE disconnect etc. makign everything difficult to track. So I reduced the code down to publish of BLE connect only. Then I found that esp_mqtt was giving a consistent disconnect after ~25s, and once I had a:

Task watchdog got triggered. The following tasks did not reset the watchdog in time:
 - IDLE (CPU 1)
Tasks currently running:
CPU 0: IDLE
CPU 1: esp_mqtt

I can't see where esp_mqtt ever actually sleeps, so added a sleep in there of 10ms if no data was available. But I also created a new task in blemqtt which published an alive message every 5s. I have now not had an mqtt disconnect in 1500 seconds. Anyway, I'll play some more and add functionality back in to see if a base level issue is resolved, and let you know

shmuelzon commented 5 years ago

Yes, those were the issues I've had with the MQTT implementation. It's not worth the effort of trying to fix it as I will move to the implementation provided by Espressif in v3.2.

I'll start porting to v3.2-beta1 soon and then we'll know if it's any better.

Meanwhile, I've committed the MQTT prefix option, feel free to test it out.

btsimonh commented 5 years ago

I've started to single-thread everything; too many callbacks operating in too many contexts :(. But yes, I think you are right; not worth the time. Lost a day of 'real' work to this today.... started adding a mechanism to push commands into a queue to serialise to the mqtt thread - but its a lot of work I've not got time for at the moment.

btsimonh commented 5 years ago

you could add the platformio.ini, and heavily caveat it. At least it builds and runs.

shmuelzon commented 5 years ago

I’ve just replaced the MQTT implementation to the one EspressIf uses (curently it’s an external componenent, in v3.2 it’s built in). So far it seems to work just fine but I will let it run for, at least, the night before I push it. The good news is that it seems to already queue up everything and only run from its own thread. BLE also does this, so I think we’re, pretty much, in the clear from that aspect.

DurandA commented 5 years ago

I will try with a Thingy:52 to see if it improves stability once you pushed the 3.2-based version. Thanks for the great work.

shmuelzon commented 5 years ago

The new MQTT implementation was already pushed. Let me know how it goes...

btsimonh commented 5 years ago

hmmm.. does not connect to mqtt. no logs in mosquitto:

tried by hostname, then by IP.

built with 3.1.1 (in theory). branch here https://github.com/btsimonh/esp32-ble2mqtt/tree/platformio2

log from reset button:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5696
ho 0 tail 12 room 4
load:0x40078000,len:7768
ho 0 tail 12 room 4
load:0x40080000,len:5892
entry 0x40080318
␛[0;32mI (31) boot: ESP-IDF 3.30101.0 2nd stage bootloader␛[0m
␛[0;32mI (31) boot: compile time 20:53:36␛[0m
␛[0;32mI (32) boot: Enabling RNG early entropy source...␛[0m
␛[0;32mI (36) boot: SPI Speed      : 40MHz␛[0m
␛[0;32mI (40) boot: SPI Mode       : DIO␛[0m
␛[0;32mI (44) boot: SPI Flash Size : 4MB␛[0m
␛[0;32mI (48) boot: Partition Table:␛[0m
␛[0;32mI (52) boot: ## Label            Usage          Type ST Offset   Length␛[0m
␛[0;32mI (59) boot:  0 nvs              WiFi data        01 02 00009000 00004000␛[0m
␛[0;32mI (67) boot:  1 otadata          OTA data         01 00 0000d000 00002000␛[0m
␛[0;32mI (74) boot:  2 phy_init         RF data          01 01 0000f000 00001000␛[0m
␛[0;32mI (82) boot:  3 ota_0            OTA app          00 10 00010000 00180000␛[0m
␛[0;32mI (89) boot:  4 ota_1            OTA app          00 11 00190000 00180000␛[0m
␛[0;32mI (97) boot:  5 storage          Unknown data     01 82 00310000 00080000␛[0m
␛[0;32mI (104) boot: End of partition table␛[0m
␛[0;32mI (108) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x308bc (198844) map␛[0m
␛[0;32mI (188) esp_image: segment 1: paddr=0x000408e4 vaddr=0x3ffc0000 size=0x05a90 ( 23184) load␛[0m
␛[0;32mI (197) esp_image: segment 2: paddr=0x0004637c vaddr=0x3ffc5a90 size=0x00000 (     0) load␛[0m
␛[0;32mI (197) esp_image: segment 3: paddr=0x00046384 vaddr=0x40080000 size=0x00400 (  1024) load␛[0m
␛[0;32mI (206) esp_image: segment 4: paddr=0x0004678c vaddr=0x40080400 size=0x09884 ( 39044) load␛[0m
␛[0;32mI (230) esp_image: segment 5: paddr=0x00050018 vaddr=0x400d0018 size=0xcdaf0 (842480) map␛[0m
␛[0;32mI (528) esp_image: segment 6: paddr=0x0011db10 vaddr=0x40089c84 size=0x0b338 ( 45880) load␛[0m
␛[0;32mI (547) esp_image: segment 7: paddr=0x00128e50 vaddr=0x400c0000 size=0x00000 (     0) load␛[0m
␛[0;32mI (547) esp_image: segment 8: paddr=0x00128e58 vaddr=0x50000000 size=0x00000 (     0) load␛[0m
␛[0;32mI (566) boot: Loaded app from partition at offset 0x10000␛[0m
␛[0;32mI (566) boot: Disabling RNG early entropy source...␛[0m
␛[0;32mI (568) cpu_start: Pro cpu up.␛[0m
␛[0;32mI (571) cpu_start: Starting app cpu, entry point is 0x4008132c␛[0m
␛[0;32mI (0) cpu_start: App cpu up.␛[0m
␛[0;32mI (581) heap_init: Initializing. RAM available for dynamic allocation:␛[0m
␛[0;32mI (588) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM␛[0m
␛[0;32mI (594) heap_init: At 3FFD9C28 len 000063D8 (24 KiB): DRAM␛[0m
␛[0;32mI (600) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM␛[0m
␛[0;32mI (607) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM␛[0m
␛[0;32mI (613) heap_init: At 40094FBC len 0000B044 (44 KiB): IRAM␛[0m
␛[0;32mI (619) cpu_start: Pro cpu start user code␛[0m
␛[0;32mI (302) cpu_start: Starting scheduler on PRO CPU.␛[0m
␛[0;32mI (0) cpu_start: Starting scheduler on APP CPU.␛[0m
␛[0;32mI (348) BLE2MQTT: Version: 1␛[0m
␛[0;32mI (348) Config: Initializing configuration␛[0m
␛[0;32mI (388) Config: version: 70893f4d772a0e371bccf55cbea46aeb␛[0m
␛[0;32mI (388) OTA: Initializing OTA␛[0m
I (388) wifi: wifi driver task: 3ffe3e78, prio:23, stack:3584, core=0
I (388) wifi: wifi firmware version: b2c9a19
I (398) wifi: config NVS flash: enabled
I (398) wifi: config nano formating: disabled
␛[0;32mI (408) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE␛[0m
␛[0;32mI (418) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE␛[0m
I (438) wifi: Init dynamic tx buffer num: 32
I (438) wifi: Init data frame dynamic rx buffer num: 32
I (438) wifi: Init management frame dynamic rx buffer num: 32
I (438) wifi: Init static rx buffer size: 1600
I (448) wifi: Init static rx buffer num: 10
I (448) wifi: Init dynamic rx buffer num: 32
␛[0;32mI (448) BTDM_INIT: BT controller compile version [7b0770a]
␛[0m
␛[0;32mI (458) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE␛[0m
␛[0;32mI (528) phy: phy_version: 4000, b6198fa, Sep  3 2018, 15:11:06, 0, 0␛[0m
␛[0;32mI (808) WiFi: Connecting to SSID MYSSID␛[0m
I (818) wifi: mode : sta (30:ae:a4:e8:9e:98)
I (948) wifi: n:11 0, o:1 0, ap:255 255, sta:11 0, prof:1
I (1778) wifi: state: init -> auth (b0)
I (1788) wifi: state: auth -> assoc (0)
I (1798) wifi: state: assoc -> run (10)
I (1808) wifi: connected with MYSSID, channel 11
I (1808) wifi: pm start, type: 1

␛[0;32mI (1808) WiFi: Connected␛[0m
␛[0;32mI (2388) event: sta ip: 192.168.1.212, mask: 255.255.255.0, gw: 192.168.1.1␛[0m
␛[0;32mI (2388) BLE2MQTT: Connected to WiFi, connecting to MQTT␛[0m
␛[0;32mI (2388) MQTT: Connecting MQTT client␛[0m
␛[0;32mI (2428) MQTT_CLIENT: Sending MQTT CONNECT message, type: 1, id: 0000␛[0m
␛[0;31mE (2428) MQTT_CLIENT: Error network response␛[0m
␛[0;32mI (2428) MQTT_CLIENT: Error MQTT Connected␛[0m
␛[0;32mI (2428) MQTT_CLIENT: Reconnect after 10000 ms␛[0m
␛[0;32mI (2438) MQTT: MQTT client disconnected␛[0m
␛[0;32mI (2438) BLE2MQTT: Disonnected from MQTT, stopping BLE␛[0m
␛[0;31mE (2448) BT_BTM: BTM_BleScan scan not active
␛[0m
␛[0;33mW (2448) BT_APPL:  bta_dm_ble_scan stop scan failed, status=0x6
␛[0m
␛[0;32mI (2458) MQTT: Connecting MQTT client␛[0m
␛[0;31mE (2458) BLE: Scanning stopping failed, status: 0x1␛[0m

shmuelzon commented 5 years ago

Well, I can't really account for MQTT failing to connect. I've tested this against the same ESP-IDF version as in the README and didn't have any issues. Can you please try to build without PlatformIO to see if it makes a difference?

I do, however, see a bug where stopping a BLE scan shouldn't be done if it wasn't started to begin with. But that's just aesthetics...

btsimonh commented 5 years ago

I'm resisting using the esp-IDF direct :) after tall this issue is about can we build in platfromio.. Diagnosis shows it connects at the TCP level, and does a send of the connect message, but the response is not read. I'm thinking IP conflict or something in my config.json;
could you post your config.json (user & pwd obscured)?

shmuelzon commented 5 years ago

Yeah, we kind-of lost track of the original issue :)

The relevant part of my configuration is:

  "mqtt": {
    "server": {
      "host": "192.168.1.98",
      "port": 1883
    },
    "publish": {
      "retain": true
    }
  }

My server is mosquitto running on a Raspberry Pi, if that helps... BTW - I've pushed a branch with porting to ESP-IDF v3.2-beta1. The WiFi and BLE connections are more stable than what I used to have, but I still have some weird crashes from time to time. Some seem like null dereferencing and others WDT on core 0.

btsimonh commented 5 years ago

ok, so no username & pwd. I have: { "wifi": { "ssid": "MYSSID", "password": "xxxxxxxxxxxx" }, "mqtt": { "server": { "host": "192.168.1.210", "port": 1883, "username": "xxxxx", "password": "xxxxxxx" }, "topics":{ "prefix": "t/" }, "publish": { "retain": true } } } as an experiment, I may add url, and target a different mqtt server that I have on WSS:, implemented in a different way.

shmuelzon commented 5 years ago

OK, let me know how it goes. If there's a bug, open another issue and I'll look into it.

btsimonh commented 5 years ago

ok, I've looked more into the v3.2 commits; it's currently feverish around wifi and ble, so I'm going to put my esp32 wrooms aside for a couple of months and hope for a stable 3.2 with more reliable functionality. Seems we're still really tight on RAM on this platform, and code size is also an issue :(. I successfully built and ran https://github.com/microreset/esp32-mqtt-ble - only to find that a) it's bigger than a single OTA partition b) it bleeds ram c) stalls in BLEScan regularly (task warning & death). All the proposed fixes for the arduino BLE library it uses are based on 3.2.... (see here and here - may be useful info in them for you....)

May as well close the issue for now?

shmuelzon commented 5 years ago

Before you completely give up, I suggest you try this project as was intended. Compile it with ESP-IDF directly (use the commit ID in the README) and use the latest tag of this repo, v0.3.2 (before the MQTT library change). I didn't have any RAM issues (out of memory?) and the 4MB of flash available on the ESP-WROOM-32 is enough for 2 OTA partitions and the SPIFFS partition.

The project you mention, while it does use PlatformIO :), seems to only track advertisements, it doesn't connect to the devices themselves but, just like it, I also use this project to control my devices via Home Assistant for quite some time now and it has yet to fail me.

We can get back to PlatformIO once a working stable version of ESP-IDF is available...

shmuelzon commented 5 years ago

I'm going to go ahead and close this for now. We can re-visit this when it becomes more relevant.

shmuelzon / esp32-ble2mqtt

thoughts on platform.io support #6

define CONFIG_ESP_MQTT_ENABLED 1

define CONFIG_ESP_MQTT_TASK_STACK_SIZE 4096

define CONFIG_ESP_MQTT_TASK_STACK_PRIORITY 5

define CONFIG_ESP_MQTT_EVENT_QUEUE_SIZE 64

define CONFIG_LWIP_SO_RCVBUF 1