Closed jslcom closed 1 year ago
I have a bunch of uncommitted changes for the stsdk module I need to commit and also fix the build. I have another branch I am working on travis building so we can confirm everything builds that would include the stsdk binary.
The key can be changed via command line but the onboarding_config.json is fixed and I 100% agree this needs to be added to the configuration file. I will keep this open to track that change.
The older releases here do contain the stsdk version for manual flashing. If I can resolve some of these build issues it would be good to get it back into the current releases. Any help appreciated so be prepared to help me test :c)
Best SM
TODO: R&D needs to be done to see if it is ok to just store key known values or store the entire json to allow free form. Presumably the use of fixed would not be very future proof. The config supports multi line entries but the CLI command do not. Maybe best to just keep the file outside of the standard config and let the stsdk module load from the file system and look for it on the SD card if attached falling back to the spiffs partition if not found and failing with error if no onboarding_config.json is found.
I have a bunch of uncommitted changes for the stsdk module I need to commit and also fix the build. I have another branch I am working on travis building so we can confirm everything builds that would include the stsdk binary.
The key can be changed via command line but the onboarding_config.json is fixed and I 100% agree this needs to be added to the configuration file. I will keep this open to track that change.
The older releases here do contain the stsdk version for manual flashing. If I can resolve some of these build issues it would be good to get it back into the current releases. Any help appreciated so be prepared to help me test :c)
Best SM
Sean, Thanks for the reply, and it's good to see that you're still working on this. I retired a few months ago so I have more time to fiddle with things like this. I would be more than happy to help you develop & test some of this stuff. Let me know if there's anything I can do. I just got my Amazon delivery so I should have uSD very soon. https://www.amazon.com/dp/B0989SM146 https://www.amazon.com/dp/B08J4HJ98L
Thanks again, Jeff
TODO: R&D needs to be done to see if it is ok to just store key known values or store the entire json to allow free form. Presumably the use of fixed would not be very future proof. The config supports multi line entries but the CLI command do not. Maybe best to just keep the file outside of the standard config and let the stsdk module load from the file system and look for it on the SD card if attached falling back to the spiffs partition if not found and failing with error if no onboarding_config.json is found.
Sean, I've been unsuccessful in getting my uSD card to work with the pre-built code. It looks like your choice of GPIO pins (MOSI=15, MISO=2, CLK=14, CS=low) is not compatible with my ESP32-WROOM32UE. If the source code would build, then I could easily change this, but it doesn't right now and I don't think that it's worth the effort to disassemble the image and try to patch it. I can either wait for you to get the current revision to build, or I could try building an older rev. Can you recommend one?
Thanks, Jeff
Ok in dev branch it should be able to be built again. Let me know. I still need to do a clean and try building from scratch again downloading the stsdk stuff all again etc. Updated README docs on my build steps after testing again.
@jslcom I was able to run some tests on the dev branch and got the build steps nailed down a little better in the README.
I tested uploading and configuring a device but I was not able to adopt it. I kept getting an error and it did not look like my phone was even talking to the AD2IoT. Usually I see some logs as it connects as a client but I did not see anything. I am on 2.4 wifi so that should not be the issue. I will keep testing but I suspect it is my phone I probably need to check permissions. TBD.
Please test and let me know if you have any success.
Best SM
@jslcom I was able to run some tests on the dev branch and got the build steps nailed down a little better in the README.
I tested uploading and configuring a device but I was not able to adopt it. I kept getting an error and it did not look like my phone was even talking to the AD2IoT. Usually I see some logs as it connects as a client but I did not see anything. I am on 2.4 wifi so that should not be the issue. I will keep testing but I suspect it is my phone I probably need to check permissions. TBD.
Please test and let me know if you have any success.
Best SM
Thanks Sean. I downloaded the dev blob yesterday and I've just begun to give it a try. I'll give you a status update later (hopefully today). I'll be heading out of town for a while after tomorrow, but I'll try to make some time to continue working on this.
Happy New Year! Jeff
@jslcom I was able to run some tests on the dev branch and got the build steps nailed down a little better in the README. I tested uploading and configuring a device but I was not able to adopt it. I kept getting an error and it did not look like my phone was even talking to the AD2IoT. Usually I see some logs as it connects as a client but I did not see anything. I am on 2.4 wifi so that should not be the issue. I will keep testing but I suspect it is my phone I probably need to check permissions. TBD. Please test and let me know if you have any success. Best SM
Thanks Sean. I downloaded the dev blob yesterday and I've just begun to give it a try. I'll give you a status update later (hopefully today). I'll be heading out of town for a while after tomorrow, but I'll try to make some time to continue working on this.
Happy New Year! Jeff
Okay... I tried building both in my new and old environments (which should have been the same, but behaved a bit differently because I forgot to update everything). Both build environments now get through the compilation/link process and produce a flash-able image. (Thank you Sean!)
I will now proceed with adjustments to the SPI GPIO definitions so my uSD card will work.
Update: uSD card is working, but with some consequences:
1) I can no longer flash new code while the uSD adapter is connected. This probably has something to do with pull-up resistors. For now, I can disconnect the reader when re-flashing. (It's a good thing that I used a connector between the target and the reader.)
2) Running the (webui) code with the uSD card attached crashes after trying to initialize the HW ETH PHY. Probably because some of the GPIO pins are shared and the Ethernet code is getting invalid status: !IOT: N (5729) UARTCLI: Starting main task. !IOT: N (5729) AD2_IoT: 'netmode' set to 'E'. !IOT: I (5729) HAL: network TCP/IP stack init start !IOT: I (5739) HAL: network TCP/IP stack init finish !IOT: I (5739) HAL: ETH hardware init start !IOT: I (5739) gpio: GPIO[12]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0 !IOT: I (5769) system_api: Base MAC address is not set !IOT: I (5769) system_api: read default base MAC address from EFUSE !IOT: E (5789) lan87xx: lan87xx_init(499): wrong chip ID !IOT: E (5789) esp_eth: esp_eth_driver_install(223): init phy failedESP_ERROR_CHECK failed: esp_err_t 0xffffffff (ESP_FA4 file: "../main/device_control.cpp" line 702 func: void hal_init_eth(std::__cxx11::string&) expression: esp_eth_driver_install(&config, ð_handle)
abort() was called at PC 0x40088947 on core 0
Backtrace:0x40081962:0x3ffbb0b00x40088951:0x3ffbb0d0 0x4008f8da:0x3ffbb0f0 0x40088947:0x3ffbb160 0x400e06db:0x3ffbb180 0
ELF file SHA256: 55bc157586e30257
I (5831) esp_core_dump_flash: Save core dump to flash...
I (5836) esp_core_dump_flash: Erase flash 20480 bytes @ 0x70000
I (6093) esp_core_dump_flash: Write end offset 0x4084, check sum length 4
I (6093) esp_core_dump_flash: Core dump has been saved to flash.
Rebooting...
3) Disabling the HW ETH PHY in menuconfig renders the code UN-buildable. I added some more conditionals to device_control.cpp so it builds and runs now, but now the WiFi doesn't work. I need to stop now, but I will pick it up again in a few days.
4) While doing my initial troubleshooting on the uSD card, I saw a condition where the card would not mount, but the behavior did not match the code. It should have printed an error after "!IOT: N (619) HAL: Mounting uSD on '/sdcard': fail.", but the error was never displayed. I suspect memory corruption, but I did not dive into it.
Happy New Year! Jeff
I recall reading something about boot issues with uSD and I recall it was related to what GPIO pin was used for special boot pins. I have many ESP32 boards here and breadboard and flash reader boards so I can duplicate your work if you provide some details.
I would set netmode to wifi only. It should allowing compiling with ETH code still enabled but wont cause any issues because the driver wont be loaded. This should prevent issues with Wifi also breaking. At least that was my intent when I redid several times the startup code. If it is not working I would like to get that fixed so people dont have to make too many changes in menuconfig.
Best SM
I recall reading something about boot issues with uSD and I recall it was related to what GPIO pin was used for special boot pins. I have many ESP32 boards here and breadboard and flash reader boards so I can duplicate your work if you provide some details.
I would set netmode to wifi only. It should allowing compiling with ETH code still enabled but wont cause any issues because the driver wont be loaded. This should prevent issues with Wifi also breaking. At least that was my intent when I redid several times the startup code. If it is not working I would like to get that fixed so people dont have to make too many changes in menuconfig.
Best SM
Thanks for the reply Sean. I'm still traveling out of state, but I've VPN'ed into my other house so the dev board can connect to the alarm panel there. After a full-rebuild with the same skconfig file it's working better. The /sdcard is there and the WiFi network comes up. It talks to the alarm panel (via my Raspberry Pi-based AlarmDecoder and ser2sock), and it tries to grab a OTA update. I can connect via a web browser, but there's not much functionality probably because I have not edited the default configuration files on the /sdcard. There are a number of errors indicated on the console and I've tried to attach a capture file so you can look at them if you like, but Github says; "Something went really wrong, and we can't process that file. Try again." I will paste the output below. It's only 14k bytes. I will now try to create a valid configuration...
Jeff
I (473) cpu_start: Single core mode I (481) cpu_start: Pro cpu start user code I (481) cpu_start: cpu freq: 160000000 I (481) cpu_start: Application information: I (481) cpu_start: Project name: alarmdecoder_ad2iot_esp32 I (487) cpu_start: App version: v1.7.0-50-g6ec0fe5-dirty I (492) cpu_start: Compile time: Dec 30 2022 21:40:11 I (497) cpu_start: ELF file SHA256: 4467296015b450d5... I (502) cpu_start: ESP-IDF: v4.4 I (506) heap_init: Initializing. RAM available for dynamic allocation: I (512) heap_init: At 3FF80000 len 00002000 (8 KiB): RTCRAM I (518) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM I (523) heap_init: At 3FFB9E90 len 00026170 (152 KiB): DRAM I (528) heap_init: At 3FFE0440 len 0001FBC0 (126 KiB): D/IRAM I (533) heap_init: At 40078000 len 00008000 (32 KiB): IRAM I (539) heap_init: At 40094AF0 len 0000B510 (45 KiB): IRAM I (545) spi_flash: detected chip: generic I (548) spi_flash: flash io: qio I (552) esp_core_dump_flash: Init core dump to flash I (556) esp_core_dump_flash: Found partition 'coredump' @ 70000 65536 bytes I (574) esp_core_dump_flash: Core dump data checksum is correct I (574) e
!IOT: N (582) AD2_IoT: Starting AlarmDecoder AD2IoT network appliance version (AD2IOT-1102) build flag (webui)
!IOT: N (582) HAL: ESP32 with 2 CPU cores, WiFi/BT/BLE, silicon revision 1, 4MB external flash
!IOT: N (592) Initialize NVS subsystem start. Done.
!IOT: N (602) HAL: Mounting SPIFFS on '/spiffs' : Done.
!IOT: N (612) HAL: SPIFFS partition size: 350396 B, free: 331069 B.
!IOT: N (612) HAL: Mounting uSD on '/sdcard': pass.
!IOT: N (642) HAL: uSD fat32 partition size: 31166464 KiB, free: 31165600 KiB.
!IOT: N (642) AD2UTIL: Attempting to load config file: sdcard/ad2iot.ini success.
!IOT: N (692) AD2_IoT: Approximate Configuration memory usage: 24204 B
!IOT: I (692) AD2API: AD2PStates[00040000] not found adding partition ID(1)
!IOT: N (702) AD2_IoT: init partition slot 1 address 18 zones '2,3,4,5,6,24,25,26'
!IOT: N (712) UARTCLI: Send '.' three times in the next 5 seconds to stop the init.
AD2IOT #
!IOT: N (5712) UARTCLI: Starting main task.
!IOT: N (5712) Delaying start of ad2source SOCKET after network is up.
!IOT: N (5712) AD2_IoT: 'netmode' set to 'W'.
!IOT: I (5722) HAL: network TCP/IP stack init start
!IOT: I (5722) HAL: network TCP/IP stack init finish
!IOT: I (5722) HAL: WiFi hardware init start
!IOT: I (5742) wifi:wifi driver task: 3ffcaa74, prio:23, stack:6656, core=0
!IOT: I (5742) system_api: Base MAC address is not set
!IOT: I (5742) system_api: read default base MAC address from EFUSE
!IOT: I (5752) wifi:wifi firmware version: 7679c42
!IOT: I (5752) wifi:wifi certification version: v7.0
!IOT: I (5762) wifi:config NVS flash: enabled
!IOT: I (5772) wifi:config nano formating: disabled
!IOT: I (5772) wifi:Init data frame dynamic rx buffer num: 32
!IOT: I (5772) wifi:Init management frame dynamic rx buffer num: 32
!IOT: I (5782) wifi:Init management short buffer num: 32
!IOT: I (5782) wifi:Init dynamic tx buffer num: 32
!IOT: I (5792) wifi:Init static rx buffer size: 1600
!IOT: I (5792) wifi:Init static rx buffer num: 10
!IOT: I (5792) wifi:Init dynamic rx buffer num: 32
!IOT: I (5802) wifi_init: rx ba win: 6
!IOT: I (5802) wifi_init: tcpip mbox: 32
!IOT: I (5802) wifi_init: udp mbox: 6
!IOT: I (5822) wifi_init: tcp mbox: 6
!IOT: I (5822) wifi_init: tcp tx win: 5744
!IOT: I (5822) wifi_init: tcp rx win: 5744
!IOT: I (5832) wifi_init: tcp mss: 1440
!IOT: I (5832) wifi_init: WiFi IRAM OP enabled
!IOT: I (5832) wifi_init: WiFi RX IRAM OP enabled
!IOT: I (5842) HAL: DHCP Mode selected
!IOT: I (5842) phy_init: phy_version 4670,719f9f6,Feb 18 2021,17:07:07
!IOT: I (5922) wifi:mode : sta (40:91:51:9b:c7:fc)
!IOT: I (5932) wifi:enable tsf
!IOT: I (5932) HAL: WiFi hardware init done
!IOT: N (5982) TWILIO: Init done. Found and configured 5 virtual switches.
!IOT: N (6022) PUSHOVER: Init done. Found and configured 6 virtual switches.
!IOT: N (6022) WEBUI: Init done, daemon starting.
!IOT: N (6022) MQTT client disabled.
!IOT: N (6032) FTPD: Init starting
!IOT: N (6042) FTPD: Init done. Daemon starting.
!IOT: I (6532) wifi:new:<6,0>, old:<1,0>, ap:<255,255>, sta:<6,0>, prof:1
!IOT: I (7082) wifi:state: init -> auth (b0)
!IOT: I (7082) wifi:state: auth -> assoc (0)
!IOT: I (7092) wifi:state: assoc -> run (10)
!IOT: I (7102) wifi:connected with JSL.COM, aid = 1, channel 6, BW20, bssid = 94:83:c4:22:67:11
!IOT: I (7102) wifi:security: WPA2-PSK, phy: bgn, rssi: -73
!IOT: I (7112) wifi:pm start, type: 1
!IOT: I (7112) HAL: WIFI_EVENT_STA_CONNECTED
!IOT: I (7152) wifi:AP's beacon interval = 102400 us, DTIM period = 2
!IOT: W (7232) wifi:
Looks good reviewing all of the log. Ya I am 99% sure the webui httpd_ws errors are all related to older HTML content on the flash that is not compatible with the current webui ws protocol. Just Copy the contrib/webUI/flash-drive/www to the root of the flash drive. The web server on the esp32 uses /sdcard/www for the document root on the newer release to be sure the config file that I see was successfully loaded from your logs in /sdcard/adtiot.ini is not directly accessible via a browser.
I saw the AD2 configuration update received over ser2sock and the ser2sock client connection. Not sure if it is a AD2 Web Appliance or some other HA system but it is typical to send the AD2 a config setting at first connect. The AD2IoT also addresses this with the ad2config setting and command. This will actually fight to keep the setting so a problem could exist where we have two systems who want to make a change and they conflict. This battle could ultimately destroy the AD2* flash. I will have to review the ESP32 code and be sure the battle is limited.
It looks like the ESP32-POE-ISO is using Single SPI mode to the SD card. I only see 3 lines CLOCK, DI and DO. Also it shows PULLUP on DI, DO and the other data lines that are not used are pulled up to 3v3. Also the CS line is pulled up to have it constantly enabled.
GPIO2 is part of the bootstrap logic but I think as long as GPIO0 is pulled UP GPIO2 will be ignored and it will preform a SPI Flash Boot.
It looks like you wired the flash for quad spi mode. See what happens if you match more closely the schematics of the ESP32-POE-IOS board and add a few pullup/down as needed on the unused pins on the flash board.
I recommend a 10uF cap be added to EN pin to avoid flashing problems. Some boards need it some don't just depends. Would indicate that adding this to the EN pin just prevents noise(signals) on the Enable line in general. This could likely be a much smaller cap and get the job done but 10k is assured this will only be a flag line not signal.
It looks like the ESP32-POE-ISO is using Single SPI mode to the SD card. I only see 3 lines CLOCK, DI and DO. Also it shows PULLUP on DI, DO and the other data lines that are not used are pulled up to 3v3. Also the CS line is pulled up to have it constantly enabled.
GPIO2 is part of the bootstrap logic but I think as long as GPIO0 is pulled UP GPIO2 will be ignored and it will preform a SPI Flash Boot.
It looks like you wired the flash for quad spi mode. See what happens if you match more closely the schematics of the ESP32-POE-IOS board and add a few pullup/down as needed on the unused pins on the flash board.
I recommend a 10uF cap be added to EN pin to avoid flashing problems. Some boards need it some don't just depends. Would indicate that adding this to the EN pin just prevents noise(signals) on the Enable line in general. This could likely be a much smaller cap and get the job done but 10k is assured this will only be a flag line not signal.
Thanks Sean. My uSD is wired for single SPI mode. I did wire the CS line and noted that you left it out. I'm fine with the setup that I have. I can disconnect the uSD reader whenever I want to re-flash the code. I'll put all the changes I made into my fork of your project so that you can use them if you like. I may not get that done until next week though.
Since I'm away from that house, I haven't tried the stsdk build yet. I'll get to that next week. The webui build seems to work pretty well with Chrome, but it doesn't seem to work at all with Firefox. I'll try to figure out why later.
Sean, I've updated my fork of your repo. My changes include:
The repo is at: https://github.com/jslcom/AlarmDecoder-IoT
I haven't done a whole lot with it since I got back from my trip. Right now I'm trying to get the webui to work with my system here. (Don't take this as a complaint because I would never complain about anything you've created here.) I'm having difficulty understanding the allowed syntax for some of the files (such as the zone information in the ad2iot.ini file). I don't want to bother you with questions when I can just dive deeper into the code to figure it out myself, but if you have any detailed documentation on the zone description syntax, please point me in the right direction.
BTW, I verified that the html on the uSD card matches what is in the repo here, and it continues to spew the same errors on the console when accessed from a browser: !IOT: W (203892) httpd_txrx: httpd_sock_err: error in recv : 11 !IOT: W (203892) httpd_ws: httpd_ws_recv_frame: Failed to receive payload !IOT: E (203892) WEBUI: httpd_ws_recv_frame failed with -1 !IOT: W (205122) httpd_ws: httpd_ws_recv_frame: WS frame is not properly masked. !IOT: E (205132) WEBUI: httpd_ws_recv_frame failed with 259
Thanks!
Jeff
This is a great project and I have some high hopes for doing some interesting integrations once I get it working.
I tried to get this going in February, but got stuck with some problems and dropped it for a while. I picked it up again a few days ago and started fresh. I'm trying to build this using command-line tools and no IDE. This should be possible. Below are a few of the issues I encountered while building on Ubuntu: 1) The build script encounters shell script syntax errors because the default system shell on Ubuntu is dash not bash. Rectified with: sudo dpkg-reconfigure dash 2) Directory empty at AlarmDecoder-IoT/components/twilio/lib/fmt. Obtained required source tree and manually placed it there. 3) Build failure from missing component: https://github.com/brofield/simpleini The platformio.ini file references this, but it's not clear to me how it integrates into the AlarmDecoder-IoT source tree. (I copied SimpleIni.h and ConvertUTF.h to main and it seems happy.) 4) Reference to undefined structure element: ../main/alarmdecoder_main.cpp: In function 'bool _show_pretty_process_list()': ../main/alarmdecoder_main.cpp:605:60: error: 'TaskStatus_t' {aka 'struct xTASK_STATUS'} has no member named 'xCoreID' (unsigned int)pxTSArray[x].xCoreID, ^
~~ 5) Undefined identifiers (STSDK_CONFIG_SECTION and STSDK_SUBCMD_ENABLE): ../main/alarmdecoder_main.cpp:1165:33: error: 'STSDK_CONFIG_SECTION' was not declared in this scope ad2_get_config_key_bool(STSDK_CONFIG_SECTION, STSDK_SUBCMD_ENABLE, &stEN); ^~~~~~~~ ../main/alarmdecoder_main.cpp:1165:33: note: suggested alternative: 'AD2PART_CONFIG_SECTION' ad2_get_config_key_bool(STSDK_CONFIG_SECTION, STSDK_SUBCMD_ENABLE, &stEN); ^~~~~~~~ AD2PART_CONFIG_SECTION ../main/alarmdecoder_main.cpp:1165:55: error: 'STSDK_SUBCMD_ENABLE' was not declared in this scope ad2_get_config_key_bool(STSDK_CONFIG_SECTION, STSDK_SUBCMD_ENABLE, &stEN); ^~~~~~~ ../main/alarmdecoder_main.cpp:1165:55: note: suggested alternative: 'STSDK_ENABLE' ad2_get_config_key_bool(STSDK_CONFIG_SECTION, STSDK_SUBCMD_ENABLE, &stEN); ^~~~~~~I'm going to keep plowing away at this. Last time I tried, I was able to get an image built and installed, but I encountered some other issues that may have been resolved since then. No filesystem was available to host the needed configuration files for the webui build. My platform has no microSD card and the internal flash-based filesystem didn't work. (I will have a microSD card on my platform after this Wednesday when the parts I ordered arrive.) Last time, I was able to flash new binaries from Nutech OTA, but they had the same filesystem deficiency and did not work.
The documentation describes two build types: webui and stsdk. The webui build does not include SmartThings integration and the stsdk build does not include the web interface. Is it possible to make a build with both of these features enabled? It does seem to be possible to create such a configuration with menuconfig, but I don't know if it will work.
Despite being unable to build this, I was able to load the pre-built binary (webui 1101) and try a few things. As expected, the web interface displays the following: "file not found. Connect a uSD card with a FAT32 partition and the html content in the root directory before the device starts." (I expect to have uSD capability on my target sometime soon after the parts I ordered from Amazon arrive.) After performing an OTA "upgrade", my version was downgraded to webui 1100. The "top" command is now missing, and the target can no longer access the OTA server: AD2IOT # upgrade webui AD2IOT # Starting OTA update with build flags 'webui'.
AD2IOT # !IOT: E (1914865) esp-tls: Failed to connnect to host (errno 113) !IOT: E (1914865) esp-tls: Failed to open new connection !IOT: E (1914865) TRANS_SSL: Failed to open a new connection !IOT: E (1914875) HTTP_CLIENT: Connection failed, sock < 0 !IOT: E (1914875) AD2OTA: ota_https_update_device: Failed to open HTTP connection: 28674 !IOT: E (1914885) AD2OTA: Firmware Upgrades Failed (28674) !IOT: E (1914885) AD2OTA: Exiting task due to fatal error... AD2IOT #
I noticed that the stsdk pre-built binary is unavailable here, but I was able to load it via OTA. I'm wondering if the developer key is present in the binary blob. If so, it would be great if the key were instead stored apart from the binary (somewhere in flash) so that image could be distributed too.
Thanks!
Jeff