HTML interface not loading correctly - Settings/Aliases not loaded to UI

[southwalesboy]

In the latest versions 1.11.0, the web interface does not display the Alias or the settings in the dialogue boxes. Restarting the device and refreshing the page multiple times the settings eventually appear

[sidoh]

Hard for me to be of much help here without a way to reproduce.

Can you share the output of /about and /settings? (make sure to censor any passwords) Could you also share a screenshot of browser dev tools showing the network tab when things aren't working as expected?

[sidoh]

Output from Serial logs would potentially be helpful as well.

[southwalesboy]

Hard for me to be of much help here without a way to reproduce.

Can you share the output of /about and /settings? (make sure to censor any passwords) Could you also share a screenshot of browser dev tools showing the network tab when things aren't working as expected?

/about gives:

{
    "firmware": "milight-hub",
    "version": "1.11.0",
    "ip_address": "192.168.50.162",
    "reset_reason": "Software/System restart",
    "variant": "nodemcuv2",
    "free_heap": 17304,
    "arduino_version": "3.1.2",
    "free_stack": 1604,
    "flash_used": 93874,
    "flash_total": 957314,
    "flash_pct_free": 90,
    "queue_stats": {
        "length": 0,
        "dropped_packets": 0
    }
}

/settings on Edge browser gives a blank page. On Chrome I get the following:

{"admin_username":"","admin_password":"","ce_pin":4,"csn_pin":15,"reset_pin":0,"led_pin":-2,"radio_interface_type":"nRF24","packet_repeats":21,"http_repeat_factor":2,"auto_restart_period":240,"mqtt_server":"192.168.50.10:1883","mqtt_username":"xxxxx","mqtt_password":"xxxxx","mqtt_topic_pattern":"milight/:device_id/:device_type/:group_id","mqtt_update_topic_pattern":"milight/update/:device_id/:device_type/:group_id","mqtt_state_topic_pattern":"milight/states/:device_id/:device_type/:group_id","mqtt_client_status_topic":"milight/client_status","simple_mqtt_client_status":true,"discovery_port":48899,"listen_repeats":3,"state_flush_interval":50,"mqtt_state_rate_limit":500,"mqtt_debounce_delay":500,"mqtt_retain":false,"packet_repeat_throttle_sensitivity":0,"packet_repeat_throttle_threshold":200,"packet_repeat_minimum":6,"enable_automatic_mode_switching":false,"led_mode_wifi_config":"Fast toggle","led_mode_wifi_failed":"On","led_mode_operating":"Off","led_mode_packet":"Fast blip","led_mode_packet_count":3,"hostname":"milight-hub","rf24_power_level":"MAX","rf24_listen_channel":"LOW","wifi_static_ip":"","wifi_static_ip_gateway":"","wifi_static_ip_netmask":"","packet_repeats_per_loop":7,"home_assistant_discovery_prefix":"mqttauto","wifi_mode":"g","default_transition_period":150,"rf24_channels":["LOW","MID","HIGH"],"device_ids":[4369,10302,24186,28235,28317,43690,43691,65296,65297],"gateway_configs":[],"group_state_fields":["state","brightness","mode","color_temp","bulb_mode","computed_color"]}

On Edge and Chrome all settings appear blank (even though the hub is working correctly):

[southwalesboy]

With regards to the Chorme Dev tools, is this what you was asking for:

[sidoh]

I'm guessing the call to /settings is failing transiently and doesn't have anything to do with browser (although good thought trying). I'm able to load /settings with Edge just fine on mine.

Hard to say what's going on here, but my best guess is there's something funky going on with your chip's flash memory. Are you able to collect serial logs?

[marcin1965]

I have the same issue. Version 1.10.9 is working fine (nodemcu v2). After upgrading to 1.11.1 I can't see any settings in the web interface. I can add an alias and ID, but these disappear after the page reloading. I've tried different web browsers, and it is the same. Do you have any other suggestions?

[sidoh]

Only the same debugging pointer I mentioned -- try checking serial logs. Hard for me to be of much help when I can't reproduce. :(

[domoticaDude]

I had the same problem closing the browser (firefox) and reload the page fixed it.

[marcin1965]

I erased flash and did a fresh install of 1.11.1 on nodemcu v2. Serial console connected (see below). I can configure all settings, but after the device restart I can't see them - all fields are empty, including Alias and Device ID /settings shows all settings, but they can not be seen in GUI Version 1.10.9 works fine and I can see all settings in GUI (the same web browser)

Serial console output:

`Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Saving 1 aliases Settings successfully updated Restarting... ��P9�͎����Reading 1 aliases loaded 1 aliases ERROR: Failed to connect to MQTT server rc=-2 wm:AutoConnect wm:Connect Wifi, ATTEMPT # 1 of 5 wm:Connecting to SAVED AP: kwilno2 wm:AutoConnect: SUCCESS *wm:STA IP Address: 192.168.1.15 Wifi connected succesfully

Setup complete (version 1.11.1) Unhandled websocket event: 10 Saving 1 aliases Saving 1 aliases Saving 1 aliases Unhandled websocket event: 10`

[SelverC]

Same issue as Marcin1965, downgrading to 1.10.9 works fine. Tried both 1.11.1 and 1.11.0 none of them work for seeing the settings. I'm using a ESP-12F

[sidoh]

Thanks for the reports -- definitely sounds like an issue.

I'm not getting it, though. The GET /settings route works as expected. Have hit it a few dozen times with no issue. The homepage also loads correctly. So I'll need some help figuring out what's going on.

@marcin1965 -- it looks like your ESP reboots in the middle of those logs. Was that something you did, or did it reboot on its own?

To everyone having this issue, the following would help a bunch:

1. Try loading GET /settings directly (you can just navigate to <ip-of-hub>/settings). Try several times. I suspect the call to this route is failing transiently. Confirming that this is the issue would be helpful.
2. If you notice issues here, capture serial logs while you're loading the route.
3. Try running a continuous ping pointed at the hub (ideally while you're loading the homepage). Let me know if there are any breaks in reachability.

[SelverC]

Thanks for the reports -- definitely sounds like an issue.

I'm not getting it, though. The GET /settings route works as expected. Have hit it a few dozen times with no issue. The homepage also loads correctly. So I'll need some help figuring out what's going on.

@marcin1965 -- it looks like your ESP reboots in the middle of those logs. Was that something you did, or did it reboot on its own?

To everyone having this issue, the following would help a bunch:

1. Try loading GET /settings directly (you can just navigate to <ip-of-hub>/settings). Try several times. I suspect the call to this route is failing transiently. Confirming that this is the issue would be helpful.
2. If you notice issues here, capture serial logs while you're loading the route.
3. Try running a continuous ping pointed at the hub (ideally while you're loading the homepage). Let me know if there are any breaks in reachability.

1.) It takes 2-3s to load every time(so it's slow, but it's loading).

I did notice that the backup of settings is not working(it tries to download but almost always fails).

I'll add some serial logs later if @marcin1965 doesn't do it before me

I did see this in the developer console when loading from the UI:

It says the XHR fails to load:

I did copy the CURL from the developer console and tried to replicate it and it as well fails(never finishes) with Curl:

curl 'http://192.168.1.230/settings' \ -H 'Accept: application/json, text/javascript, /; q=0.01' \ -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,sv;q=0.7' \ -H 'Connection: keep-alive' \ -H 'Referer: http://192.168.1.230/' \ -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36' \ -H 'X-Requested-With: XMLHttpRequest' \ --compressed \ --insecure

My guess would be that something is wrong with the Ajax call or that it expects it to be a lot quicker that it is

[sidoh]

This call should be quick (sub-second). It's just serving the contents of a file on flash. It is reliably fast on my system (I do pretty extensive testing before cutting releases):

 hey -c 1 -n 100 http://milight-hub/settings

Summary:
  Total:        17.4482 secs
  Slowest:      0.3388 secs
  Fastest:      0.0936 secs
  Average:      0.1745 secs
  Requests/sec: 5.7312

  Total data:   151500 bytes
  Size/request: 1515 bytes

Response time histogram:
  0.094 [1]     |■
  0.118 [38]    |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.143 [5]     |■■■■■
  0.167 [0]     |
  0.192 [1]     |■
  0.216 [20]    |■■■■■■■■■■■■■■■■■■■■■
  0.241 [20]    |■■■■■■■■■■■■■■■■■■■■■
  0.265 [6]     |■■■■■■
  0.290 [8]     |■■■■■■■■
  0.314 [0]     |
  0.339 [1]     |■

Latency distribution:
  10% in 0.0975 secs
  25% in 0.1018 secs
  50% in 0.2041 secs
  75% in 0.2257 secs
  90% in 0.2640 secs
  95% in 0.2695 secs
  99% in 0.3388 secs

My best guess here is something is either causing your ESP to crash on this call or something in the new Arduino SDK is causing network instability.

Can you try the following?

1. Loading /about instead of /settings. Is this always quick? (it would also be helpful to share the output of this route)
2. Run a continuous ping while you're testing and see if there are breaks in connectivity when a route is hanging

[SelverC]

This call should be quick (sub-second). It's just serving the contents of a file on flash. It is reliably fast on my system (I do pretty extensive testing before cutting releases):

 hey -c 1 -n 100 http://milight-hub/settings

Summary:
  Total:        17.4482 secs
  Slowest:      0.3388 secs
  Fastest:      0.0936 secs
  Average:      0.1745 secs
  Requests/sec: 5.7312

  Total data:   151500 bytes
  Size/request: 1515 bytes

Response time histogram:
  0.094 [1]     |■
  0.118 [38]    |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.143 [5]     |■■■■■
  0.167 [0]     |
  0.192 [1]     |■
  0.216 [20]    |■■■■■■■■■■■■■■■■■■■■■
  0.241 [20]    |■■■■■■■■■■■■■■■■■■■■■
  0.265 [6]     |■■■■■■
  0.290 [8]     |■■■■■■■■
  0.314 [0]     |
  0.339 [1]     |■

Latency distribution:
  10% in 0.0975 secs
  25% in 0.1018 secs
  50% in 0.2041 secs
  75% in 0.2257 secs
  90% in 0.2640 secs
  95% in 0.2695 secs
  99% in 0.3388 secs

My best guess here is something is either causing your ESP to crash on this call or something in the new Arduino SDK is causing network instability.

Can you try the following?

1. Loading /about instead of /settings. Is this always quick? (it would also be helpful to share the output of this route)
2. Run a continuous ping while you're testing and see if there are breaks in connectivity when a route is hanging

1. /about loads in 100ms every single time
2. When running a continuous ping it breaks connectivity every single time i load /settings but it works fine for /about.

ping 192.168.1.230 PING 192.168.1.230 (192.168.1.230): 56 data bytes 64 bytes from 192.168.1.230: icmp_seq=0 ttl=255 time=55.776 ms 64 bytes from 192.168.1.230: icmp_seq=1 ttl=255 time=13.588 ms 64 bytes from 192.168.1.230: icmp_seq=2 ttl=255 time=97.750 ms 64 bytes from 192.168.1.230: icmp_seq=3 ttl=255 time=22.788 ms 64 bytes from 192.168.1.230: icmp_seq=4 ttl=255 time=16.128 ms 64 bytes from 192.168.1.230: icmp_seq=5 ttl=255 time=14.441 ms 64 bytes from 192.168.1.230: icmp_seq=6 ttl=255 time=8.665 ms 64 bytes from 192.168.1.230: icmp_seq=7 ttl=255 time=12.537 ms 64 bytes from 192.168.1.230: icmp_seq=8 ttl=255 time=56.479 ms 64 bytes from 192.168.1.230: icmp_seq=9 ttl=255 time=14.109 ms 64 bytes from 192.168.1.230: icmp_seq=10 ttl=255 time=14.283 ms 64 bytes from 192.168.1.230: icmp_seq=11 ttl=255 time=11.106 ms 64 bytes from 192.168.1.230: icmp_seq=12 ttl=255 time=14.081 ms 64 bytes from 192.168.1.230: icmp_seq=13 ttl=255 time=35.654 ms 64 bytes from 192.168.1.230: icmp_seq=14 ttl=255 time=15.790 ms Request timeout for icmp_seq 15 64 bytes from 192.168.1.230: icmp_seq=16 ttl=255 time=77.854 ms 64 bytes from 192.168.1.230: icmp_seq=17 ttl=255 time=17.985 ms 64 bytes from 192.168.1.230: icmp_seq=18 ttl=255 time=34.869 ms 64 bytes from 192.168.1.230: icmp_seq=19 ttl=255 time=13.021 ms 64 bytes from 192.168.1.230: icmp_seq=20 ttl=255 time=13.938 ms 64 bytes from 192.168.1.230: icmp_seq=21 ttl=255 time=31.340 ms 64 bytes from 192.168.1.230: icmp_seq=22 ttl=255 time=9.117 ms 64 bytes from 192.168.1.230: icmp_seq=23 ttl=255 time=34.641 ms 64 bytes from 192.168.1.230: icmp_seq=24 ttl=255 time=24.141 ms 64 bytes from 192.168.1.230: icmp_seq=25 ttl=255 time=30.592 ms 64 bytes from 192.168.1.230: icmp_seq=26 ttl=255 time=20.599 ms 64 bytes from 192.168.1.230: icmp_seq=27 ttl=255 time=35.402 ms 64 bytes from 192.168.1.230: icmp_seq=28 ttl=255 time=26.719 ms 64 bytes from 192.168.1.230: icmp_seq=29 ttl=255 time=17.404 ms 64 bytes from 192.168.1.230: icmp_seq=30 ttl=255 time=33.183 ms 64 bytes from 192.168.1.230: icmp_seq=31 ttl=255 time=49.137 ms Request timeout for icmp_seq 32

[sidoh]

Okay, that's helpful. I'd expect the same behavior as /settings here, but can you also try to load /aliases.bin?

Serial logs will hopefully help here. It seems likely there's a crash or something similar.

Can you share the output of /about?

[SelverC]

Okay, that's helpful. I'd expect the same behavior as /settings here, but can you also try to load /aliases.bin?

Serial logs will hopefully help here. It seems likely there's a crash or something similar.

Can you share the output of /about?

/aliases.bin works fine, no disconnects when running the ping command and nothing in the serial log.

serial log when running /settings is: Saving 0 aliases Saving 0 aliases

Output of /about:

{"firmware":"milight-hub","version":"1.11.1","ip_address":"192.168.1.230","reset_reason":"External System","variant":"esp12","free_heap":20016,"arduino_version":"3.1.2","free_stack":2320,"flash_used":9538,"flash_total":957314,"flash_pct_free":99,"queue_stats":{"length":0,"dropped_packets":0}}

[marcin1965]

Thanks for the reports -- definitely sounds like an issue.

I'm not getting it, though. The GET /settings route works as expected. Have hit it a few dozen times with no issue. The homepage also loads correctly. So I'll need some help figuring out what's going on.

@marcin1965 -- it looks like your ESP reboots in the middle of those logs. Was that something you did, or did it reboot on its own?

I did a restart after providing all settings. So, it was an intentional restart.

To everyone having this issue, the following would help a bunch:

1. Try loading GET /settings directly (you can just navigate to <ip-of-hub>/settings). Try several times. I suspect the call to this route is failing transiently. Confirming that this is the issue would be helpful. I did it 15 times, It is slow, takes 2-3 sec, but never breaks for me.
2. If you notice issues here, capture serial logs while you're loading the route.

no issues with calling /settings there is an issue while calling the homepage:

Unhandled websocket event: 10

3. Try running a continuous ping pointed at the hub (ideally while you're loading the homepage). Let me know if there are any breaks in reachability.

It happens that there is a packet lost for both calling homepage and /settings But not always, I would say once per 5 calls in average

[marcin1965]

One more comment - the board can remember settings, and it works. Domoticz can connect via mqtt and control the lights.

However, I can't use web GUI because it is missing the device ID - I can see calls in mqtt broker but with 0x0 device id (e.g. if I press ON/OFF). Web GUI can't display all the settings (they are empty), but mqtt connection works ok. But calls are not effective because the device ID is missing.

If this is helpful, we can connect via Teams or some other software. I can demonstrate the issue.

I tried a different nodemcu board, and it has the same issue.

[sidoh]

Okay, that's helpful. I'd expect the same behavior as /settings here, but can you also try to load /aliases.bin? Serial logs will hopefully help here. It seems likely there's a crash or something similar. Can you share the output of /about?

/aliases.bin works fine, no disconnects when running the ping command and nothing in the serial log.

serial log when running /settings is: Saving 0 aliases Saving 0 aliases

Output of /about:

{"firmware":"milight-hub","version":"1.11.1","ip_address":"192.168.1.230","reset_reason":"External System","variant":"esp12","free_heap":20016,"arduino_version":"3.1.2","free_stack":2320,"flash_used":9538,"flash_total":957314,"flash_pct_free":99,"queue_stats":{"length":0,"dropped_packets":0}}

Hmmm this is all pretty confusing.

Can you try a route that modifies something on flash? Maybe something like:

curl -X PUT -H 'Content-Type: application/json' -d '{"packet_repeats":49}' 10.133.8.88/setting

One more comment - the board can remember settings, and it works. Domoticz can connect via mqtt and control the lights.

However, I can't use web GUI because it is missing the device ID - I can see calls in mqtt broker but with 0x0 device id (e.g. if I press ON/OFF). Web GUI can't display all the settings (they are empty), but mqtt connection works ok. But calls are not effective because the device ID is missing.

If this is helpful, we can connect via Teams or some other software. I can demonstrate the issue.

I tried a different nodemcu board, and it has the same issue.

Right, I'm guessing what's going on is the call to /settings fails sometimes for whatever mysterious reason, which would cause the UI to not function.

What board are you using? If you have a direct Amazon (or similar) link I can get one to test with.

I've tested with d1_minis and nodemcuv3s.

[marcin1965]

I'm using nodemcu v2: https://www.amazon.pl/ElectroWorldFR-NodeMcu-rozwojowa-kompatybilna-Raspberry/dp/B0CDGY6HBC/ref=sr_1_6?crid=1F34M0TI7ZAFN&keywords=nodemcu+v2&qid=1699708752&sprefix=NodeMcu+V2%2Caps%2C119&sr=8-6

I've bought today nodemcu v3 and can test it around Tue/Wed once it comes to me.

[SelverC]

`> > > Okay, that's helpful. I'd expect the same behavior as/settingshere, but can you also try to load/aliases.bin`?

Serial logs will hopefully help here. It seems likely there's a crash or something similar. Can you share the output of /about?

/aliases.bin works fine, no disconnects when running the ping command and nothing in the serial log. serial log when running /settings is: Saving 0 aliases Saving 0 aliases Output of /about: {"firmware":"milight-hub","version":"1.11.1","ip_address":"192.168.1.230","reset_reason":"External System","variant":"esp12","free_heap":20016,"arduino_version":"3.1.2","free_stack":2320,"flash_used":9538,"flash_total":957314,"flash_pct_free":99,"queue_stats":{"length":0,"dropped_packets":0}}

Hmmm this is all pretty confusing.

Can you try a route that modifies something on flash? Maybe something like:

curl -X PUT -H 'Content-Type: application/json' -d '{"packet_repeats":49}' 10.133.8.88/setting

One more comment - the board can remember settings, and it works. Domoticz can connect via mqtt and control the lights. However, I can't use web GUI because it is missing the device ID - I can see calls in mqtt broker but with 0x0 device id (e.g. if I press ON/OFF). Web GUI can't display all the settings (they are empty), but mqtt connection works ok. But calls are not effective because the device ID is missing. If this is helpful, we can connect via Teams or some other software. I can demonstrate the issue. I tried a different nodemcu board, and it has the same issue.

Right, I'm guessing what's going on is the call to /settings fails sometimes for whatever mysterious reason, which would cause the UI to not function.

What board are you using? If you have a direct Amazon (or similar) link I can get one to test with.

I've tested with d1_minis and nodemcuv3s.

I'm using Witty board (https://www.amazon.com/ESP8266-Serial-Development-ESP-12F-nodemcu/dp/B0CJM7DDJZ/ref=sr_1_1?crid=9VWSM9TT9IIN&keywords=witty+esp8266&qid=1699712437&sprefix=witty+esp826%2Caps%2C157&sr=8-1)

Using that curl worked fine but it cleared all my other settings it seems because it stopped working until I re-added the settings.

I have a feeling it could be something with my and Marcin1965 settings? So here are my settings from the bin


√ßí0{"admin_username":"","admin_password":"","ce_pin":4,"csn_pin":15,"reset_pin":0,"led_pin":-2,"radio_interface_type":"nRF24","packet_repeats":500,"http_repeat_factor":0,"auto_restart_period":720,"mqtt_server":"192.168.1.42:1883","mqtt_username":"user","mqtt_password":"pass","mqtt_topic_pattern":"milight/:device_id/:device_type/:group_id","mqtt_update_topic_pattern":"milight/updates/:device_id/:device_type/:group_id","mqtt_state_topic_pattern":"milight/state/:device_id/:device_type/:group_id","mqtt_client_status_topic":"","simple_mqtt_client_status":false,"discovery_port":0,"listen_repeats":0,"state_flush_interval":10,"mqtt_state_rate_limit":500,"mqtt_debounce_delay":500,"mqtt_retain":true,"packet_repeat_throttle_sensitivity":10,"packet_repeat_throttle_threshold":500,"packet_repeat_minimum":50,"enable_automatic_mode_switching":false,"led_mode_wifi_config":"Fast toggle","led_mode_wifi_failed":"On","led_mode_operating":"Slow blip","led_mode_packet":"Flicker","led_mode_packet_count":3,"hostname":"milight-hub","rf24_power_level":"MAX","rf24_listen_channel":"LOW","wifi_static_ip":"192.168.1.220","wifi_static_ip_gateway":"192.168.1.1","wifi_static_ip_netmask":"255.255.255.0","packet_repeats_per_loop":50,"home_assistant_discovery_prefix":"homeassistant","wifi_mode":"g","default_transition_period":500,"rf24_channels":["LOW","MID","HIGH"],"device_ids":[0,64,80,255,1000,35436],"gateway_configs":[],"group_state_fields":["state","brightness","color","mode","color_temp","bulb_mode"]}

[sidoh]

I'm using nodemcu v2: https://www.amazon.pl/ElectroWorldFR-NodeMcu-rozwojowa-kompatybilna-Raspberry/dp/B0CDGY6HBC/ref=sr_1_6?crid=1F34M0TI7ZAFN&keywords=nodemcu+v2&qid=1699708752&sprefix=NodeMcu+V2%2Caps%2C119&sr=8-6

I've bought today nodemcu v3 and can test it around Tue/Wed once it comes to me.

Hmm... IIRC nodemcu v3 is almost the same hardware as v2 with a different (bigger) footprint. Let me know how it works, though.

`> > > Okay, that's helpful. I'd expect the same behavior as/settingshere, but can you also try to load/aliases.bin`?

Serial logs will hopefully help here. It seems likely there's a crash or something similar. Can you share the output of /about?

/aliases.bin works fine, no disconnects when running the ping command and nothing in the serial log. serial log when running /settings is: Saving 0 aliases Saving 0 aliases Output of /about: {"firmware":"milight-hub","version":"1.11.1","ip_address":"192.168.1.230","reset_reason":"External System","variant":"esp12","free_heap":20016,"arduino_version":"3.1.2","free_stack":2320,"flash_used":9538,"flash_total":957314,"flash_pct_free":99,"queue_stats":{"length":0,"dropped_packets":0}}

Hmmm this is all pretty confusing. Can you try a route that modifies something on flash? Maybe something like:

curl -X PUT -H 'Content-Type: application/json' -d '{"packet_repeats":49}' 10.133.8.88/setting

One more comment - the board can remember settings, and it works. Domoticz can connect via mqtt and control the lights. However, I can't use web GUI because it is missing the device ID - I can see calls in mqtt broker but with 0x0 device id (e.g. if I press ON/OFF). Web GUI can't display all the settings (they are empty), but mqtt connection works ok. But calls are not effective because the device ID is missing. If this is helpful, we can connect via Teams or some other software. I can demonstrate the issue. I tried a different nodemcu board, and it has the same issue.

Right, I'm guessing what's going on is the call to /settings fails sometimes for whatever mysterious reason, which would cause the UI to not function. What board are you using? If you have a direct Amazon (or similar) link I can get one to test with. I've tested with d1_minis and nodemcuv3s.

I'm using Witty board (https://www.amazon.com/ESP8266-Serial-Development-ESP-12F-nodemcu/dp/B0CJM7DDJZ/ref=sr_1_1?crid=9VWSM9TT9IIN&keywords=witty+esp8266&qid=1699712437&sprefix=witty+esp826%2Caps%2C157&sr=8-1)

Using that curl worked fine but it cleared all my other settings it seems because it stopped working until I re-added the settings.

I have a feeling it could be something with my and Marcin1965 settings? So here are my settings from the bin

�√ßí0�{"admin_username":"","admin_password":"","ce_pin":4,"csn_pin":15,"reset_pin":0,"led_pin":-2,"radio_interface_type":"nRF24","packet_repeats":500,"http_repeat_factor":0,"auto_restart_period":720,"mqtt_server":"192.168.1.42:1883","mqtt_username":"user","mqtt_password":"pass","mqtt_topic_pattern":"milight/:device_id/:device_type/:group_id","mqtt_update_topic_pattern":"milight/updates/:device_id/:device_type/:group_id","mqtt_state_topic_pattern":"milight/state/:device_id/:device_type/:group_id","mqtt_client_status_topic":"","simple_mqtt_client_status":false,"discovery_port":0,"listen_repeats":0,"state_flush_interval":10,"mqtt_state_rate_limit":500,"mqtt_debounce_delay":500,"mqtt_retain":true,"packet_repeat_throttle_sensitivity":10,"packet_repeat_throttle_threshold":500,"packet_repeat_minimum":50,"enable_automatic_mode_switching":false,"led_mode_wifi_config":"Fast toggle","led_mode_wifi_failed":"On","led_mode_operating":"Slow blip","led_mode_packet":"Flicker","led_mode_packet_count":3,"hostname":"milight-hub","rf24_power_level":"MAX","rf24_listen_channel":"LOW","wifi_static_ip":"192.168.1.220","wifi_static_ip_gateway":"192.168.1.1","wifi_static_ip_netmask":"255.255.255.0","packet_repeats_per_loop":50,"home_assistant_discovery_prefix":"homeassistant","wifi_mode":"g","default_transition_period":500,"rf24_channels":["LOW","MID","HIGH"],"device_ids":[0,64,80,255,1000,35436],"gateway_configs":[],"group_state_fields":["state","brightness","color","mode","color_temp","bulb_mode"]}

Interesting, thanks for sharing.

All of this points at some kinda wacky problems with flash. If editing any setting is clearing the whole thing, I suspect there's some kind of problem modifying flash.

Can you try adding an alias with something like this?

> curl -X POST -H 'Content-Type: application/json' -d '{"alias":"abc", "device_id": 1, "group_id": 1, "device_type": "rgb_cct"}' milight-hub/aliases
{"success":true,"id":1}%

You should be able to see the added alias with this:

> curl milight-hub/aliases
{"aliases":[{"alias":"abc","id":1,"device_id":1,"group_id":1,"device_type":"rgb_cct"}],"page":1,"count":1,"num_pages":1}

If you haven't yet, can you try wiping flash with something like this and then re-flashing?

> python3 -m esptool erase_flash

Could you also share the command (or steps) you're using to install the firmware?

[marcin1965]

While trying to add an alias, it says: {"error":"Alias already exists: kuchnia"}%

"kuchnia" is the name of my alias, so the setting is there but can't be seen via GUI

Commands I'm using to install the firmware:

esptool.py --port /dev/cu.usbserial-0001 --baud 115200 erase_flash

esptool.py --port /dev/cu.usbserial-0001 --baud 115200 write_flash -fm=dio -fs=4MB 0x00000 ./esp8266_milight_hub_nodemcuv2-1.11.1.bin

[marcin1965]

One more odd observation:

1. I open web GUI and can't see both alias and device ID
2. I try to add an alias, and it works ok - alias is added
3. I try to add the device id, and it is added, but the alias disappears
4. I open the drop-down menu, and alias is there, but if you select it then devices id is disappearing
5. It is not possible to have both, if you select one of them, another is disappearing.
6. If I reload the GUI or restart the device, I can't see any of them - they are also not available in the drop-down menus

This is rather not a problem with Flash - the device is working ok with Domoticz via mqtt. You just can't see the settings in GUI and use GUI as well.

[sidoh]

Yeah, I think I understand roughly what's going on (although I have no clue what's causing it). There's something about saving the settings that is causing some kind of flash corruption. But it's not happening for me! This is why it's so confusing. I have tried with 5 different ESP boards at this point with no issue. Have installed with platformio and with the same esptool commands you're using.

It's possible that you and @SelverC have different issues. Perhaps we should produce a set of commands that definitely produces some issue for you.

It sounds like you're able to save aliases, but let's just confirm (choose an alias that doesn't exist):

> curl -X POST -H 'Content-Type: application/json' -d '{"alias":"abc", "device_id": 1, "group_id": 1, "device_type": "rgb_cct"}' milight-hub/aliases
{"success":true,"id":1}%

verify that it shows up here:

> curl milight-hub/aliases
{"aliases":[{"alias":"abc","id":1,"device_id":1,"group_id":1,"device_type":"rgb_cct"}],"page":1,"count":1,"num_pages":1}

change a setting key:

> curl -X PUT -H 'Content-Type: application/json' -d '{"packet_repeats":49}' 10.133.8.229/settings
{"success":true}%

verify that it stuck:

> curl -s 10.133.8.229/settings | jq '.packet_repeats'
49

if all of this is working, can you take a screenshot of your web browser's dev tools?

[marcin1965]

The first three works with no problem.

curl -s 192.168.1.15/settings | jq '.packet_repeats' gives: jq: parse error: Unfinished string at EOF at line 1, column 949

[sidoh]

Can you paste the output of curl -s 192.168.1.15/settings? (If you have any passwords, make sure to censor them)

[marcin1965]

curl -s 192.168.1.15/settings

{"admin_username":"","admin_password":"","ce_pin":4,"csn_pin":15,"reset_pin":0,"led_pin":-2,"radio_interface_type":"nRF24","packet_repeats":49,"http_repeat_factor":1,"auto_restart_period":0,"mqtt_server":"192.168.1.11:1883","mqtt_username":"","mqtt_password":"","mqtt_topic_pattern":"milight/:device_id/:device_type/:group_id","mqtt_update_topic_pattern":"milight/updates/:device_id/:device_type/:group_id","mqtt_state_topic_pattern":"milight/states/:device_id/:device_type/:group_id","mqtt_client_status_topic":"","simple_mqtt_client_status":false,"discovery_port":48899,"listen_repeats":3,"state_flush_interval":10000,"mqtt_state_rate_limit":500,"mqtt_debounce_delay":500,"mqtt_retain":true,"packet_repeat_throttle_sensitivity":0,"packet_repeat_throttle_threshold":200,"packet_repeat_minimum":3,"enable_automatic_mode_switching":false,"led_mode_wifi_config":"Fast toggle","led_mode_wifi_failed":"On","led_mode_operating":"Off","led_mode_packet":"F%

[SelverC]

1.) curl -X POST -H 'Content-Type: application/json' -d '{"alias":"abc", "device_id": 1, "group_id": 1, "device_type": "rgb_cct"}' 192.168.1.230/aliases

{"success":true,"id":2}%

2.) curl 192.168.1.230/aliases

{"aliases":[{"alias":"abc","id":2,"device_id":1,"group_id":1,"device_type":"rgb_cct"},{"alias":"test","id":1,"device_id":1,"group_id":0,"device_type":"rgb_cct"}],"page":1,"count":2,"num_pages":1}%

3.) curl -X PUT -H 'Content-Type: application/json' -d '{"packet_repeats":49}' 192.168.1.230/settings {"success":true}%

4.) curl -s 192.168.1.230/settings | jq '.packet_repeats' jq: parse error: Unfinished string at EOF at line 1, column 949

I have the same results as @marcin1965

And here is my @sidoh curl -s 192.168.1.230/settings

^[[A^[[B{"admin_username":"","admin_password":"","ce_pin":4,"csn_pin":15,"reset_pin":0,"led_pin":-2,"radio_interface_type":"nRF24","packet_repeats":49,"http_repeat_factor":3,"auto_restart_period":1440,"mqtt_server":"192.168.1.42:1883","mqtt_username":"user","mqtt_password":"password","mqtt_topic_pattern":"milight/:device_id/:device_type/:group_id","mqtt_update_topic_pattern":"milight/updates/:device_id/:device_type/:group_id","mqtt_state_topic_pattern":"milight/state/:device_id/:device_type/:group_id","mqtt_client_status_topic":"","simple_mqtt_client_status":false,"discovery_port":48899,"listen_repeats":0,"state_flush_interval":10,"mqtt_state_rate_limit":500,"mqtt_debounce_delay":500,"mqtt_retain":true,"packet_repeat_throttle_sensitivity":10,"packet_repeat_throttle_threshold":500,"packet_repeat_minimum":50,"enable_automatic_mode_switching":false,"led_mode_wifi_config":"Fast toggle","led_mode_wifi_failed":"On","led_mode_operatin%

[SelverC]

Hi @sidoh,

Have you had any time to figure out something? Just ping if you need anything more.

[marcin1965]

FYI

I reported the issue for NodeMCU v2. I have two pieces and both of them have this issue.

Now I bought NodeMCU v3 and it is working fine with version 1.11.1

[sidoh]

Hey guys, sorry for the slow response here. This looks like the same issue -- something is causing your settings file to get truncated when it's updated. I just went through every spare ESP8266 I own (about 20 split across d1 minis, nodemcu v2/v3s) without seeing this issue while trying to replicate using the exact same steps.

I'm at a bit of a loss here. There's clearly an issue, but I'm kind of left needing to assume there's some kind of hardware compatibility problem that is likely below the application layer.

The biggest change from 1.10 to 1.11 is a bunch of library/SDK upgrades. In particular, I upgraded the version of the arduino SDK from one released 5+ years ago. Perhaps there's some kind of weird incompatibility with older flash chips?

Do you guys see any meaningful differences in the output of esptool flash_id?

> python3 -m esptool flash_id
esptool.py v4.6.2
Found 1 serial ports
Serial port COM3
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: c8:c9:a3:10:b7:7e
Uploading stub...
Running stub...
Stub running...
Manufacturer: 5e
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

[marcin1965]

For the board that has issues with settings (NodeMCU v2) the only difference is: Manufacturer: 54

[sidoh]

`> > > Okay, that's helpful. I'd expect the same behavior as/settingshere, but can you also try to load/aliases.bin`?

Serial logs will hopefully help here. It seems likely there's a crash or something similar. Can you share the output of /about?

/aliases.bin works fine, no disconnects when running the ping command and nothing in the serial log. serial log when running /settings is: Saving 0 aliases Saving 0 aliases Output of /about: {"firmware":"milight-hub","version":"1.11.1","ip_address":"192.168.1.230","reset_reason":"External System","variant":"esp12","free_heap":20016,"arduino_version":"3.1.2","free_stack":2320,"flash_used":9538,"flash_total":957314,"flash_pct_free":99,"queue_stats":{"length":0,"dropped_packets":0}}

Hmmm this is all pretty confusing. Can you try a route that modifies something on flash? Maybe something like:

curl -X PUT -H 'Content-Type: application/json' -d '{"packet_repeats":49}' 10.133.8.88/setting

One more comment - the board can remember settings, and it works. Domoticz can connect via mqtt and control the lights. However, I can't use web GUI because it is missing the device ID - I can see calls in mqtt broker but with 0x0 device id (e.g. if I press ON/OFF). Web GUI can't display all the settings (they are empty), but mqtt connection works ok. But calls are not effective because the device ID is missing. If this is helpful, we can connect via Teams or some other software. I can demonstrate the issue. I tried a different nodemcu board, and it has the same issue.

Right, I'm guessing what's going on is the call to /settings fails sometimes for whatever mysterious reason, which would cause the UI to not function. What board are you using? If you have a direct Amazon (or similar) link I can get one to test with. I've tested with d1_minis and nodemcuv3s.

I'm using Witty board (https://www.amazon.com/ESP8266-Serial-Development-ESP-12F-nodemcu/dp/B0CJM7DDJZ/ref=sr_1_1?crid=9VWSM9TT9IIN&keywords=witty+esp8266&qid=1699712437&sprefix=witty+esp826%2Caps%2C157&sr=8-1)

Using that curl worked fine but it cleared all my other settings it seems because it stopped working until I re-added the settings.

I have a feeling it could be something with my and Marcin1965 settings? So here are my settings from the bin

�√ßí0�{"admin_username":"","admin_password":"","ce_pin":4,"csn_pin":15,"reset_pin":0,"led_pin":-2,"radio_interface_type":"nRF24","packet_repeats":500,"http_repeat_factor":0,"auto_restart_period":720,"mqtt_server":"192.168.1.42:1883","mqtt_username":"user","mqtt_password":"pass","mqtt_topic_pattern":"milight/:device_id/:device_type/:group_id","mqtt_update_topic_pattern":"milight/updates/:device_id/:device_type/:group_id","mqtt_state_topic_pattern":"milight/state/:device_id/:device_type/:group_id","mqtt_client_status_topic":"","simple_mqtt_client_status":false,"discovery_port":0,"listen_repeats":0,"state_flush_interval":10,"mqtt_state_rate_limit":500,"mqtt_debounce_delay":500,"mqtt_retain":true,"packet_repeat_throttle_sensitivity":10,"packet_repeat_throttle_threshold":500,"packet_repeat_minimum":50,"enable_automatic_mode_switching":false,"led_mode_wifi_config":"Fast toggle","led_mode_wifi_failed":"On","led_mode_operating":"Slow blip","led_mode_packet":"Flicker","led_mode_packet_count":3,"hostname":"milight-hub","rf24_power_level":"MAX","rf24_listen_channel":"LOW","wifi_static_ip":"192.168.1.220","wifi_static_ip_gateway":"192.168.1.1","wifi_static_ip_netmask":"255.255.255.0","packet_repeats_per_loop":50,"home_assistant_discovery_prefix":"homeassistant","wifi_mode":"g","default_transition_period":500,"rf24_channels":["LOW","MID","HIGH"],"device_ids":[0,64,80,255,1000,35436],"gateway_configs":[],"group_state_fields":["state","brightness","color","mode","color_temp","bulb_mode"]}

Just got the witty board. Tested out changing settings and everything is working fine.

> python3 -m esptool flash_id
esptool.py v3.3.1
Found 1 serial ports
Serial port COM3
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 08:3a:8d:e7:e1:3d
Uploading stub...
Running stub...
Stub running...
Manufacturer: 5e
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

Sort of at a loss here. If anyone has any ideas on how I might be able to replicate this, I'm all ears

[marcin1965]

@sidoh I can send you my NodeMCU v2 that has the issue. I already replaced it with NodeMCU v3, which is working fine.

[SelverC]

I did buy a new nodemcuv3 but it didn't help. So I tried to manually to transfer the settings(Seems like something in the conversion didn't work, between the two version there is a change between backup settings).

And one more change that I did is to re-wire it(my cables were not correct but it worked anyway? maybe that caused the crash as well).

Not sure exactly what helped me but I would start fresh don't backup settings and make sure all the wires are correct

Update: after a reboot the issue returned so still no solution for me :-(

[markg85]

I've not gone over all messages here but based on some those that i did, my findings might be interesting to some.

The issue i had was the light state not updating properly on home assistant. After firing up an MQTT explorer i noticed:

When turning lights on/off the state updates were only updated in this part:

The topic named states didn't change. So for completeness, that's this part:

This confused me as my settings were just the default ones and they clearly said:

I didn't change anything in the settings but just - to be sure - pressed submit again. Low and behold, the states are now updated in the states topic! Amazing :)

For me, this fixed the issue i had.

@sidoh you might have a bug here. I'm guessing submitting the values in the settings page puts those values in some config file which you - at some other point - probably load in the code and use for actual MQTT topic stuff. My guess is that whatever variables you use, they probably empty if you haven't submitted the settings yet. If this assumption is correct then i'm still quite curious where those default values in the settings page come from. Clearly there's an edge case where you do see them in the field but they aren't used in the code else i wouldn't even have this bug to begin with.

I'm still wondering why i now have milight/states/:hex_device_id AND milight/:hex_device_id in my mqtt. Both update it's values when i toggle a light though the latter seems to be just the on/off state. It seems, according to the settings, that milight/:hex_device_id should not exist when you have your state topic set as milight/states/:hex_device_id.

[rhamblen]

Ive got the same problem on a mix of D1 mini old (6+years) and new (6months) and a NodeMCU v1 (6+ years) Used to work fine but in last 6-8 months have shown this problem. Windows 10, windows 11, chrome and edge, and IOS with safari Rebooting and refreshing does fix the problem. BUT to be honest I use these via MQTT and home automation so the web page is more a backup

[markg85]

I just had this problem again. Completely out of the blue. Saving settings (while not changing anything) and rebooting fixed it. Again.

So incredible weird.