Port 8443 closed after reboot

fredericseiler commented 3 months ago

Hi,

Something goes wrong during my Tahoma Switch (fw 2024.3.3-10) startup sequence:

Every time it reboots (e.g. power outage during a storm), port 8443 is closed.

The only solution I found so far is to re-sync it ("pin in the hole" reset procedure) when it happens. Port 8443 will then stay open until the next reboot.

What can I do to debug this?

greg-szabo commented 1 month ago

I have a similar problem with my Tahoma Switch, firmware 2024.4.3-11. This is what I found out so far.

The boot process takes about 35-45 seconds. At the end, the white status light at the bottom of the device is on. At this point, I can connect to the local API endpoint:

curl -k -X 'GET' \
  'https://gateway-xxxx-xxxx-xxxx.local:8443/enduser-mobile-web/1/enduserAPI/apiVersion' \
  -H 'accept: application/json'
{"protocolVersion":"2024.4.3-11"}

After about 5-15 seconds, the status light blinks red three times. After that, I cannot connect to the endpoint anymore. Connecting to port 8443 times out (after the standard 5 seconds). The port is closed.

A power cycle shows the same symptoms. Local API port is available for a few seconds after reboot, but it's closed after the three red blinks.

I reset the device on the web interface (with the option that the house was sold, but the device was left in place). This turned off the local API feature, and after registering the device again, I had to request to activate it again.

If the local API was enabled on the web interface, the local API endpoint worked until the next reboot. So, it seems that the local API thread/process borks after reboot for some unknown reason. But it can happily run if the activation process started it.

After the next reboot, the above described symptoms happen again.

The "pin in the hole" reset procedure, did not work for me. I assume, you mean holding a pin in the bottom hole for about 7-10 seconds and resetting the device. This put it into discovery mode (blue light blinking on the top) and I had to give it the Wifi details again. It did not enable the local API, though. If there is a way to restart the device without unplugging it, please share.

fredericseiler commented 1 month ago

you mean holding a pin in the bottom hole for about 7-10 seconds and resetting the device.

Unplug, pin in the hole, plug, wait until red light blinks, remove the pin. The Tahoma Switch will re-apply its settings.

For now, I've put it behind a UPS power backup.

greg-szabo commented 1 month ago

Thank you for the clarification; I was not aware of this method. I can confirm that using the "pin in the hole" method you described, I have the device running and responding to local API requests. (So we have the same issue, yay.)

On a side note, even with the pin-in-the-hole bootup, there is a three-blink red LED after bootup, but the local API stays responsive. So, that might be a literal red herring. It's just a good indicator of how long the local API is responsive when doing a regular boot.

Do you have any more information (or did you find any documentation) on the pin-in-the-hole boot process? Is it some kind of extended/safe/maintenance mode boot that might do something differently?

I also considered putting the device on a power backup, but I'd still like to get to the bottom of the issue. The device will be installed in a barely accessible place, so I'd rather not deal with pins and reboots.

Any developers here who could help? I'm happy to gather dev logs (I'm happy to read through them myself too).

fredericseiler commented 1 month ago

Is it some kind of extended/safe/maintenance mode boot that might do something differently?

It's some mystical voodoo out-of-nowhere procedure that french users are sharing on Somfy forums, I don't even know if there's something real about it (not officially documented anywhere), but hey, it's working 🤷‍♂️

Any developers here who could help?

Good luck with that.

greg-szabo commented 1 month ago

I've done a bit more digging. The less exciting finding is that the Tahoma Switch uses HTTPS (TLSv1.3 / TLSv1.2) and client certificates for communication, a welcome delight in the Internet-of-Things space. However, it makes it hard to see what the communication is about. Also, it uses ports 443 and 803 for some reason.

The more interesting finding is that with the pin-in-the-hole boot, the device seems to boot up twice: the first boot looks like it's downloading a recovery firmware (about 4.5MB), and then it does a regular boot. It could be a way to recover from a dead firmware upgrade.

The download during the first boot is initiated from cdn.overkiz.com, Somfy's IoT partner. I contacted them on their website for help, and I'll write an update here if they respond.

I think the pin-in-the-hole boot boots into a different firmware than the regular boot. I have a feeling something has changed in the release firmware that's causing the local API port to close (possibly a process dies), but that has not changed in the recovery firmware. This is all hypothetical, as we can't see into the device. I hope the developers at Overkiz can help.

greg-szabo commented 1 month ago

Quick update:

I went through the Somfy Pro programming guide for dealers. It's not as detailed as I expected or as we would need it here.
I signed up to SomfyU and went through the training for the Tahoma Beecon, Switch and MyLink devices and the RTS technology. They talk about the LED lights in some detail, but it's not enough for our needs. I learned that a MyLink wouldn't be appropriate for me as I have more than 16 channels to manage.
My Tahoma Switch is developing faults every 1-2 days and disconnects for various reasons. (Red LED.) I have it on a UPS now, so it might be the wifi rearranging itself. (I use Ubiquiti Unifi Access Points.) I ordered an Ethernet Adaptor to try to work around any wi-fi issues. I'll report back when I was able to test it. The point is that because of the instability, I need the firmware to be able to keep port 8443 open without any special reboots. I can deal with regular reboots.

Finally, Overkiz sent a blanket statement:

Dear Greg,

Sorry but our API are not open to every customer.
We are just providing it to our B2B customer in case of a connected project.

Have a nice day.
Sebastien

I'm willing to put developer time into fixing this issue. I hope Somfy can point me in the right direction because I like their technology and because of SomfyU, I now even know about their company history. I actually have the Tahoma Beecon too, but I bought the Tahoma Switch specifically for the local API feature. (I ran the Beecon for a year with Home Assistant and the API limits are killing my automations. I have more than 20 shades that I programmed to go up and down every day and I had to put delays in to keep it working. I know about aggregating multiple motors on a channel.)

@vhenriet-sfy , @llavorel-somfy , is there anything we can do to help this issue move forward?

I develop open-source software daily. If you need it, I'm willing to help professionally to troubleshoot and fix this issue.

iMicknl commented 1 month ago

My Tahoma Switch is developing faults every 1-2 days and disconnects for various reasons. (Red LED.) I have it on a UPS now, so it might be the wifi rearranging itself. (I use Ubiquiti Unifi Access Points.)

I actually have the same issue for my TaHoma Switch (which I only use for development with Local API, no real devices connected). I stopped caring about it, and it is just permanently on with the red led. I also use Ubiquity WiFi access points.

qelanhari commented 1 month ago

That is strange, I use ubiquity access points (Used 3 different models, Lite, AC Pro and AC HD), also behind a UPS (Eaton), and I've never had an issue like that.

greg-szabo commented 1 month ago

I only meant the Ubiquiti comment as a side note, but apparently, it struck a nerve for people. :) I remember this older issue, #131, where Ubiquiti was mentioned, along with the local API not responding. (I thought that issue was more networking-related. Hence, I started adding my notes to this one.)

To keep things clear, let's keep this issue focused on "port 8443 closed". We can open a new connectivity issue that is Ubiquiti-related. I suggest doing some tests (possibly with a different brand access point) and submitting the results in the new issue.

For now, I plan to work around Ubiquiti with the Ethernet adaptor. If that makes a difference, I'll post it here.

Unfortunately, the web shop where I ordered the adaptor told me that "Somfy has been on shutdown since the middle of September due to changing to a new system." and I should expect an extra 2 weeks of delay at least.

fredericseiler commented 1 month ago

Well, it seems the latest update fixed my problem. 🤷‍♂️

acommuni commented 3 days ago

I have the same issue with my Tahoma Switch, the Local Api port is closed after a reboot. I have the last version 1.28.

I have to reset the box with a pin in order to reactivate the local API port, but the port is closed after a reboot.

This issue is not closed for me.

acommuni commented 2 days ago

How can I help you ?

Somfy-Developer / Somfy-TaHoma-Developer-Mode

Port 8443 closed after reboot #139