Soma Integration does not read actual position from device

olaf7 commented 3 years ago

The problem

When using both (mixed usage) HA and the Soma Shades App, The Soma Shades App reads the actual position of a Soma device. The HA integration does not. So if you move or tilt shades the HA integration still thinks the blinds are in the position the last time the HA app was used. As this is not the case it can be that blinds can no longer be shut or opened or even the minumum and maximum positions are off.

What is version of Home Assistant Core has the issue?

2021.3.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Soma Connect

Link to integration documentation on our website

https://www.home-assistant.io/integrations/soma/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

Devices firmware and Soma App are latest versions. Same for HA App

probot-home-assistant[bot] commented 3 years ago

Hey there @ratsept, mind taking a look at this issue as its been labeled with an integration (soma) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

frenck commented 3 years ago

I think it should, however, this integration uses polling, thus it can take a while for it to pick it up.

ratsept commented 3 years ago

Hey,

Can you check what are the versions for both the Connect used to bridge this connection and the specific motor that is having this issue. We recently amended the position logic in Connect and the position data should now be cached on the Connect and updated from the motor BLE ADV packets. This should work better with large numbers of motors being polled often.

olaf7 commented 3 years ago

All my devices seem to be impacted. device 1 : SOMA Smart Shades 2 firmware 2.2.7 device 2 : SOMA Smart Shades 2 firmware 2.2.7 device 3 : SOMA Tilt 2 firmware 2.2.7 Soma Connect 2.2.5 Also see the log I did send 2 days ago or something through the app. (Is there another source for this besides the play store?)

ratsept commented 3 years ago

Hmm ... I will try to recreate this with my own system but my HA is at 2021.4.5 so I hope this won't make much difference. We also have a slightly newer version of firmware for Connect ready (https://support.somasmarthome.com/hc/en-us/articles/360035521234-Install-SOMA-Connect-software-on-SOMA-Connect-Raspberry-Pi) but that shouldn't change the position reporting part too much.

ratsept commented 3 years ago

It seems to work on my system for now. Can you please try updating your Connect to the latest version and see if the issue remains. The older versions of Connect used to make a new BLE connection for every call to the position and battery API. Newer versions should get this data from the Advertising packets and this should vastly reduce the number of connections that have to be made to the motors to keep everything up to date.

olaf7 commented 3 years ago

Good news. Unfortunately I will not be near the setup for some time to test it. In the meantime I tried to remotely upgrade the software by SSH-ing into the soma-connect. This fails as the Amazon hosted repository is not signed and gives a HTTP 403 error. Also it is a dev repo and not a stable one according to its name. Please advise. Is there a way (curl, python, ...) to check both values of the HA-plugin and the devices to see if they are in balance? (if so, I can partly test remotely)

ratsept commented 3 years ago

We don't really support or recommend updating over SSH. The safest way is to just rewrite the SD card with the newest image. I know this is not ideal and we really do want to improve the update process for Connect but unfortunately there is always something more pressing on our plates and so this has been a mess for a long time already. Sorry about that.

I'm not sure what you mean by "check both values of the HA-plugin and the devices to see if they are in balance". Do you mean the versions you are running? Or the position values of the motors? You can query the "/list_devices" API to see the Connect version. HA version should be easy to check in the frontend. You should be able to use curl to poll the Connect API. I think something like "curl 127.0.0.1:3000/list_devices" should work if you are in the Connect machine. I can't think of a way to get the motor firmware version from the API right now. But that is something that I should definitely add to the API now that I think about it.

olaf7 commented 3 years ago

If I could query a device directly through SomaConnect using the Connect API and could read the value (position of motor) and I also could read the value as currently known by HA I could compare them. Reading the value using the Soma Connect API seems possible according to https://support.somasmarthome.com/hc/en-us/articles/360026064234-HTTP-API If the same is possible using https://developers.home-assistant.io/docs/api/rest/ I could check if values match/are in sync or not. Have not tried/tested this yet. It might also open up the possibility to 'link' devices and block certain requests in code. Casus: vertical blinds with 2 Shades2 devices. One for tilting and one for moving the blinds over the rail. Not a good idea to do so when tilted to close position. But currently the the devices behave independent so this can happen if the user makes a mistake.

olaf7 commented 3 years ago

If the integration works with polling it would be nice if the polling frequency is configurable. Also the option to query when the last poll happened would be nice. Also for debugging purposes. I understand these will not be high on the to do list. Just a suggestion.

ratsept commented 3 years ago

The "linking" could definitely be done. But since that will be very user specific this would likely be best to do inside HA with automations for example.

Polling frequency should be pretty easy to make configurable. I'm not sure about the last poll thing - this is really my first and only HA programming experience and it was a pretty scary thing to get into. The last poll thing sounds like something that HA might provide out of the box already. If not then maybe this could be added as a property to the polled values.

olaf7 commented 3 years ago

In the meantime I created some code to test settings. It can be found here: https://github.com/olaf7/poc-soma Further testing would probably mostly requires my physical presence. This can take some time.

skallan commented 3 years ago

Hi, I am having the same problem and I don't think the issue is in the HA part but in the actual SOMA connect. @ratsept can you look at this or should I reach out to SOMA customer service? The HTTP call to get the state always responds with 100

If I am in the smart cover app this cover is 90% closed

If i make the call: http://XXX.XXX.XX.XXX:3000/get_shade_state/"MAC" Response: { "result": "success", "version": "2.2.7", "mac": "MAC", "position": 100 }

olaf7 commented 3 years ago

I have not yet found the time to do some physical testing, but this might help: https://github.com/olaf7/poc-soma The Soma Connect is now upgraded to the latest version. Still I feel the problem is more likely to reside in the HA Integration due to the flow (please adapt if incorrect!) as provided in a Mermaid graph in this repository.

olaf7 commented 3 years ago

Added a comparison/test script to above mentioned repository to check positional values. I got some nasty timeout issues (do not know why yet) which is the reason my setup is not fully tested (yet) and I cannot comment on results. Any feedback and improvements much appreciated.

olaf7 commented 3 years ago

Fixed bugs in compare.sh script in above repository. Still puzzled why I got a timeout. Requires further testing and comparison of actual position and reported positions. Will do so later. Other points of view/data much appreciated.

ratsept commented 3 years ago

Thank you for all the work on this. I'm sorry we've not been very involved. But I have passed this on to my colleague who is looking at this as soon as possible. We have been completely swamped with sourcing parts for the actual manufacturing due to the worldwide shortage of everything. This means that at some point soon there will be some minor changes in the hardware. Those will also bring slight changes to the firmware and likely also the app and Connect. All of these combined will hopefully also contain a fix for whatever is causing this. Right now our suspicion is that the error might even be on the motor firmware. The Connect currently uses the position value reported by the motor in ADV packets. And that value is cached multiple times in both the motor and Connect. So we think there might be an issue somewhere causing these caches to not update every time the position changes. This feature (ADV reporting and caching on the Connect) were added fairly recently and we never heard of this issue before that so this is at least in my mind the most likely thing to look at. I will try to keep you updated here on our progress as well.

skallan commented 3 years ago

Hi,

Great that you and your colleagues are looking into the code in the connect @ratsept When testing this and playing around a bit I can see that the issue is when I have been using the services cover.close_cover and cover.open_cover and stop it. Then the position is not updated to the position I stopped at. If I instead use cover.set_cover_position and let it run to the set position then the correct position will be saved on the cover entity.

I found a new error in the log of my setup:

If you want me to debug this further please give me some pointer on how.

Logger: homeassistant.helpers.entity Source: components/soma/cover.py:90 First occurred: June 7, 2021, 18:27:01 (120 occurrences) Last logged: 07:46:50

Update for cover.soma_MAC fails Update for cover.soma_MAC2 fails Update for cover.sovrum_v_0 fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 350, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 558, in async_device_update raise exc File "/usr/src/homeassistant/homeassistant/components/soma/cover.py", line 90, in async_update self.current_position = 100 - response["position"] TypeError: unsupported operand type(s) for -: 'int' and 'str'

ratsept commented 3 years ago

I think this here means that response["position"] was not successfully evaluated as a number. Likely because of the failed connection to the motor. I don't think this would be fatal or even necessarily bad as it is. But I'm not really a Python expert so I'm not sure.

The connection over BLE can fail at any time and this integration has to be able to handle that.

I've been talking to another developer working on an integration for a different system and they also have a similar position issue where the reported position is different in the SOMA app and through the API. And it seems that the reason could be that the data we provide in the API is read in a different way (Connect uses the data from the broadcast ADV packet while the app actually polls the target position characteristic value). So my plan now is to find out why the ADV data would not be updated while the polling data is. I will keep you updated here on my progress as well.

olaf7 commented 3 years ago

On several occasions I have seen reports from one API reporting position 40 and the other 60 (or 19 and 81). Their sum is always 100 but one would expect a match.

On the subject of the App:

please add MAC-address listing in the device setup. Makes testing, development and support easier compared to the need to call the REST API to find out the device MAC address.
please distinguish between firmware versions of Soma Connect and Soma devices
would love to be able to configure the Shades2 as vertical instead of horizontal blinds. And please add three modes in that case: left-to-right, right-to-left and left+right-to-middle closing/opening

On the subject of hardware:

you could consider removing the solarpanel(but keep it as an option) from the kit to reduce cost and maybe that makes live easier on obtaining parts as well. I now have three expensive solarpanels not being used.
have the three wheel types included in the Shades 2 product so you have only 1 version of the product. Most competition does package it this way.

ratsept commented 3 years ago

What do you mean by "...eports from one API reporting position 40 and the other 60"? Do you mean from HA vs the Connect API? Because that is exactly how it's supposed to work. HA has the open/close values opposite to what we use in the Connect. All the different integrations for window covers seem to have randomly chosen whether 100 means fully open or fully closed.

Showing MAC in setup - noted. I will see what the design/UX guys say about that and if possible we will add that.

Not sure what you mean by "...distinguish between firmware versions of Soma Connect and Soma devices". You can see the Connect version in the app and you can also see motor firmware (and hardware) version info as well. I think recently we have tried to match Connect versions with motor versions if at all possible.

We have been looking into supporting different window cover types for a while now. I hope we can release something soon. It will only be a cosmetic change in the app though - the motors and API will remain exactly the same.

The current solar panel is actually quite low cost to us so it's not a big issue. But we have talked about this before and I think we actually started out with the solar panel as extra when we first released the product. In a bigger hardware update coming next year (hopefully) the solar panel is likely to become much more complex and will definitely become an optional extra.

We do only have one version of the motor. Even the SOMA Tilt is built on top of that base. But we want to make the distinction somewhere so that support would be easier. We've seen too many people use the wrong kind of wheel for what their blinds and at best the wheel just breaks, at worst it actually breaks the blinds. We are also working on some updates to the wheels which would allow us to package all the different wheel types with every order. But that will create more useless waste just like with the extra solar panels you have left over now.

olaf7 commented 3 years ago

Yes. Exactly what I mean: the two different APIs (Soma and HA) sum up to 100. But not always! Sometime it is like for example 60=60 so they report the same number! Thanks for the clarification.

Some other issue which annoys me a lot is the change of MAC-address of the Soma Connect after reboot or power cuts. I connected the Soma Connect using ethernet (and probably also using WiFi as that is the default setup of the Soma Connect. This is not a bug/issue with the Soma Connect software itself, but is a 'feature' of the raspberry. See for instance: https://www.raspberrypi.org/forums/viewtopic.php?p=1560841 The Smart Shades App seems to cope with this. But the Home Assistant integration configuration likes an IP or resolvable hostname. Enforcing an IP in the DHCP-server/Modem does not work as the MAC-address changes. Please enforce a fixed MAC-address or allow for (easy) configuring the Soma Connect with a fixed IP.

To elaborate on MAC-addresses: I can not check this at the moment as all Soma device have up to date firmware, but iirc when checking the status of a device using the Smart Shades App, it is not (always?) the Firmware and MAC-address of the device that is reported, but more likely the details of the Soma Connect. Hence my request to distinguish between them and allow reporting of MAC-address and Firmware version of the currently selected device in the App instead of the Soma Connect MAC-address and/or firmware version.

On the subject of Open/Close and Up/Down: This indeed is a troublesome thing. Especially if multiple definitions are being used. Lets take for example the Soma Tilt on horizontal blinds. With standard shades both the 0% and 100% positions (manually maximized to left and right) limit the view and can be considered as being 'closed' while the 50% position would mean a maximum view, so non-technical users would consider this 'open'. So what would you consider 'close': setting the tilt in the far out minimum or maximum position? The difference could be the amount of sun reflection, but it especially matters when you have blinds with combined function like: https://www.ambiancewindowcoverings.com/standard-tilt-vs-hidden-tilt-shutters-omaha-blinds/ Or http://www.hunterdouglascomponents.com/en/our-products/products/venetian-blinds/ (see the VarioFlex shades which has a predecessor called DuoFlex, which is the ones I have a Soma Tilt connected to) This allows for: bottom closed/top open, bottom open/top closed, bottom open/top open, bottom closed/top closed. This can be tricky with manual control, but with the Tilt it does not make it easier especially as the App and software have no idea. A similar thing can be said about the Shades device. You can use it to tilt blinds opening up the discussion as above but also they can be used to move vertical shades along a rail. Is 0% open or closed and what about 100%? This can be something you can struggle with if you are not tech savvy. With manual control you just handle stuff without thinking and by automating it one starts thinking about it and it gets complicated. At least that is the case where I have to instruct someone how to use the App and integration. Another tricky thing here is when you have 2 Soma devices controlling the blinds: tilt + move up/down over a rail. The user wants to open the blinds. What is open here? I know this is not a Soma unique thing. It is just something I need to get a grip on so the end user can get along.

In the meantime I have update my repository again and the test script can now take an argument to specify the position so it can be fully scriptable for testing. Please let me know what I can do to further help the process of tackling this caching issue.

olaf7 commented 3 years ago

To further comment on the changing MAC-address: you can also considering adding mdns/bonjour support so somaconnect.local (or something similar) is available if not filtered from the network. Having all three options (fixed MAC, fixed IP, mdns) would be really nice and handy for integrations like this one. Now various issues/suggestions are within this thread/issue. Is a bugtracker for Soma-Connect an idea?

austwhite commented 3 years ago

Just subscribing to this issue to monitor. I have not had any issues with changing MAC address, but the tilt position issue is definitely still present. This does look like a SOMA Connect issue though and not Home Assistant as it does the same thing in Apple Home Kit, which is completely independent of HA.

olaf7 commented 3 years ago

As mentioned before this issue has become a bit messy, so it might be a good idea to summarize and show status updates.

Synchronization issue The original and main reported issue. Potentially involved are the components: Smart Shades App, Soma devices (firmware), Soma Connect (Soma server package managing BLE linked devices and providing the REST API), HA integration. Issue has been confirmed by other users and Tiit (ratsept)/Soma/Wazombi Labs. The issue is being addressed. It is not yet clear (to me) which (combination of) components are involved. It might be that various components need software updates. Status: open
Timeout issue To test things myself and to allow others to test I wrote some code and published it at: https://github.com/olaf7/poc-soma A major issue here is that the code is unreliable as timeouts can occur. This issue was discussed outside of this Github issue. It might be related to the sync issue. The error you might get is: requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.178.116', port=3000): Max retries exceeded with url: /list_devices (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa71be11588>: Failed to establish a new connection: [Errno 111] Connection refused')) Status: open
APT upgrade The soma-server package / The Soma Connect can not easily upgraded, but requires the removal of the SD-card and reflashing it. This is an inconvenience and not so much a related issue. This feature is not (yet?) supported. Status: low priority if any?
Polling frequency Currently the Soma HA integration polls the Soma Connect. How often it does this is not configurable. Status: feature request
Linking of two Soma devices operating on the same blinds linked/unlinked devices operating of the same blinds might have impact on (the timing of) requests/commands being sent to the Soma Connect and if it can handle those without timeouts. Status: too use case specific
Skallan reported a different issue Logs supplied. Might be (a related?) sync issue Status: open?
Stable MAC-address of Soma Connect / broadcasting the MAC-address The stability of the MAC-address seems to be no longer an issue now the SD-card is no longer corrupt(ed) Having reporting/monitoring would help. mDNS/Bonjour support still looks like a desirable feature. If this problem is resolved can only be confirmed over time. The feature request remains. Status: unknown
MAC-address reporting Reporting the MAC-address of a device in the Soma Shades App would be nice. Status: feature request
Support of various blind types in the Soma App Feature request to support various vertical and horizontal blinds. Status: feature request

I think this list sums up all discussed. If not: please add. As there is no bugtracker it got a bit messy as multiple issues are raised, including ones not related to Home Assistant. As I feel being responsible for this I felt the need to create this summary.

austwhite commented 3 years ago

Thanks for the summary.
I do feel it should be noted that the issue is present in other home automation hubs that Soma integrates with and not just Home Assistant. A review on homekitnews form June 2020 mentioned the issue also.

olaf7 commented 3 years ago

Thanks for the reference. Indeed the issue is mentioned here: https://homekitnews.com/2020/06/01/soma-smart-shades-2-with-soma-connect-review/ (section: POTENTIAL ISSUES TO BE AWARE OF) However I found another interesting thing on this site. In the recent review on https://homekitnews.com/2021/06/29/the-ultimate-smart-vertical-blinds/ in the video the reviewer stumbled upon the same issue I had: controlling a single blind with 2 motors can be tricky. This is obviously not unique to Soma. I wrote some manual code (not fully tested yet) to handle this. I will upload it to the above mentioned repository. It also made me aware that a couple of features could be useful: this ability to set and read the amount of time it takes to finish an action (currently debatable configurable in my code) and the ability to query a motor to check if it is performing an action or is idle.

AJolly commented 3 years ago

I'm definitely running into a lot of issues with soma connect becoming unavailable, even with latest images/ethernet connection.

ratsept commented 3 years ago

Are the issues with Connect itself becoming unavailable or is the issue with the connection to the motors? You can check where the issue is by trying to ping the Connect when it doesn't answer over the API. The API commands that want to talk to the motors can fail if the BLE connection fails. So to see if Connect itself is ok you can try the list_devices endpoint since that uses cached values.

AJolly commented 3 years ago

Seemed to be more on HA or the blinds end - Logger: homeassistant.components.sensor Source: components/soma/sensor.py:38 Integration: Sensor (documentation, issues) First occurred: 5:34:01 PM (2 occurrences) Last logged: 5:34:01 PM

Error adding entities for domain sensor with platform soma Error while setting up soma platform for sensor Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 383, in async_add_entities await asyncio.gather(*tasks) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 448, in _async_add_entity suggested_object_id = entity.name # type: ignore[unreachable] File "/usr/src/homeassistant/homeassistant/components/soma/sensor.py", line 38, in name return self.device["name"] + " battery level" TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

I've disabled the HA integration and am now just using direct http requests to the connect API, esp as I'm not really caring about the current blind status. This also lets me use the morning mode call.

I'd be nice if I could directly send the bluetooth commands and not need soma connect at all!.

AJolly commented 3 years ago

Ex: rest_command: move_soma: url: "http://192.168.40.16:3000/set_shade_position/{{ mac }}/{{ amount }}?morning_mode=1"

austwhite commented 3 years ago

I have to admit I have no issues with SOMA Connect becoming unavailable, however the polling function does not seem to work. IF the blinds are changed in the Smartblind app directly, the position is not updated in Home Assistant. This could be related to the position issue.

@AJolly This issue is specifically about not reading the position percentage. You may be best opening a new issue for the device becoming unavailable so it can be looked at separately.

austwhite commented 3 years ago

Have any of you updated the Firmware on your Soma Tilt / Soma Tilt 2 to 2.2.9 yet? I just did based on the change log and now the SOMA Tilt 2 (I have the 2 version) correctly reports the position of the blind when the STOP button is pressed. Changelog:

https://support.somasmarthome.com/hc/en-us/articles/360014397560-Firmware-Updates

Just thought I would mention it to see if anyone else might want to test it out.

@AJolly Might be worth checking this for your issue too. Make sure your Connect is on 2.2.8 and update the blinds using the Smartshades application. They update pretty easily as long as you stay close to the blind motor during the update.

ratsept commented 3 years ago

The new 2.2.9 really only had two small changes and this was one of those. Unfortunately we still have this weird behavior where we first have to set the position to 0 or 100 (depending on direction) and only snap to the correct position when stopping. This is mostly caused by the fact that we have to support many integrations that don't differentiate between the target position and the current position. Ideally there would always be two numbers - one showing where the motor wants to move to and another one to just show where it is. I'm not sure if even Home Assistant can do that to be honest. If we were to always return the current position then all the sliders in the UI would act weird. You would drag the slider to say 100% and then as you release it it would snap back to where the motor currently is and then slowly work its way to the target (since the slider is used both for target and position). If HA does have separate values for these then I think we could work something out for that. But most other automation system unfortunately only do them combined right now (since lightbulbs seem to be the main target market and they react pretty much instantly).

austwhite commented 3 years ago

@ratsept Although it is not ideal how it is, the primary issue of not reporting the position when the stop button was pressed seems solved in 2.2.9 firmware. I do find that the Soma Connect sometimes loses bluetooth connection to the blinds, so calls sometimes fail, but this is not related to this issue at all. The response time is okay for my purpose and I think will be for most as long as they don't go sending changes too quickly, which is not the way blinds are intended to work.
The way the sliders work with the current reporting seems to be okay from my experience.

ratsept commented 3 years ago

Momentary loss of communications is something that we still see and I'm not sure we can ever be 100% sure with BLE and wireless. This is something we constantly try to improve on the protocol level (retry failed requests and cache values read from the device). But really I think the correct way is to go with Zigbee or Matter/Thread in future versions. Good to hear that someone else has also gotten this to work. I've had trouble keeping HA updated to the latest version on my home server but even in the older versions this update still seems to have fixed it.

austwhite commented 3 years ago

@ratsept I am using the latest version 2021.9.6. I have tested this 2.2.9 firmware on both Home Assistant Blue (Odroid platform) and on Virtual Box (VM platform, using Home Assistant Supervised) and it works well.

I agree "Matter" would be the way to go moving forward to make the most of the BLE to seemlessly initiate connection and then Zigbee for stability. BLE alone will never be flawless, just have to compensate best you can with any acknowledgements you can get :)

austwhite commented 3 years ago

I've found lately, last 3 HA versions, that I get a lot of "Read time out" errors when setting the blind to a position that is not fully open or fully closed. IT appears the time out period is set to only 5 seconds

`Logger: homeassistant.components.websocket_api.http.connection
Source: components/soma/cover.py:55
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 7:37:59 PM (2 occurrences)
Last logged: 7:38:12 PM

[140558182283584] HTTPConnectionPool(host='192.168.0.219', port=3000): Read timed out. (read timeout=5)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1371, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='192.168.0.219', port=3000): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 185, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1491, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1526, in _execute_service
    await handler.job.target(service_call)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 213, in handle_service
    await self.hass.helpers.service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 658, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 830, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 695, in _handle_entity_call
    await result
  File "/usr/src/homeassistant/homeassistant/components/cover/__init__.py", line 309, in async_set_cover_position
    await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/soma/cover.py", line 55, in set_cover_position
    response = self.api.set_shade_position(
  File "/usr/local/lib/python3.9/site-packages/api/soma_api.py", line 26, in set_shade_position
    return requests.get(url = "http://"+self.soma_connect_ip+":"+str(self.soma_connect_port)+"/set_shade_position/"+mac+"/"+str(position), timeout = self.timeout).json()
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='192.168.0.219', port=3000): Read timed out. (read timeout=5)`

The last line shows the "Read timed out (read timeout=5)

Should this timeout be set to a longer time as the blinds do take longer than 5 seconds to reach their set position, or is something else causing this timeout issue.

The blinds will set themself to the correct position, so the command reaches the blind. The issue seems to be with the acknowledgement reaching Home Assistant.

I am raising this as it is not just an occasional error. It happens every time the blind is set to a partial open position. If set to fully open or fully closed does not result in the error.

austwhite commented 3 years ago

@ratsept Is the above error I mentioned something on the SOMA Connect side or is it a HA error?

olaf7 commented 3 years ago

I too will give an update: currently I run the latest firmware of Connect, Tilt and Shades2. Also I run the latest stable docker HA build for Pi3. The integration does not give the right position.With the tilt the blinds physically at 50% (so fully open) yet the integration greys out one arrow. The SmartShades App runs fine at the moment, the HA integration seems way to unreliable to be useful, but that could be a version firmware/HA-build/version/Integration version mismatch. Maybe something to add to the documentation: a table with recommended versions?

austwhite commented 3 years ago

@olaf7 Are you using the 2.2.8 Connect software and 2.2.9 firmware on the blind motors? I find it does update the percentage and will have both arrows available. I just constantly get the error pop up and the time out. The update isnt always instant though. Sometimes it takes a minute to update percentage, like it is polling rather than being a push update. Also note, Home Assistant will only go from fully closed to full open. It won't go from closed down to closed up. It will only go one way. IE Closed down to open OR closed up to open. It is random as to what it decides is closed most times. Fully open is 100% and closed is 0%

That said, there is a major problem with these blind motors that Soma needs to fix. They can only connect to one Bluetooth device at a time. If the blinds are connected to the Smart blinds app, then they will not connect to Soma Connect and if they have been connected to Soma Connect it takes forever for them to disconnect from the Connect and connect to the app. Honestly, it makes them pretty well useless for use in automation. You have to chose, use the app or use connect but they cannot connect to two devices at once. I tested this and it is repeatable. If I have one blind connected to the Smart blinds App on the phone, that blind will show offline in Homekit or Home Assistant until I completely close the app

olaf7 commented 3 years ago

@austwhite Soma Connect app reports all motors (tilt, shades2 devices) firmware 2.2.9. Soma Connect firmware 2.8. HA version: Home Assistant 2021.10.3 (docker stable on Raspi 3b with Raspbian stable (Buster)) Soma Connect reports disconnected which is strange as it is UTP connected, (so this is reporting on WiFi only). Yet the Connect requires 10 cm distance to talk to the SmartHome App so it must be BT only. The SmartHome App and the HA App indeed can block each other due to single BT connection capabilities. So I confirm your findings. I would be even fine with this if the HA integration would work reliably. It does not. Far from. Regardless of timing physical position, position according to the HA integration and position according to the SmartHome App can vary and can easily be very different. I fully agree there is lots of room for improvement. Currently I hope for the best and hope the system is usable for the disabled person I installed it for. Not a situation I was aiming for when I started the project of home automation. Feel free to use my stand alone testing code from my github repo to play around with timeout settings and the Soma REST API used by the integration. It might give a better insight on what is going on.

austwhite commented 3 years ago

I use HA is in a virtual box VM. Maybe it is different in docker with communication to connect 🙂

olaf7 commented 3 years ago

I use HA is in a virtual box VM. Maybe it is different in docker with communication to connect slightly_smiling_face

Nope. The comms I describe are from Android Phone to Soma Connect and Android Phone to Soma devices. These all seem BT related. The only role the Soma Connect and wifi/UTP connections are playing a role in, seems to me, are the comms between HA and the devices by providing an API for the integrations. Which makes the API expensive; especially as that does not work as desired. (The Pi/Connect itself is not expensive, but you pay for functionality that does not deliver)

austwhite commented 3 years ago

Might be like I described before. It doesn't connect properly to two Bluetooth devices at the same time. I find it takes forever for the Smart blind app to connect if the connect is running and then it takes the blind open in the app offline in homekit and home assistant

austwhite commented 3 years ago

Well. I seem to have finally solved my issues with Soma Connect. As crazy as it seems, touch wood everything is perfect now and the blinds respond instantly. I turned off WiFi on the connect / Raspberry Pi 3 and connected it via a wired connection. No more time outs, no more errors and no delays at all. I click the percentage I want, the blind moves instantly where as on WiFi there was a significant delay of up to 5 seconds. I am thinking this is a limitation of having WiFi and BlueTooth so close together on the Pi, but am not sure. I will have to investigate further, but my solution was use wired ethernet and now the bluetooth is reliable

@ratsept If you are still monitoring this, maybe you could look at the way Soma Connect accesses WiFi and see if this is something that can be optimised.

Also, why does the Soma Connect send so much data out across the internet??? I have monitored it and it can by over 40MB a day transmitted. Even my Xiaomi stuff doesn't send that much data to the internet. Not really impressed with that

ratsept commented 3 years ago

WiFi can have an effect on BLE but it should not be that dramatic. WiFi and BLE both share the same 2.4 GHz band so they will always have collisions. There are some forms of coexistence algorithms but they will still only help when the band is not too crowded. I will let the guys working on Connect know about this. I think we have a new release coming out soon that should have some fixes for the BLE connection as well but I'm not sure what these are exactly. But we do test for WiFi and BLE before every release and from the logs it seems that that is actually the way most people use Connect. Your issue might have something to do with other devices you have interacting in weird ways with Connect. We've had weird WiFi issues with mesh WiFi routers before for example.

If you are using Connect with HA and don't need our cloud integrations it is safe to just block all traffic coming from the Connect. All that data is logging and debugging plus a tiny amount of MQTT housekeeping for cloud integrations (IFTTT, Alexa, Google Home, Smartthings). We plan on making the logging optional and when we do that I think we can also make the MQTT connection optional for people that only need the local integration. If the logs are disabled it will make it a little bit harder for us to debug issues in support but we will probably just make logging enabled by default and give expert users the option to disable it. It looks like most users prefer the easier support and don't care much about the traffic we generate.

I see there's also discussions here about multiple BLE connections. A peripheral device can't really have more than one active connection to a central role device at any one time. This is because of the way the BLE protocol works. In BLE all timing is controlled by the device in central role. Once a central devices creates a connection it will set up timeslots where the peripheral device has to be available for communications. If too many of these packets are missed by the peripheral device the connection is dropped. Now if a peripheral device had more than one connection active there would be no way to make sure there were no timeslot collisions. I have actually tried this and it is technically possible to make two connections to one motor like that but it will be extremely unreliable in some cases. And it also makes the software design for all parties involved much more complicated. Right now we assume many things while the connection is active. We assume that most values will only change when written to by the app and we never read them back if they are already cached. Every packet the motor has to send (either in an active BLE connection or during advertising) costs energy and to make these devices last as long as they do on battery there are a lot of optimization that come with their own tradeoffs. Ideally we would actually make the app talk to the motors through a Connect when one is available. We just haven't had the time to implement that as this doesn't seem to be a problem for most users. I think most phones drop the BLE connections to the motors once the app loses focus or the phone screen turns off. So the periods where Connect would have no communications should be minimal.

austwhite commented 3 years ago

@ratsept Is there a way to make the App talk WiFi to the Connect so the SmartBlinds App can just talk to the connect when a connect is present. There is already a way to configure a connect in the App so it could be done as a check box to say "talk via WiFi to the Connect" instead of talking direct to the blinds via bluetooth? That may help solve the assuming things when using a connect. OBviously the "WiFi" would only be an option if a Connect was present. As far as WiFI delays, I am just running a single access point. I have tried to different access points. I use a Raspberry Pi 3B+ with dual band, and the WiFi is better in 5GHz, but still has some delay that is not present when wired. Not sure if this will show in logs. I often got the timeout error in Home Assistant due to no response within the 5 second timeout period, even if the blinds did respond. I do not get that error at all wired. This could be the Home Assistant integration maybe? The blinds are all on latest 2.29 firmware and the Connect is on the current 2.28 release. When I see the new release I will try it out again on WiFi and see if there are improvements :).
One thing I like about SOMA over other companies is how responsive you are to the community, especially open source like Home Assistant.

ratsept commented 3 years ago

At the moment there is no way to make the app use WiFi instead of BLE. The configuration is done over BLE as well at the moment (in a really ugly hacky way but that's life - at least for now). We do plan on doing something like that as soon as possible.

I will refer this issue to the guys dealing with Connect. That delay seems unreasonably long especially if the delay is not present when using a wired connection. I can't promise the new version coming out soon will have any fixes for this but at least it will be on the board for a future release. For us to check the logs you would have to send me your Connect ID. I don't think you want to publish that here so just contact support and tell then you had this discussion here and maybe include a link to this issue as well. We have an awesome team working support and they will check the logs for anything weird that may be causing this.

I try to be responsive on all the different places our customers happen to visit but it is getting harder and harder. There are so many users now that actually finding the time to go over all the different forums about our products is becoming a huge task. But I feel like I have to be available here. We started the company using all these open source tools and stuff and I would feel like a cheat if I didn't at least try to give back. So I will keep coming back here for as long as I can (I probably have to keep this a secret from the marketing guys - can't admit our product has issues and stuff...).

olaf7 commented 3 years ago

Just installed the app on a brand new Chromebook. The app on the Chromebook has various issues: trouble connecting to the SomaConnecf, very frequent crashes after a long timeout trying to connect to a device. Also it seems to sometimes lose the permission to use Bluetooth??? It also frequently drops the connection and has to reconnect to devices. Bit off topic, yet Aust mentioned timeout isues so I felt I needed to give feedback here. Hope this helps.

home-assistant / core