hultenvp / solis-sensor

HomeAssistant integration for the SolisCloud PV Monitoring portal via SolisCloud API
Apache License 2.0
191 stars 42 forks source link

Report SolisCloud connectivity issues here #232

Open hultenvp opened 1 year ago

hultenvp commented 1 year ago

Describe the bug People report various issues with SolisCloud connectivity. Varying from not being able to configure the integration to connection errors with server.

A lot of fixes and hardening have already been introduced, but issues may remain.

Please add your issues in the comments of this issue. Before you do, make sure:

Note The "abnormal data bug" is not fixed, see #229

joncojonathan commented 1 year ago

Hi @hultenvp, many thanks for this add-on. Sadly it's not working for me. I've only installed this today, so this is a fresh set up. Home Assistant has been rebooted since installation. I only have one inverter in my SolisCloud account, and SolisCloud shows data.

I've got my Soliscloud login email address, secret, key ID and inverter ID and pasted them into the setup. I have also tried to extend the timeout in /config/custom_components/solis/soliscloud_api.py from 10 to 100 (line 468) but to no effect.

Any ideas based on the info below please? I'm unclear on how to use the test script, but if you can provide additional guidance I can give it a go.

Steps to reproduce

Versions

Logs

I reviewed the logs and note that sometimes the source is line 238, other times 242, of custom_components/solis/soliscloud_api.py.

Line 238: for record in result_json['data']['page']['records']: Line 242: elif result[STATUS_CODE] == 408:

Note that I'm not receiving any time / clock related warnings.

Logger: aiohttp.server
Source: custom_components/solis/soliscloud_api.py:242
Integration: Solis Inverter ([documentation](https://github.com/hultenvp/solis-sensor/), [issues](https://github.com/hultenvp/solis-sensor/issues))
First occurred: 16:26:10 (1 occurrences)
Last logged: 16:26:10

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 94, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 81, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 136, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 180, in post
    return await super().post(request, flow_id)
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 73, in wrapper
    result = await method(view, request, data, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 110, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 281, in async_configure
    result = await self._async_handle_step(
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 368, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
  File "/config/custom_components/solis/config_flow.py", line 118, in async_step_credentials_secret
    if await api.login(async_get_clientsession(self.hass)):
  File "/config/custom_components/solis/soliscloud_api.py", line 202, in login
    self._inverter_list = await self.fetch_inverter_list(self.config.plant_id)
  File "/config/custom_components/solis/soliscloud_api.py", line 242, in fetch_inverter_list
    elif result[STATUS_CODE] == 408:
KeyError: 'StatusCode'
Moondevil-ha commented 1 year ago

Since updating to latest version am having drop outs, failing on fetching the Inverter list. Was working fine until 3.2. Sample below:

Logger: homeassistant Source: custom_components/solis/soliscloud_api.py:242 Integration: Solis Inverter (documentation, issues) First occurred: 09:33:45 (1 occurrences) Last logged: 09:33:45

Error doing job: Task exception was never retrieved Traceback (most recent call last): File "/config/custom_components/solis/service.py", line 103, in async_discover capabilities = await self._do_discover() File "/config/custom_components/solis/service.py", line 120, in _do_discover if await self._login(): File "/config/custom_components/solis/service.py", line 92, in _login if await self._api.login(async_get_clientsession(self._hass)): File "/config/custom_components/solis/soliscloud_api.py", line 202, in login self._inverter_list = await self.fetch_inverter_list(self.config.plant_id) File "/config/custom_components/solis/soliscloud_api.py", line 242, in fetch_inverter_list elif result[STATUS_CODE] == 408: KeyError: 'StatusCode'

grlarsen commented 1 year ago

Hi @hultenvp Thank you for all your hard work making the Solis inverter sensor available for everyone. I have the exact same issue as described above by @joncojonathan. I have confirmed that the datalogger is sending data and the inverter is online. Solis cloud web service is providing real time data.

Trying to add integration through UI results in: image

I have also run your test script and resulting string. Regardless of witch line pair I uncomment, script returns: {"code": "1", "data": null, "msg": "\u6570\u636e\u5f02\u5e38 \u8bf7\u8054\u7cfb\u7ba1\u7406\u5458", "success": true}

HA log from attempt at adding integration is as follows: image

hultenvp commented 1 year ago

Since updating to latest version am having drop outs, failing on fetching the Inverter list. Was working fine until 3.2. Sample below:

Logger: homeassistant Source: custom_components/solis/soliscloud_api.py:242 Integration: Solis Inverter (documentation, issues) First occurred: 09:33:45 (1 occurrences) Last logged: 09:33:45

Error doing job: Task exception was never retrieved Traceback (most recent call last): File "/config/custom_components/solis/service.py", line 103, in async_discover capabilities = await self._do_discover() File "/config/custom_components/solis/service.py", line 120, in _do_discover if await self._login(): File "/config/custom_components/solis/service.py", line 92, in _login if await self._api.login(async_get_clientsession(self._hass)): File "/config/custom_components/solis/soliscloud_api.py", line 202, in login self._inverter_list = await self.fetch_inverter_list(self.config.plant_id) File "/config/custom_components/solis/soliscloud_api.py", line 242, in fetch_inverter_list elif result[STATUS_CODE] == 408: KeyError: 'StatusCode'

Ouch! That's a silly one from my side. Missed one path where StatusCode is not set. Fix underway.

hultenvp commented 1 year ago

I've fixed both crashes in rel 3.2.1

hultenvp commented 1 year ago

\u6570\u636e\u5f02\u5e38 \u8bf7\u8054\u7cfb\u7ba1\u7406\u5458

Hey @grlarsen , I fixed the crash, but it won;t help you much as the dreaded Abnormal data error will likely still haunt you. I'm trying to reproduce it, but it's a very elusive bug.

luftdieb commented 1 year ago

Hi, with newly provided API login data from Solis, login is not possible. It just report "Cannot login with provided URL and credentials" and login data disapear. No further errormessage. In Logbook you can find this additional info `Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: custom_components.solis.soliscloud_api Source: custom_components/solis/soliscloud_api.py:204 Integration: Solis Inverter (documentation, issues) First occurred: 14:12:40 (3 occurrences) Last logged: 19:45:46

No inverters found `

grafik

I do have a Solis inverter and therefore I've changed URL to https://www.soliscloud.com:13333 The 19 digit's station ID I've fetched from URL as described.

Config

` default_config:

frontend: themes: !include_dir_merge_named themes

tts:

my:

automation: !include automations.yaml script: !include scripts.yaml scene: !include scenes.yaml

homematic: interfaces: rf: host: 192.168.1.11 resolvenames: "json" username: "" password: "" ip: host: 192.168.1.11 port: groups: host: 192.168.1.11 port: resolvenames: "json" username: "" password: "" path: /groups hosts: ccu2: host: 192.168.1.11 port: username: "" password: "" `
logger: default: critical logs: homeassistant.core: fatal homeassistant.components.mqtt: warning homeassistant.components.python_script: warning custom_components.solis: debug homeassistant.components.smartthings.light: info custom_components.my_integration: debug

Versions HA Version : 9.4 (2022.12.8) HACS version : 20220906112053 (1.29.0) Integration version : 3.2.1 HW: Fujitsu S740 with Proxmox

Debug logs 2022-12-21 23:12:26.480 DEBUG (MainThread) [custom_components.solis.soliscloud_api] workarounds: {'correct_daily_on_grid_energy_enabled': False} 2022-12-21 23:12:27.053 DEBUG (MainThread) [custom_components.solis.soliscloud_api] Response contains unexpected data: {'success': True, 'code': '1', 'msg': '数据异常 请联系管理员', 'data': None} 2022-12-21 23:12:27.053 WARNING (MainThread) [custom_components.solis.soliscloud_api] No inverters found



![grafik](https://user-images.githubusercontent.com/117862420/208981911-5eb9d170-7166-4a29-89c5-5bc1422e7d60.png)
spiderpigsam commented 1 year ago

Move to correct Bug reporting location - apologies

HI, I am also having the same issue followed documentation to the letter, tried uninstall integration and re-install.

This error originated from a custom integration.

Logger: custom_components.solis.soliscloud_api Source: custom_components/solis/soliscloud_api.py:204 Integration: Solis Inverter (documentation, issues) First occurred: 08:15:26 (1 occurrences) Last logged: 08:15:26

No inverters found

Home Assistant 2022.12.7 Supervisor 2022.11.2 Operating System 9.4 Frontend 20221213.1 - latest Ginlong Solis PV portal integration v3.2.1

Here is the debug log extract.

`2022-12-21 10:33:40.351 DEBUG (MainThread) [custom_components.solis.soliscloud_api] Response contains unexpected data: {'success': True, 'code': '1', 'msg': '数�异常 请�系管�员', 'data': None}

2022-12-21 10:33:40.351 WARNING (MainThread) [custom_components.solis.soliscloud_api] No inverters found`

spiderpigsam commented 1 year ago

Tried new update still same issue with no inverter found and error code 1 returned will copy logs in for reference.

alienatedsec commented 1 year ago

v3.2.2

2022-12-23 17:07:15.697 INFO (MainThread) [custom_components.solis.soliscloud_api] /v1/api/inverterDetail responded with error: 1:数据异常 请联系管理员 <<< uncle Google says 'the data is abnormal, please contact administrator'
2022-12-23 17:07:15.697 WARNING (MainThread) [custom_components.solis.soliscloud_api] No inverters found
alienatedsec commented 1 year ago

Enabling debug does not bring more information:

2022-12-23 17:54:52.493 DEBUG (MainThread) [custom_components.solis.soliscloud_api] workarounds: {'correct_daily_on_grid_energy_enabled': False}
2022-12-23 17:54:52.724 INFO (MainThread) [custom_components.solis.soliscloud_api] /v1/api/inverterDetail responded with error: 1:数据异常 请联系管理员
2022-12-23 17:54:52.724 WARNING (MainThread) [custom_components.solis.soliscloud_api] No inverters found
luftdieb commented 1 year ago

Hi, I've also tried newest version 3.2.2 but still no success. So I've just added some debug information in file soliscloud_api.py --> async def _post_data_json() async def _post_data_json(self, canonicalized_resource: str, params: dict[str, Any]) -> dict[str, Any]: """ Http-post data to specified domain/canonicalized_resource. """

    header: dict[str, str] = self._prepare_header(params, canonicalized_resource)
    result: dict[str, Any] = {SUCCESS: False, MESSAGE: None, STATUS_CODE: None}
    resp = None
    _LOGGER.debug("selfConfigDomain")
    _LOGGER.debug(self.config.domain)
    _LOGGER.debug("params")
    _LOGGER.debug(params)
    if self._session is None:
        return result
    try:
        async with async_timeout.timeout(20):
            url = f"{self.config.domain}{canonicalized_resource}"
            resp = await self._session.post(url, json=params, headers=header)
            result[STATUS_CODE] = resp.status
            result[CONTENT] = await resp.json()
            if resp.status == HTTPStatus.OK:
                result[SUCCESS] = True
                result[MESSAGE] = "OK"
            else:
                result[MESSAGE] = "Got http statuscode: %d" % (resp.status)
    except (asyncio.TimeoutError, ClientError) as err:
        result[MESSAGE] = f"{repr(err)}"
        _LOGGER.debug("Error from URI (%s) : %s", `canonicalized_resource,` result[MESSAGE])
    finally:
        if resp is not None:
            await resp.release()
        _LOGGER.debug("finally_Post_Data_Json")
        _LOGGER.debug(result)
        return result

This will create this kind of logentries: 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] selfConfigDomain 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] https://www.soliscloud.com:13333 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] params 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] {'stationId': '19DigitNumber'} 2022-12-23 20:47:10.905 DEBUG (MainThread) [custom_components.solis.soliscloud_api] workarounds: {'correct_daily_on_grid_energy_enabled': False} 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] selfConfigDomain 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] https://www.soliscloud.com:13333 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] params 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] {'stationId': '19DigitNumber'} 2022-12-23 20:47:17.395 DEBUG (MainThread) [custom_components.solis.soliscloud_api] finally_Post_Data_Json 2022-12-23 20:47:17.395 DEBUG (MainThread) [custom_components.solis.soliscloud_api] {'Success': True, 'Message': 'OK', 'StatusCode': 200, 'Content': {'success': True, 'code': '1', 'msg': '数据异常 请联系管理员', 'data': None}}

Maybe that helps ? I don't have too much experience in python... Have worked more with C#. If you can give me a pointer, where to add more logfile output, just let me know. Looks like the code 1 is not as expected.

luftdieb commented 1 year ago

I can simply create the same problem by using postman grafik So i guess something is wrong with login mechanism, I've tried examples from "SolisCloud Monitoring API.pdf" which you can see in the picture. But same problem. Maybe something is wrong with signature calculation ? In manual there is encryption HmacSHA1 mentioned. grafik In soliscloud_api.py this command do the encryption "sign = base64.b64encode(hmac_obj.digest())" Is this the same ? FYI: I've add more debug information in logfile to get calculated SIGN from soliscloud_api.py , which I've used in postman with my ID ;-)

hultenvp commented 1 year ago

v3.2.2

2022-12-23 17:07:15.697 INFO (MainThread) [custom_components.solis.soliscloud_api] /v1/api/inverterDetail responded with error: 1:数据异常 请联系管理员 <<< uncle Google says 'the data is abnormal, please contact administrator'
2022-12-23 17:07:15.697 WARNING (MainThread) [custom_components.solis.soliscloud_api] No inverters found

Aww shoot, I worked around the issue for stationDetail, but it looks like the inverterDetail also causes issues with some people.

I'm still at loss what causes these errors, I'm 95% sure it's on Solis side.

hultenvp commented 1 year ago

Hi, I've also tried newest version 3.2.2 but still no success. So I've just added some debug information in file soliscloud_api.py --> async def _post_data_json() async def _post_data_json(self, canonicalized_resource: str, params: dict[str, Any]) -> dict[str, Any]: """ Http-post data to specified domain/canonicalized_resource. """

    header: dict[str, str] = self._prepare_header(params, canonicalized_resource)
    result: dict[str, Any] = {SUCCESS: False, MESSAGE: None, STATUS_CODE: None}
    resp = None
    _LOGGER.debug("selfConfigDomain")
    _LOGGER.debug(self.config.domain)
    _LOGGER.debug("params")
    _LOGGER.debug(params)
    if self._session is None:
        return result
    try:
        async with async_timeout.timeout(20):
            url = f"{self.config.domain}{canonicalized_resource}"
            resp = await self._session.post(url, json=params, headers=header)
            result[STATUS_CODE] = resp.status
            result[CONTENT] = await resp.json()
            if resp.status == HTTPStatus.OK:
                result[SUCCESS] = True
                result[MESSAGE] = "OK"
            else:
                result[MESSAGE] = "Got http statuscode: %d" % (resp.status)
    except (asyncio.TimeoutError, ClientError) as err:
        result[MESSAGE] = f"{repr(err)}"
        _LOGGER.debug("Error from URI (%s) : %s", `canonicalized_resource,` result[MESSAGE])
    finally:
        if resp is not None:
            await resp.release()
        _LOGGER.debug("finally_Post_Data_Json")
        _LOGGER.debug(result)
        return result

This will create this kind of logentries: 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] selfConfigDomain 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] https://www.soliscloud.com:13333 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] params 2022-12-23 20:47:10.896 DEBUG (MainThread) [custom_components.solis.soliscloud_api] {'stationId': '19DigitNumber'} 2022-12-23 20:47:10.905 DEBUG (MainThread) [custom_components.solis.soliscloud_api] workarounds: {'correct_daily_on_grid_energy_enabled': False} 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] selfConfigDomain 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] https://www.soliscloud.com:13333 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] params 2022-12-23 20:47:10.907 DEBUG (MainThread) [custom_components.solis.soliscloud_api] {'stationId': '19DigitNumber'} 2022-12-23 20:47:17.395 DEBUG (MainThread) [custom_components.solis.soliscloud_api] finally_Post_Data_Json 2022-12-23 20:47:17.395 DEBUG (MainThread) [custom_components.solis.soliscloud_api] {'Success': True, 'Message': 'OK', 'StatusCode': 200, 'Content': {'success': True, 'code': '1', 'msg': '数据异常 请联系管理员', 'data': None}}

Maybe that helps ? I don't have too much experience in python... Have worked more with C#. If you can give me a pointer, where to add more logfile output, just let me know. Looks like the code 1 is not as expected.

Thanks @luftdieb: It confirms what we see. For some users the API just refuses to respond with data. Now, if I read correctly, also userStationList rejects the request for you.

I'm still trying to find the root cause, and I'm still chasing 1 lead, maybe you or anyone reading along can help: 2 people have given me their credentials to test and what I see is that in all calls that fail either their sign or their MD5 has a slash ('/') in the b64 encoded string. I'm not that well informed about HTTP headers, but it might be that that confuses the server. Maybe someone knows if that's an issue or not and if it is how to escape it.

hultenvp commented 1 year ago

Maybe something is wrong with signature calculation ? In manual there is encryption HmacSHA1 mentioned.

The code does a HmacSha1. Also it works fine for a lot of people, a significant subset does have issues however.

JJ-Joris commented 1 year ago

v3.2.2

2022-12-23 17:07:15.697 INFO (MainThread) [custom_components.solis.soliscloud_api] /v1/api/inverterDetail responded with error: 1:数据异常 请联系管理员 <<< uncle Google says 'the data is abnormal, please contact administrator'
2022-12-23 17:07:15.697 WARNING (MainThread) [custom_components.solis.soliscloud_api] No inverters found

Aww shoot, I worked around the issue for stationDetail, but it looks like the inverterDetail also causes issues with some people.

I'm still at loss what causes these errors, I'm 95% sure it's on Solis side.

I have similar issues. Getting a 200 response but the data is empty. The msg in the response translates to contact the provider. I have contacted Solis about this, when they get back to me I will post it here

jmason commented 1 year ago

what I see is that in all calls that fail either their sign or their MD5 has a slash ('/') in the b64 encoded string. I'm not that well informed about HTTP headers, but it might be that that confuses the server.

Aha, that rings a bell. I bet it needs to be encoded using URL encoding (use %2F instead of the '/' char). I've seen that with other API services.

jmason commented 1 year ago

or it could be that this would be the right type of base64 to use: https://docs.python.org/3/library/base64.html#base64.urlsafe_b64encode

luftdieb commented 1 year ago

I've tried it out quickly by replacing both base64.b64encode() to base64.urlsafe_b64encode(). But still same problem.

I'm trying to build a small python script, which just build the Content-MD5 Encryption and signature string to try it out by postman. Maybe that give us more hints...

luftdieb commented 1 year ago

Maybe I'm totally wrong. But I've create a small C# application, which create the "Content-MD5" header and even the Authorization header key and make a post to the solis URL with the effect, getting the same errormessage. But interessting is, I've got an System.FormatExecption while I've added the 4 headers (Content-MD5, Content-Type, Date, Authorization) which lead me to this website https://stackoverflow.com/questions/61554970/how-to-set-content-md5-header-in-get-method-using-httpclient Here it is described, all members starting with "Content-" will be removed and have to be added in a different way.

So I've tried this also in postman, without add Content-MD5, Content-Type to my POST request and I'm getting the same returncode, compared to youre python script: image

Therefore I'm asking the question, is the format how we define the header the right way ? But it looks exactly, how it is described in solis API manual image

hultenvp commented 1 year ago

Nice discussion and thanks for the investigation! Good (or bad..) to see confirmed that with a C# implementation t fails in the same way.

@luftdieb : It could very well be that there is an issue in the header spec, but why does it work for one set of people and not for the others? Also because the server responds with "Malformed data", that suggests the issue is on the server side?

Yesterday I found a case where the "Malformed data" also occurred with a header that did not contain any slashes, so it seems that is not the base 64 encoding causing the issue.

To summarize:

Other than finding more confirmation for the above I do not see many leads still to follow. As said I'm getting more and more convinced this is a server side issue and it looks like the only thing we can do os make a strong case for Solis to investigate by giving them an easy reproduction scenario.

Does anyone have more ideas?

hultenvp commented 1 year ago

Maybe I'm totally wrong. But I've create a small C# application, which create the "Content-MD5" header and even the Authorization header key and make a post to the solis URL with the effect, getting the same errormessage. But interessting is, I've got an System.FormatExecption while I've added the 4 headers (Content-MD5, Content-Type, Date, Authorization) which lead me to this website https://stackoverflow.com/questions/61554970/how-to-set-content-md5-header-in-get-method-using-httpclient Here it is described, all members starting with "Content-" will be removed and have to be added in a different way.

So I've tried this also in postman, without add Content-MD5, Content-Type to my POST request and I'm getting the same returncode, compared to youre python script: image

Therefore I'm asking the question, is the format how we define the header the right way ? But it looks exactly, how it is described in solis API manual image

Re-reading your post. This is indeed interesting. So you ay that if you leave out the Content-MD5 and Content-Type you get exactly the same error message as in Python and if you add them that you get a "system format exception".

So this tells me that we should focus on the MD5. I still think the python code does not remove the content-* fields from the header, because part of the users does get the right output, but maybe in some cases the MD5 gets messed up. Either in Base 64 or the MD5 itself.

luftdieb commented 1 year ago

Currently, I'm just getting 408. I don't know, if they do some maintenance or if they block my IP because of too many invalid requests ...

hultenvp commented 1 year ago

Currently, I'm just getting 408. I don't know, if they do some maintenance or if they block my IP because of too many invalid requests ...

408 means your system time deviates more than 15 mins from server time. Probably you're running in a VM.

luftdieb commented 1 year ago

Problem with 408 was caused by opened connection of my C# code in the background.

I've tried to change the header to content part. But problem still remain //request.Content.Headers.Add("Content-MD5", contentMd5); --> This cause formerly reported system format exception request.Content.Headers.ContentMD5 = MD5.HashData(Encoding.UTF8.GetBytes(body)); /this cause no execption and header is included in request. but still same output

I will try to open a ticket at support of soliscloud.

devinmitchell commented 1 year ago

fwiw I opened support ticket #117702 the other day regarding this issue and this is their response so far (ticket is "Awaiting other's Support")

Shahbaz Khan said 2 days ago

Hello Solis,

We will check and update you.

Ticket no# 117702
[https://solis-service.solisinverters.com/helpdesk/tickets/117702 ](https://solis-service.solisinverters.com/helpdesk/tickets/117702%C2%A0)

Thank you for choosing Ginlong Solis!

Sincerely,
Shahbaz Khan

Ginlong Solis UK Service Team
hultenvp commented 1 year ago

Last few days I've been working at cleaning up the soliscloud API. Still al lot of work to do, but oddly I just noticed the new API seems to work with key/secrets that failed before!

I didn't have much time yet to investigate. I do get the ID's from the responses, instead of coded. For the rest there are not many changes in the header encoding.

If you want to test yourself: I've dropped the current version here: https://github.com/hultenvp/solis-sensor/tree/master/test

Make sure you have both the soliscloud_api directory and its contents and the apitest_async.py file. Add your key and secret and you're good to go. If you call with -v option then the test app will also output the JSON from the responses.

luftdieb commented 1 year ago

I've tested it... But unfortunatly, it will give the same result D:\test\solis-sensor\test>python soliscloud_test.py -v

{"code": "1", "data": null, "msg": "\u6570\u636e\u5f02\u5e38 \u8bf7\u8054\u7cfb\u7ba1\u7406\u5458", "success": true}

\u6570\u636e\u5f02\u5e38 \u8bf7\u8054\u7cfb\u7ba1\u7406\u5458 --> 数据异常 请联系管理员 --> "The data is abnormal, please contact the administrator"

There was also a typo error in line 33. But same result after fixing it and for sure I've updated line 34 and 11+12 CanonicalizedResource = "/v1/api/inveterDetail" --> CanonicalizedResource = "/v1/api/inverterDetail"

But interessting is, the output above was the same with and without typo error. Looks like, the API endpoint wasn't used and the error is show in general.

hultenvp commented 1 year ago

Hi @luftdieb:

There was also a typo error in line 33

They fixed the endpoint typo in their v1.2. before that you had to call the typo one. Now both work.

I've tested it... But unfortunatly, it will give the same result D:\test\solis-sensor\test>python soliscloud_test.py -v

That's to be expected if you still execute the original test app :-). Use apitest_async.py. interested to learn what you see. Best to first run without -v, gives you better readable output.

luftdieb commented 1 year ago

@hultenvp : Sorry! thanks for pointing me in the right direction ;-) But also both commands give same result: D:\test\solis-sensor\test>python apitest_async.py Exceptions: [ApiError('数据异常 请联系管理员')]

D:\test\solis-sensor\test>python apitest_async.py -v Verbose Exceptions: [ApiError('数据异常 请联系管理员')]

But Verbose isn't report anything because it is using existing soliscloud_api.py, which abort the script at line 637 because of errorcode 1 image

If you like we can do a remote session together by using teamviewer etc ?

grlarsen commented 1 year ago

I get the same result with new test program: ...\Python\Scripts\apitest_async.py" Exceptions: [ApiError('数据异常 请联系管理员')]

Have also raised a service ticket with Solis. Service team has asked for, and rreceived my Secret and key ID to investigate the issue at their side. I will post an update here once I hear something from Solis.

luftdieb commented 1 year ago

Looks like they are working on the interface. Now result looks different: D:\test\solis-sensor\test>python soliscloud_test.py -v

{"error": "Forbidden", "message": "wrong sign \u6570\u5b57\u7b7e\u540d\u4e0d\u6b63\u786e", "path": "/v1/api/inverterDetail", "status": 403, "timestamp": 1672305315524}

D:\test\solis-sensor\test>python apitest_async.py -v Verbose Exceptions: [HttpError('Http status code: 403')]

Postman result: { "timestamp": 1672305630780, "status": 403, "error": "Forbidden", "message": "wrong sign 数字签名不正确", "path": "/v1/api/inverterList" }

I'm also in contact with Solis support. They wrote me back 4h ago "I should try it again" ;-) Looks like we are getting progress here :-)

grlarsen commented 1 year ago

Solis has done something!

Ran testscript again with success! Ran integration with success!

EDIT: Enteties are reading and states are provided! Hurrah!

luftdieb commented 1 year ago

@grlarsen: Have you done some adjustment or it is running straight ahead ? Because all access methods will cause the same result on my side. Also the original HA implementation give me this output: 2022-12-29 11:43:14.495 INFO (MainThread) [custom_components.solis.soliscloud_api] {'Success': False, 'Message': 'Got http statuscode: 403', 'StatusCode': 403, 'Content': {'timestamp': 1672310594474, 'status': 403, 'error': 'Forbidden', 'message': 'wrong sign 数字签名不正确', 'path': '/v1/api/inverterList'}}

Maybe Solis locked my account ?

grlarsen commented 1 year ago

@luftdieb After reading your post about Solis' reply I ran the updated test script again loacally, this time with the following result:

user_station_list: [OK] station_detail(station_id= [I HAVE REMOVED THIS] ): [OK] collector_list(station_id= [I HAVE REMOVED THIS] ): [OK] collector_detail(station_id= [I HAVE REMOVED THIS] , collector_sn= [I HAVE REMOVED THIS] ): [OK] station_day( [I HAVE REMOVED THIS] ): [OK] station_month( [I HAVE REMOVED THIS] ): [OK] station_year( [I HAVE REMOVED THIS] ): [OK] station_all( [I HAVE REMOVED THIS] ): [OK] Exceptions: [None]

Following this I ran the integration installation through the UI (v. 3.2.1). And it worked right away. Currently no issues. ...Except a GND error somewere in the grid causing hypervoltage on one phase and the inverter to temporarily shut it self off. Datalogger is functional however.

Edit: corrected typo "expect" - "except"

devinmitchell commented 1 year ago

Hmmm, I'm still getting the same results as before. I've also not received a response back from Solis support.

test % python3 ./apitest_async.py -v
Verbose
Exceptions:  [ApiError('数据异常 请联系管理员')]
test % python3 ./soliscloud_test.py -v

{"code": "1", "data": null, "msg": "\u6570\u636e\u5f02\u5e38 \u8bf7\u8054\u7cfb\u7ba1\u7406\u5458", "success": true}

edit: I decided not to open another ticket. Better to be patient than to add to their existing workload.

hultenvp commented 1 year ago

Let's see what Solis comes up with in terms of fixing the API. I'm pretty convinced there's still (a) bug(s) on their side.

I've updated the test app. If userstationList call fails it now tries stationDetailList to get the list of station ID's --> Partially tested as the call fails for me with the abnormal data error If that also fails you can override the detect by defining the station_ids list yourself; you can add one or more station id's here.

devinmitchell commented 1 year ago

Thanks for all your work on this @hultenvp , this really is a great and much needed addition. I'll let you know as soon as I hear back from Solis support. I also just tried the updated code just to see, but the results are pretty much the same (though it's a bit more interesting at least!). :). First run was with plant_ids commented out, and the second run was after I put my plant_id in the file.

test % python3 ./apitest_async.py -v
Verbose
user_station_list(): Failed with: 数据异常 请联系管理员
Falling back to station_detail_list
station_detail_list(): Failed with: 数据异常 请联系管理员
Cannot retrieve station ID's, giving up
Exceptions:  [None]
test % python3 ./apitest_async.py -v
Verbose
Using predefined station list
station_detail(station_id= 1 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 2 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 9 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 8 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 4 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 9 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 1 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 9 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 1 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 9 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 4 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 4 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 8 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 9 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 1 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 7 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 6 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 9 ): Failed with: 数据异常 请联系管理员
station_detail(station_id= 2 ): Failed with: 数据异常 请联系管理员
collector_list( 1 ): Failed with: 数据异常 请联系管理员
collector_list( 2 ): Failed with: 数据异常 请联系管理员
collector_list( 9 ): Failed with: 数据异常 请联系管理员
collector_list( 8 ): Failed with: 数据异常 请联系管理员
collector_list( 4 ): Failed with: 数据异常 请联系管理员
collector_list( 9 ): Failed with: 数据异常 请联系管理员
collector_list( 1 ): Failed with: 数据异常 请联系管理员
collector_list( 9 ): Failed with: 数据异常 请联系管理员
collector_list( 1 ): Failed with: 数据异常 请联系管理员
collector_list( 9 ): Failed with: 数据异常 请联系管理员
collector_list( 4 ): Failed with: 数据异常 请联系管理员
collector_list( 4 ): Failed with: 数据异常 请联系管理员
collector_list( 8 ): Failed with: 数据异常 请联系管理员
collector_list( 9 ): Failed with: 数据异常 请联系管理员
collector_list( 1 ): Failed with: 数据异常 请联系管理员
collector_list( 7 ): Failed with: 数据异常 请联系管理员
collector_list( 6 ): Failed with: 数据异常 请联系管理员
collector_list( 9 ): Failed with: 数据异常 请联系管理员
collector_list( 2 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 1 ): Failed with: Timeout error occurred
inverter_list(station_id = 2 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 9 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 8 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 4 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 9 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 1 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 9 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 1 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 9 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 4 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 4 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 8 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 9 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 1 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 7 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 6 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 9 ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 4 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 8 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 1 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 7 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 7 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 7 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 7 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 6 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 6 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 6 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 6 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 9 ): Failed with: 数据异常 请联系管理员
station_day(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_month(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_year(station_id = 2 ): Failed with: 数据异常 请联系管理员
station_all(station_id = 2 ): Failed with: 数据异常 请联系管理员
Exceptions:  [None]
hultenvp commented 1 year ago

Hey @devinmitchell,

I think you made a mistake in the manual config of station id. The comma is needed to separate multiple station IDs, not the individual digits of a station:

station_ids = ["xxxxxxxxxxxxx","yyyyyyyyyyyy"]

For two

station_ids = ["xxxxxxxxxxxxx"] 

Is you have one

At least some calls should pass.

alienatedsec commented 1 year ago
user_station_list(): Failed with: 数据异常 请联系管理员
Falling back to station_detail_list
station_detail_list(): Failed with: 数据异常 请联系管理员
Cannot retrieve station ID's, giving up
Exceptions:  [None]
alienatedsec commented 1 year ago
Using predefined station list
station_detail(station_id= XXXXXXXXXXXX  ): Failed with: 数据异常 请联系管理员
collector_list( XXXXXXXXXXXX  ): Failed with: 数据异常 请联系管理员
inverter_list(station_id = XXXXXXXXXXXX  ): Failed with: 数据异常 请联系管理员
station_day(station_id = XXXXXXXXXXXX  ): Failed with: 数据异常 请联系管理员
station_month(station_id = XXXXXXXXXXXX ): Failed with: 数据异常 请联系管理员
station_year(station_id = XXXXXXXXXXXX ): Failed with: 数据异常 请联系管理员
station_all(station_id = XXXXXXXXXXXX ): Failed with: 数据异常 请联系管理员
Exceptions:  [None]

Made a mistake with station ids. Still, failing with the same message

luftdieb commented 1 year ago

Thanks a lot for your efforts, hultenvp! I can give you postiive feedback. By adding the PlantID, I'm getting this output D:\test\solis-sensor\test>python apitest_async.py Using predefined station list station_detail(station_id= STATION-ID-NO ): [OK] collector_list(station_id= STATION-ID-NO ): [OK] collector_detail(station_id= STATION-ID-NO , collector_sn= COLLECTOR-SN ): [OK] inverter_list( STATION-ID-NO ): [OK] inverter_detail(station_id = STATION-ID-NO , inverter_sn = INVERTER-ID-NO ): [OK] station_day( STATION-ID-NO ): [OK] station_month( STATION-ID-NO ): [OK] station_year( STATION-ID-NO ): [OK] station_all( STATION-ID-NO ): [OK] inverter_day( INVERTER-ID-NO ): [OK] inverter_month( INVERTER-ID-NO ): [OK] inverter_year( INVERTER-ID-NO ): [OK] inverter_all( INVERTER-ID-NO ): [OK] Exceptions: [None]

And if I'm adding -v, I'm also getting further information like (also for the other instances) inverter_all( INVERTER-ID-NO ): [OK]

[{"batteryChargeEnergy": 548, "batteryDischargeEnergy": 662, "consumeEnergy": 2457.0, "energy": 1758.0, "energyPec": "0.001", "energyStr": "MWh", "errorFlag": 0, "fullHour": 488.33, "gridPurchasedEnergy": 1045, "gridPurchasedIncome": 0.0, "gridSellEnergy": 460, "gridSellIncome": 0.0, "homeLoadEnergy": 2307, "id": "1308675218218778359", "money": 140.64, "moneyPec": "1", "moneyStr": "EUR", "oneSelf": 750.0, "produceEnergy": 1758.0, "timeZone": 1.0, "year": 2022}]

That looks really good! But this works only as you have suggested if you add your PlantId in quotation marks "" ;-)

But without the PlantID, it looks still as before D:\test\solis-sensor\test>python soliscloud_test.py

{"error": "Forbidden", "message": "\u4e0d\u5b58\u5728\u7684 appid invalidxxxxx", "path": "/v1/api/inveterDetail", "status": 403, "timestamp": 1672392756206}

Parallel I'm also still in contact with Solis support. They wrote back yesterday night and asked again for keyID and keySecret. So I gave them my data and even my C# code, for reproduction. Let's see, what happens.

Do you plan to add PlantID as an additional input field for general implementation in HA?

alienatedsec commented 1 year ago

@luftdieb @hultenvp will this matter if python2 or python3 is used?

luftdieb commented 1 year ago

good question. I'm testing this on my Windows with Python 3.11.0 installed. In HomeAssistent I'm using Python 3.10.9

I've just clone the latest version to my windows, make my adjustment to file apitest_async.py by adding these 3 informations image and then just calling the command "python apitest_async.py" in folder \test.

alienatedsec commented 1 year ago

just calling the command "python apitest_async.py" in folder \test.

Same here, but on the Ubuntu box

image

alienatedsec commented 1 year ago

good question. I'm testing this on my Windows with Python 3.11.0 installed. In HomeAssistent I'm using Python 3.10.9

Python 3.10.6 on my box

luftdieb commented 1 year ago

I don't know if it was a good idea, but I've logged in into my HA, have cloned your repo into /tmp/ and have tried to execute "python apitest_async.py". That first create an error, because aiohttp and requests from import section can't be used because missing in my HA environment. But after sudo pip install aiohttp and same for requests, it also give positive result image

luftdieb commented 1 year ago

In addition, I've received this feedback from Solis Support image

But if I'm using soliscloud_test. I'm still getting D:\test\solis-sensor\test>python soliscloud_test.py {"error": "Forbidden", "message": "\u4e0d\u5b58\u5728\u7684 appid invalidxxxxx", "path": "/v1/api/inveterDetail", "status": 403, "timestamp": 1672396455235}

BUT, if I'm using current implementaion in HA, it works :-) (Sorry, it's in german and the message is "Sucess! Configuration for station xxxx created." image

And after a while, I can see real values from cloud image

alienatedsec commented 1 year ago

I tested the integration and it still fails, but I managed to install requests and aiohttp as you suggested. Are you using HA OS? @luftdieb