Cannot authenticate/retrieve info about apliances

asierralozano commented 2 months ago

Hello!

I was having some issues with the latest dev branch updates regarding authentication, and I decided to debug a bit the issue. I've tried running a simple pyhOn --user USERNAME --pass PASSWORD, and I get the following error:

DEBUG:asyncio:Using selector: EpollSelector
DEBUG:pyhon.connection.auth:Starting OAuth2 authorization
DEBUG:pyhon.connection.auth:Logging in
DEBUG:pyhon.connection.auth:Getting tokens
DEBUG:pyhon.connection.auth:Trying to retrieve Cognito token
Traceback (most recent call last):
  File "/usr/local/bin/pyhOn", line 8, in <module>
    sys.exit(start())
             ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/__main__.py", line 110, in start
    asyncio.run(main())
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/__main__.py", line 76, in main
    async with Hon(*get_login_data(args), test_data_path=test_data_path) as hon:
  File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 47, in __aenter__
    return await self.setup()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 75, in setup
    await self._create_appliance(appliance, self._api)
  File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 58, in _create_appliance
    await appliance.load_statistics()
  File "/usr/local/lib/python3.12/site-packages/pyhon/appliance.py", line 211, in load_statistics
    self._statistics |= await self.api.load_maintenance(self)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/connection/api.py", line 170, in load_maintenance
    async with self._session.get(
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/connection/handler/hon.py", line 40, in _request
    async with super()._request(method, *args, **kwargs) as response:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/connection/handler/base.py", line 78, in _request
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1093, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url='https://api-iot.he.services/commands/v1/maintenance-cycle?macAddress=90-38-0c-14-eb-84'
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f645333cad0>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f64534c83e0>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f6452ec2b70>, 7912.299655275), (<aiohttp.client_proto.ResponseHandler object at 0x7f6452ec1d90>, 7912.507932813), (<aiohttp.client_proto.ResponseHandler object at 0x7f6452ec2ab0>, 7914.279372112)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f6452ee0050>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f6452ee0530>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f6452e53470>, 7910.747858079)]', '[(<aiohttp.client_proto.ResponseHandler object at 0x7f6452ec39b0>, 7911.276245414)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f6452ee0560>

I've tried reinstalling pyhOn with a simple pip uninstall pyhOn --force and pip install pyhOn to use the oficial pip one, and everything works as expected, so I guess there is an issue within this one.

Let me know if there is anything I can do :)

Thanks!

kubasaw commented 2 months ago

Can you check if the same problem exists in original pyhOn @ 0.16.1 version? This is a last version without MQTT.

We have the same mechanism to load all appliances data (especially MQTT topics). After this steps we switch to MQTT to get updates (instead of load it again with polling), so at this moment both versions should work the same way.

If there are differences, we are be able to investigate, where I inserted bugs :)

asierralozano commented 2 months ago

I can confirm that with the version 0.16, everything works as expected (regarding the initialisation of the integration). With that version, we have the known issue about losing the connection and having no updates, needing to restart the integration in order to work, but that's basically the issue you were trying to fix with the MQTT updates :)

asierralozano commented 2 months ago

I didn't have enough time to debug it properly, so I did a simple single tests:

I've removed the line self._statistics |= await self.api.load_maintenance(self) from the appliance.py, and it is not complaining. Not sure if that is gonna make the code work at all (as I said, I didn't have the time to properly the impact of that line within the whole code), but at least is not failing

I'm sharing that with you in case it is helpful for you :)

kubasaw commented 2 months ago

For me beta 11 is not working with or without docker exec command

Error setting up entry xxxxxxxxxxxxx@gmail.com for hon
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 604, in async_setup
    result = await component.async_setup_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/hon/__init__.py", line 34, in async_setup_entry
    hon = await Hon(
          ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 75, in setup
    await self._create_appliance(appliance, self._api)
  File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 58, in _create_appliance
    await appliance.load_statistics()
  File "/usr/local/lib/python3.12/site-packages/pyhon/appliance.py", line 211, in load_statistics
    self._statistics |= await self.api.load_maintenance(self)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/connection/api.py", line 177, in load_maintenance
    async with self._session.get(
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/connection/handler/hon.py", line 40, in _request
    async with super()._request(method, *args, **kwargs) as response:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/connection/handler/base.py", line 78, in _request
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1093, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url='https://api-iot.he.services/commands/v1/maintenance-cycle?macAddress=XX-XX-XX-XX-XX-XX'

Originally posted by @Doktor-X in https://github.com/IoTLabs-pl/hon/issues/2#issuecomment-2308118223

kubasaw commented 2 months ago

Updated to beta 12 pyhon is 0.18.0b1 I have flexis plus AC

Home Assistant Core
2024-08-24 19:55:54.974 DEBUG (MainThread) [pyhon.connection.auth] Starting OAuth2 authorization
2024-08-24 19:55:56.570 DEBUG (MainThread) [pyhon.connection.auth] Logging in
2024-08-24 19:55:57.574 DEBUG (MainThread) [pyhon.connection.auth] Getting tokens
2024-08-24 19:55:59.662 DEBUG (MainThread) [pyhon.connection.auth] Trying to retrieve Cognito token
2024-08-24 19:56:01.087 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module with args ('pyhon.appliances.ac',) inside the event loop by custom integration 'hon' at custom_components/hon/__init__.py, line 34: hon = await Hon( (offender: /usr/local/lib/python3.12/site-packages/pyhon/appliance.py, line 52: self._extra: Optional[ApplianceBase] = importlib.import_module(), please create a bug report at https://github.com/IoTLabs-pl/hon/issues
2024-08-24 19:56:03.712 ERROR (MainThread) [pyhon.connection.handler.base] hOn Authentication Error
File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 75, in setup
File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 58, in _create_appliance
File "/usr/local/lib/python3.12/site-packages/pyhon/appliance.py", line 211, in load_statistics
File "/usr/local/lib/python3.12/site-packages/pyhon/connection/api.py", line 177, in load_maintenance
File "/usr/local/lib/python3.12/site-packages/pyhon/connection/handler/hon.py", line 40, in _request
File "/usr/local/lib/python3.12/site-packages/pyhon/connection/handler/base.py", line 79, in _request

Originally posted by @Doktor-X in https://github.com/IoTLabs-pl/hon/issues/2#issuecomment-2308476299

kubasaw commented 2 months ago

Same as @Doktor-X

Originally posted by @AndreaSegoni in https://github.com/IoTLabs-pl/hon/issues/2#issuecomment-2308143714

kubasaw commented 2 months ago

I'm not able to resolve this issue without help. Somebody who experienced this issue may help to discover causes by running pyhOn@0.16.1 (last @Andre0512, non-MQTT version) and last version from the fork and notice differences in the requests sent. I will try to prepare more detailed info, how to do it in case if anybody needs help in investigation.

arthanoss commented 2 months ago

It seems that the earlier versions of pyhOn just ignored that this maintenance-cycle endpoint fails with 400. I'm not exactly sure what it's for and if it even worked before, but given this piece of code:

https://github.com/IoTLabs-pl/pyhOn/blob/74f0b42d34130d39184fe224808fe42a16a29db0/pyhon/appliance.py#L209C1-L211C66

it might be optional after all. Perhaps it only works for certain kinds of appliances.

kubasaw commented 2 months ago

It's probably fixed in v0.18.0-beta.2 (hOn v0.14.0-beta.13). I will close this thread if you confirm fix is working well.

arthanoss commented 2 months ago

It's still there.

I dug deeper and discovered that there are actually nice error messages given back from the API:

GET /commands/v1/maintenance-cycle?macAddress=xx-xx-xx-xx-xx-xx HTTP/1.1
(...)

HTTP/1.1 400 Bad Request
(...)

{"error":{"message":"IAT","statusCode":400,"detail":"Invalid applianceType"}}

So it confirms my theory about the wrong kind of appliance (i only have AC to test). PyhOn should probably handle these kinds of errors in the future (or, in this case, maybe never send a request in the first place if we know more about which appliances work with this endpoint). Or just ignore it like in previous versions :P

Doktor-X commented 2 months ago

It's probably fixed in v0.18.0-beta.2 (hOn v0.14.0-beta.13). I will close this thread if you confirm fix is working well.

Its not working for my ac

hOn Authentication Error 1 200 - https://api-iot.he.services/commands/v1/appliance 2 200 - https://api-iot.he.services/commands/v1/appliance/XX-XX-XX-XX-XX-XXfavourite 3 200 - https://api-iot.he.services/commands/v1/retrieve?applianceType=AC&applianceModelId=312&macAddress=XX-XX-XX-XX-XX-XX&os=android&appVersion=2.6.5&code=AAA6S4E0000&firmwareId=41&fwVersion=5.15.0&series=flexis 4 200 - https://api-iot.he.services/commands/v1/appliance/XX-XX-XX-XX-XX-XX/history 5 200 - https://api-iot.he.services/commands/v1/context?macAddress=XX-XX-XX-XX-XX-XX&applianceType=AC&category=CYCLE 6 200 - https://api-iot.he.services/commands/v1/statistics?macAddress=XX-XX-XX-XX-XX-XX&applianceType=AC 7 400 - https://api-iot.he.services/commands/v1/maintenance-cycle?macAddress=XX-XX-XX-XX-XX-XX ================ Error ================= 400, message='Bad Request', url='https://api-iot.he.services/commands/v1/maintenance-cycle?macAddress=XX-XX-XX-XX-XX-XX'

asierralozano commented 2 months ago

I can confirm that it is not working at least for AC appliance type.

Also, based on your explanation about how the Mqtt connection is being used, I think that it is related to the MQTT Authentication, because I've tried to run a simple StartCommand after initializing the Hon object (no mqtt time to handle any update using it), and there is no issue at all, the command executes successfully.

Not sure if it is helpful or not.

kubasaw commented 2 months ago

Thank you for testing! I thought it was a bug: while massively editing the API polling logic, I accidentally changed the endpoint from maintenance-cycle to maintenance. I’ll add a try-catch block soon to silence this exception.

Dekkee commented 2 months ago

After updating to 0.14.0-beta14 (pyhon 0.18.0b3), I get new error:

2024-08-31 12:55:57.146 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry **** for hon
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 604, in async_setup
    result = await component.async_setup_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/hon/__init__.py", line 38, in async_setup_entry
    hon = await Hon(
          ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 51, in setup
    async for a in Appliance.create_from_data(self._api, appliance_data):
  File "/usr/local/lib/python3.12/site-packages/pyhon/appliances/_base.py", line 84, in create_from_data
    await appliance.load_commands()
  File "/usr/local/lib/python3.12/site-packages/pyhon/appliances/_base.py", line 292, in load_commands
    self.commands, self.additional_data = loader(self, payload)
                                          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/command_loader.py", line 37, in loader
    _clean_name(category_name): HonCommand(
                                ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/commands.py", line 37, in __init__
    self._load_parameters(attributes)
  File "/usr/local/lib/python3.12/site-packages/pyhon/commands.py", line 74, in _load_parameters
    self._create_parameters(data, name, key)
  File "/usr/local/lib/python3.12/site-packages/pyhon/commands.py", line 85, in _create_parameters
    self._rules.append(HonRuleSet(self, data["fixedValue"]))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 22, in __init__
    self._parse_rule(rule)
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 32, in _parse_rule
    self._parse_conditions(param_key, trigger_key, trigger_data)
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 54, in _parse_conditions
    self._parse_conditions(param_key, extra_key, extra_data, extra)
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 43, in _parse_conditions
    for multi_trigger_value, param_data in trigger_data.items():
                                           ^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'items'

I have AC. Last working version in beta10.

kubasaw commented 2 months ago

@Dekkee Thank you for feedback. I was oriented to make my own washing machine work properly, but this was a mistake. In the next step I will setup test suite based on available appliance data from https://github.com/Andre0512/hon-test-data to ensure that changes will not lead to regressions. After that, I will try to fix problem with Rules.

welderpb commented 1 month ago

After updating to 0.14.0-beta14 (pyhon 0.18.0b3), I get new error:

2024-08-31 12:55:57.146 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry **** for hon
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 604, in async_setup
    result = await component.async_setup_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/hon/__init__.py", line 38, in async_setup_entry
    hon = await Hon(
          ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/hon.py", line 51, in setup
    async for a in Appliance.create_from_data(self._api, appliance_data):
  File "/usr/local/lib/python3.12/site-packages/pyhon/appliances/_base.py", line 84, in create_from_data
    await appliance.load_commands()
  File "/usr/local/lib/python3.12/site-packages/pyhon/appliances/_base.py", line 292, in load_commands
    self.commands, self.additional_data = loader(self, payload)
                                          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/command_loader.py", line 37, in loader
    _clean_name(category_name): HonCommand(
                                ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/commands.py", line 37, in __init__
    self._load_parameters(attributes)
  File "/usr/local/lib/python3.12/site-packages/pyhon/commands.py", line 74, in _load_parameters
    self._create_parameters(data, name, key)
  File "/usr/local/lib/python3.12/site-packages/pyhon/commands.py", line 85, in _create_parameters
    self._rules.append(HonRuleSet(self, data["fixedValue"]))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 22, in __init__
    self._parse_rule(rule)
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 32, in _parse_rule
    self._parse_conditions(param_key, trigger_key, trigger_data)
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 54, in _parse_conditions
    self._parse_conditions(param_key, extra_key, extra_data, extra)
  File "/usr/local/lib/python3.12/site-packages/pyhon/rules.py", line 43, in _parse_conditions
    for multi_trigger_value, param_data in trigger_data.items():
                                           ^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'items'

I have AC. Last working version in beta10.

Same issue with my AC..

kubasaw commented 1 month ago

I've updated the diagnostic part of the code, and it would be helpful if anyone experiencing issue could export their device data.

To do this:

Install pyhon from the branch:

pip install git+https://github.com/IoTLabs-pl/pyhOn.git@dev

Run the following command to export the data:

python -m pyhon --user EMAIL --password PASSWORD --anonymous export --zip

This will create a ZIP file with raw API responses to reproduce issue and try to resolve it.

welderpb commented 1 month ago

@kubasaw this command return error:

DEBUG:asyncio:Using selector: EpollSelector DEBUG:pyhon.apis.auth:Trying to retrieve Cognito token DEBUG:pyhon.apis.auth:Starting OAuth2 authorization DEBUG:pyhon.apis.auth:Logging in DEBUG:pyhon.apis.auth:Getting tokens Traceback (most recent call last): File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/diagnostic/tool.py", line 46, in payload return self.processor(await self.response.json()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/diagnostic/tool.py", line 180, in call.processor = lambda x: tool.load(x).anonymize().get_result() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/diagnostic/_dict_tools.py", line 138, in get_result r = DictTool.__inflate(self._data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/diagnostic/_dict_tools.py", line 89, in __inflate key = next(iter(data.keys())) ^^^^^^^^^^^^^^^^^^^^^^^ StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/main.py", line 115, in asyncio.run(main()) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/main.py", line 91, in main await tool.Diagnoser.from_raw_api_data(hon._api, path, anon, as_zip) # noqa: SLF001 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/diagnostic/tool.py", line 89, in from_raw_api_data await diagnoser.api_dump(directory, anonymous, as_zip) File "/home/welder/temp/env/lib/python3.12/site-packages/pyhon/diagnostic/tool.py", line 190, in api_dump } | {call.filename: await call.payload() for call in self.call_data} ^^^^^^^^^^^^^^^^^^^^ RuntimeError: coroutine raised StopIteration

kubasaw commented 1 month ago

@welderpb Thanks! It is the other bug on my side. I will try to fix it tomorrow.

EDIT: I merged fix into dev branch

johnhoran commented 1 month ago

python -m pyhon --user EMAIL --password PASSWORD --anonymous export --zip

I tried running this, but the files it produced weren't anonymous.

kubasaw commented 1 month ago

@johnhoran Can you identify which json keys (or value patterns) should be anonymised?

welderpb commented 3 weeks ago

Can you identify which json keys (or value patterns) should be anonymised?

Might be serial numbers, MAC's, etc

kubasaw commented 3 weeks ago

Generally, at the moment, all data should be anonymised with -x/--anonymous flag in CLI. If it is not, there might be some issue to be reported and resolved. I cannot reproduce this issue: my all personal data was replaced in dumps.

I have rewritten this part of code not to replace MACs to XX-XX-XX-XX-XX-XX, but some random value which will be the same in all dumped data parts. It allows not to lose relations between partial data. The same rules are executed on timestamps, serial numbers etc.

Hatoris commented 3 weeks ago

Hello @kubasaw ,

First of all, thank you for keeping the nice Hon integration on track :) .

I got a connection error, initially it was the same error as the PR but now it's differents but still related to Auth. I join the Log and the debugging.

For context, I have move from Andre to your integration, by removing device and reinstalling yours, maybe I miss to clean stuff in process.

If you need help on testing, or need futher info please do not hesitate :)

IoTLabs-pl / pyhOn

Cannot authenticate/retrieve info about apliances #5