Needs to import mqtt from awsiotsdk, not awsiot

[bwarden]

mqtt_connection_builder comes from awsiotsdk, not awsiot https://github.com/MTrab/pyworxcloud/blob/8dd13fd80da2af33e264d1b37e8e0ed579d03227/pyworxcloud/utils/mqtt.py#L14 https://github.com/MTrab/pyworxcloud/blob/8dd13fd80da2af33e264d1b37e8e0ed579d03227/requirements.txt#L2 https://github.com/MTrab/pyworxcloud/blob/8dd13fd80da2af33e264d1b37e8e0ed579d03227/pyproject.toml#L23 https://github.com/MTrab/pyworxcloud/blob/8dd13fd80da2af33e264d1b37e8e0ed579d03227/poetry.lock#L4 https://github.com/MTrab/pyworxcloud/blob/8dd13fd80da2af33e264d1b37e8e0ed579d03227/poetry.lock#L11

[MTrab]

This ain't the issue ;) mqtt_connection_builder is also present in awsiot, but for some reason that module doesn't get installed when installing this module.

[bwarden]

My mistake, you don't need to change the import; just the requirements/deps. It's definitely not in the awsiot wheel from pypi:

bwarden@snafu:/tmp$ wget -N https://files.pythonhosted.org/packages/32/cd/b7d1db293de37b62fc83a4c639ba4b917f3ba49b31f8f1589637ef7e4eed/awsiot-0.1.3-py3-none-any.whl
--2023-03-31 10:41:37--  https://files.pythonhosted.org/packages/32/cd/b7d1db293de37b62fc83a4c639ba4b917f3ba49b31f8f1589637ef7e4eed/awsiot-0.1.3-py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 199.232.145.63, 2a04:4e42:64::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|199.232.145.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5525 (5.4K) [binary/octet-stream]
Saving to: ‘awsiot-0.1.3-py3-none-any.whl’

awsiot-0.1.3-py3-none-any.whl                                       100%[==================================================================================================================================================================>]   5.40K  --.-KB/s    in 0s

2023-03-31 10:41:37 (27.4 MB/s) - ‘awsiot-0.1.3-py3-none-any.whl’ saved [5525/5525]

bwarden@snafu:/tmp$ zipinfo awsiot-0.1.3-py3-none-any.whl
Archive:  awsiot-0.1.3-py3-none-any.whl
Zip file size: 5525 bytes, number of entries: 13
-rw-r--r--  2.0 unx        0 b- defN 17-Aug-02 20:20 commands/__init__.py
-rw-r--r--  2.0 unx     2648 b- defN 17-Dec-07 03:47 commands/awsiot.py
-rw-r--r--  2.0 unx        0 b- defN 17-Aug-02 05:41 factories/__init__.py
-rw-r--r--  2.0 unx      215 b- defN 17-Aug-02 21:25 factories/aws/__init__.py
-rw-r--r--  2.0 unx     3825 b- defN 17-Aug-31 01:48 factories/aws/thing.py
-rw-r--r--  2.0 unx     1070 b- defN 17-Aug-31 00:37 factories/aws/thingtype.py
-rw-r--r--  2.0 unx       10 b- defN 17-Dec-10 23:00 awsiot-0.1.3.dist-info/DESCRIPTION.rst
-rw-r--r--  2.0 unx       49 b- defN 17-Dec-10 23:00 awsiot-0.1.3.dist-info/entry_points.txt
-rw-r--r--  2.0 unx     1064 b- defN 17-Dec-10 23:00 awsiot-0.1.3.dist-info/metadata.json
-rw-r--r--  2.0 unx       19 b- defN 17-Dec-10 23:00 awsiot-0.1.3.dist-info/top_level.txt
-rw-r--r--  2.0 unx       92 b- defN 17-Dec-10 23:00 awsiot-0.1.3.dist-info/WHEEL
-rw-r--r--  2.0 unx      782 b- defN 17-Dec-10 23:00 awsiot-0.1.3.dist-info/METADATA
-rw-r--r--  2.0 unx     1053 b- defN 17-Dec-10 23:00 awsiot-0.1.3.dist-info/RECORD
13 files, 10827 bytes uncompressed, 3777 bytes compressed:  65.1%

But it is in the wheel for awsiotsdk:

bwarden@snafu:/tmp$ wget -N https://files.pythonhosted.org/packages/47/e2/403be8d63f0bc1754c56e165125fc45262fb43f62ba0058a107946ad4ecf/awsiotsdk-1.12.6-py3-none-any.whl
--2023-03-31 10:42:31--  https://files.pythonhosted.org/packages/47/e2/403be8d63f0bc1754c56e165125fc45262fb43f62ba0058a107946ad4ecf/awsiotsdk-1.12.6-py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 199.232.145.63, 2a04:4e42:64::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|199.232.145.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 67577 (66K) [application/octet-stream]
Saving to: ‘awsiotsdk-1.12.6-py3-none-any.whl’

awsiotsdk-1.12.6-py3-none-any.whl                                   100%[==================================================================================================================================================================>]  65.99K  --.-KB/s    in 0.002s

2023-03-31 10:42:31 (27.8 MB/s) - ‘awsiotsdk-1.12.6-py3-none-any.whl’ saved [67577/67577]

bwarden@snafu:/tmp$ zipinfo awsiotsdk-1.12.6-py3-none-any.whl
Archive:  awsiotsdk-1.12.6-py3-none-any.whl
Zip file size: 67577 bytes, number of entries: 16
-rw-r--r--  2.0 unx     6203 b- defN 23-Mar-22 17:59 awsiot/__init__.py
-rw-r--r--  2.0 unx    32644 b- defN 23-Mar-22 17:50 awsiot/eventstreamrpc.py
-rw-r--r--  2.0 unx     7921 b- defN 23-Mar-22 17:50 awsiot/greengrass_discovery.py
-rw-r--r--  2.0 unx    25425 b- defN 23-Mar-22 17:50 awsiot/iotidentity.py
-rw-r--r--  2.0 unx    66866 b- defN 23-Mar-22 17:50 awsiot/iotjobs.py
-rw-r--r--  2.0 unx    74545 b- defN 23-Mar-22 17:50 awsiot/iotshadow.py
-rw-r--r--  2.0 unx    36984 b- defN 23-Mar-22 17:50 awsiot/mqtt5_client_builder.py
-rw-r--r--  2.0 unx    28186 b- defN 23-Mar-22 17:50 awsiot/mqtt_connection_builder.py
-rw-r--r--  2.0 unx     2337 b- defN 23-Mar-22 17:50 awsiot/greengrasscoreipc/__init__.py
-rw-r--r--  2.0 unx    61462 b- defN 23-Mar-22 17:50 awsiot/greengrasscoreipc/client.py
-rw-r--r--  2.0 unx    60881 b- defN 23-Mar-22 17:50 awsiot/greengrasscoreipc/clientv2.py
-rw-r--r--  2.0 unx   198720 b- defN 23-Mar-22 17:50 awsiot/greengrasscoreipc/model.py
-rw-r--r--  2.0 unx     5515 b- defN 23-Mar-22 17:59 awsiotsdk-1.12.6.dist-info/METADATA
-rw-r--r--  2.0 unx       92 b- defN 23-Mar-22 17:59 awsiotsdk-1.12.6.dist-info/WHEEL
-rw-r--r--  2.0 unx        7 b- defN 23-Mar-22 17:59 awsiotsdk-1.12.6.dist-info/top_level.txt
-rw-rw-r--  2.0 unx     1334 b- defN 23-Mar-22 17:59 awsiotsdk-1.12.6.dist-info/RECORD
16 files, 609122 bytes uncompressed, 65397 bytes compressed:  89.3%

[MTrab]

That's strange then

[MTrab]

Oh dammit - just saw - they install in the same path?!

[bwarden]

Well, no, they install in paths that don't match themselves. awsiot installs in 'commands' and 'factories' like a big jerk :D

[MTrab]

Wow how pro of them

[MTrab]

This might prove a problem :( awsiotsdk can't install on HAOS

[fra87]

Sorry for the question, but why did you decide to use awsiot instead of a "pure" and more classic paho-mqtt? I had a very quick look at the code, but it seems to me you implemented most of the code yourself (I did not have time to check the authentication, so maybe that is the answer).

Please consider this as a genuine question; I'm genuinely curious about what are the advantages you saw using this library instead of the other one

Thank you for the great work and the continuous support

[MTrab]

Might as well try going back - I just used what was provided in the API documentation from Positec

[GSzabados]

This solution might can help you out to install awsiotsdk:

https://github.com/rowysock/home-assistant-HomeWhiz/pull/57

But there is another issue open in the same repo, that it would not work if the RaspberryPi OS 32bit is used.

https://github.com/rowysock/home-assistant-HomeWhiz/issues/97

[fra87]

Tomorrow I have some free time; I'll try to fork the repo and see if I can have it implemented in paho-mqtt, and if it works I'll post a PR so you can review it and maybe this will help you in fixing this

[pszypowicz]

Tomorrow I have some free time; I'll try to fork the repo and see if I can have it implemented in paho-mqtt, and if it works I'll post a PR so you can review it and maybe this will help you in fixing this

@fra87 Please let us know about the progress or your findings about replacing it with paho. So far I see that they use ‚custom authentication’ in awsiotsdk. Also AWS docs states that they require ALPN for SSL context (but i may interpret current connection wrong)

https://docs.aws.amazon.com/iot/latest/developerguide/protocols.html

[pszypowicz]

For anyone who wants to participate, I will simplify some things for a sake of discussion:

Current configuration of mqtt connection is handled here: https://github.com/MTrab/pyworxcloud/blob/master/pyworxcloud/utils/mqtt.py#L102-L111

You can set a break point in your debugger to see which endpoint you need, and essentially: accesstokenparts.

This is (afaik) what we need to figure out with paho-mqtt.

[MTrab]

Might already have this figured out ;)

[pszypowicz]

@MTrab I'm not doubting that :D But since you are under NDA, I want to direct fellow devs in right direction :)

[pszypowicz]

I think I have it?

Of course I copied user_id, token and MQTT_TOPIC, from debugger.

import paho.mqtt.client as mqtt
from uuid import uuid4
import urllib.parse
import ssl

MQTT_TOPIC = ""

brandprefix = "WX"
user_id = ""
endpoint = "iot.eu-west-1.worxlandroid.com"
token = ""

# Define on connect event function
# We shall subscribe to our Topic in this function
def on_connect(client, userdata, flags, rc):
    """on connect"""
    mqttc.subscribe(MQTT_TOPIC, 0)

# Define on_message event function.
# This function will be invoked every time,
# a new message arrives for the subscribed topic
def on_message(mosq, obj, msg):
    """on message"""
    print("Topic: " + str(msg.topic))
    print("QoS: " + str(msg.qos))
    print("Payload: " + str(msg.payload))

def on_subscribe(mosq, obj, mid, granted_qos):
    """on subscribe"""
    print("Subscribed to Topic: " + MQTT_TOPIC + " with QoS: " + str(granted_qos))

# Assign event callbacks

ssl_context = ssl.create_default_context()
ssl_context.set_alpn_protocols(["mqtt"])

accesstokenparts = token.replace("_", "/").replace("-", "+").split(".")

mqttc = mqtt.Client(
    client_id=f"{brandprefix}/USER/{user_id}/bot/{uuid4()}",
)

mqttc.tls_set_context(context=ssl_context)

mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe

mqttc.username_pw_set(
    f"bot?jwt={urllib.parse.quote(accesstokenparts[0])}.{urllib.parse.quote(accesstokenparts[1])}&x-amz-customauthorizer-signature={urllib.parse.quote(accesstokenparts[2])}"
)

mqttc.connect(host=endpoint, port=443)
mqttc.loop_forever()

[fra87]

I tried today to complete it, but my landroid was offline and so I could not test anything.

I'll try to test it tomorrow, fix the behavior and if it works then I'll post a PR

[MTrab]

.tls_set_context(context=ssl_context)

Just about the same I have come to, but it just don't seem to work as expected from my tests.

[MTrab]

I have publish what I have so far, in this branch: https://github.com/MTrab/pyworxcloud/blob/Moving-to-Paho-MQTT/pyworxcloud/utils/mqtt.py

Not sure how much more I'll get around to try today, and will most certainly have no time tomorrow as I'm off to work the whole day. I have fixed test_async.py so it should be working and awaiting data. Nothing ever arrives, and no log entries from the _on_connect void, so not sure the connection string is working as intended.

[pszypowicz]

I was able to receive messages running my code, and I triggered an update using your current implementation

cloud = WorxCloud(EMAIL, PASS, TYPE)
cloud.authenticate()
cloud.connect()

device = cloud.devices["name"]

cloud.update(device.serial_number)

[pszypowicz]

I have publish what I have so far, in this branch: https://github.com/MTrab/pyworxcloud/blob/Moving-to-Paho-MQTT/pyworxcloud/utils/mqtt.py

Not sure how much more I'll get around to try today, and will most certainly have no time tomorrow as I'm off to work the whole day. I have fixed test_async.py so it should be working and awaiting data. Nothing ever arrives, and no log entries from the _on_connect void, so not sure the connection string is working as intended.

@MTrab You have an issue here: https://github.com/MTrab/pyworxcloud/blob/Moving-to-Paho-MQTT/pyworxcloud/utils/mqtt.py#L111

At very least signature is in accesstokenparts[2] not accesstokenparts[1] in x-amz-customauthorizer-signature={urllib.parse.quote(accesstokenparts[1])}

also, I did not had to define at all `x-amz-customauthorizer-name='``

[MTrab]

Arh! Dammit - thank you ;) Now it connects, but still doesn't seem to receive data

[MTrab]

Okay - seems like I'm sending data now. At least I can see a refresh in timestamp for last update in the app. Still not receiving though - might just be a small issue and me being awfully tired right now

[MTrab]

DOH! Dummy slap! DO NOT subscribe before the connection is established :D Not to figure out the other errors that just turned up. Should be easy though.

[pszypowicz]

Yeah, that's what I always see in paho, subscribe happening in the on_connect callback.

Good luck!

[MTrab]

Fixed in 3.1.0 which will be up shortly

[fra87]

Great job

As for me, I learned that I cannot work on something remotely, I need to be there (my landroid blocked again in the garden, so no MQTT messages for me again -.-)

Just a question. I noticed that you left the default QoS in the subscribe call (which is 0, so "at most one"), and explicitly set it to 0 in all the publish calls, while previously they were all set to 1 (at least one). Was this done for a specific reason?

[MTrab]

No, it was a mistake ;) Will be reverted to 1 in next release