feat(uat): implement request response features in python paho client

Description of changes:

Update Python Paho client to handle PUBLISH Response Topic in Tx and Rx directions
Update Python Paho client to handle PUBLISH Correlation Data in Tx and Rx directions
Update Python Paho client to handle CONNECT Request Response Information

(handling response information in CONNACK was implemented earlier)

Why is this change necessary:

Part of Python Paho client implementation

How was this change tested: Manual run of mqtt-client-control and python client. MQTT connect, subscribe, unsubscribe, publish, delivery of received MQTT message and disconnect are successful. Linking and shutdown steps are successful.

Test results: Python Paho client output

client-python-paho.exe python-paho-agent
[DEBUG] 2023-07-12 17:21:50.751 asyncio - Using selector: SelectSelector
[INFO ] 2023-07-12 17:21:50.754 GRPCLib - Initialize gRPC library
[INFO ] 2023-07-12 17:21:50.754 GRPCLink - Making gPRC client connection with 127.0.0.1:47619 as python-paho-agent...
[INFO ] 2023-07-12 17:21:50.958 GRPCLink - Client connection with Control is established, local address is 127.0.0.1
[DEBUG] 2023-07-12 17:21:50.959 grpc._cython.cygrpc - Using AsyncIOEngine.POLLER as I/O engine
[INFO ] 2023-07-12 17:21:51.031 MQTTLib - Initialize Paho MQTT library
[INFO ] 2023-07-12 17:21:51.031 GRPCLink - Handle gRPC requests
[INFO ] 2023-07-12 17:21:51.032 GRPCControlServer - Server awaiting termination
[INFO ] 2023-07-12 17:21:54.105 GRPCControlServer - createMqttConnection: clientId Python_Paho_Client broker a3t8vwpkw3sltg-ats.iot.eu-west-1.amazonaws.com:8883
[INFO ] 2023-07-12 17:21:54.112 MqttConnection - Creating MQTT 5 client with TLS
[INFO ] 2023-07-12 17:21:54.113 MqttConnection - CONNECT TX user property 'region', 'US'
[INFO ] 2023-07-12 17:21:54.113 MqttConnection - CONNECT TX user property 'type', 'JSON'
[INFO ] 2023-07-12 17:21:54.113 MqttConnection - Copied TX request response information 'True'
[INFO ] 2023-07-12 17:21:54.113 MqttConnection - MQTT connection ID 0 connecting...
[DEBUG] 2023-07-12 17:21:54.419 AsyncioHelper - On socket open
[DEBUG] 2023-07-12 17:21:54.420 AsyncioHelper - On socket register write
[DEBUG] 2023-07-12 17:21:54.420 AsyncioHelper - On socket unregister write
[INFO ] 2023-07-12 17:21:54.535 MqttConnection - MQTT connection ID 0 connected, client id Python_Paho_Client
[DEBUG] 2023-07-12 17:21:59.656 GRPCControlServer - Subscription: filter test/topic QoS 0 noLocal 0 retainAsPublished 0 retainHandling 2
[INFO ] 2023-07-12 17:21:59.657 GRPCControlServer - SubscribeMqtt connection_id 0
[INFO ] 2023-07-12 17:21:59.657 MqttConnection - SUBSCRIBE TX user property 'region', 'US'
[INFO ] 2023-07-12 17:21:59.657 MqttConnection - SUBSCRIBE TX user property 'type', 'JSON'
[DEBUG] 2023-07-12 17:21:59.657 AsyncioHelper - On socket register write
[DEBUG] 2023-07-12 17:21:59.657 AsyncioHelper - On socket unregister write
[DEBUG] 2023-07-12 17:21:59.761 MqttConnection - Subscribed on filters 'test/topic' with QoS 0 no local False retain as published False retain handing 2 with message id 1
[INFO ] 2023-07-12 17:22:09.782 GRPCControlServer - PublishMqtt connection_id 0 topic test/topic retain 0
[INFO ] 2023-07-12 17:22:09.782 MqttConnection - PUBLISH TX user property 'region', 'US'
[INFO ] 2023-07-12 17:22:09.783 MqttConnection - PUBLISH TX user property 'type', 'JSON'
[INFO ] 2023-07-12 17:22:09.783 MqttConnection - Copied TX payload format indicator 'True'
[INFO ] 2023-07-12 17:22:09.783 MqttConnection - Copied TX message expiry interval 3600
[WARNING] 2023-07-12 17:22:09.783 MqttConnection - Copied TX response topic '/thing1/response/topic'
[WARNING] 2023-07-12 17:22:09.784 MqttConnection - Copied TX correlation data 'b'correlation_data''
[INFO ] 2023-07-12 17:22:09.784 MqttConnection - Copied TX content type 'text/plain; charset=utf-8'
[DEBUG] 2023-07-12 17:22:09.784 AsyncioHelper - On socket register write
[DEBUG] 2023-07-12 17:22:09.785 AsyncioHelper - On socket unregister write
[DEBUG] 2023-07-12 17:22:09.864 MqttConnection - Published on topic 'test/topic' QoS 1 retain 0 with rc 0 message id 2
[DEBUG] 2023-07-12 17:22:09.866 MqttConnection - onMessage message b'Hello World!' from topic 'test/topic'
[INFO ] 2023-07-12 17:22:09.866 MqttConnection - Copied RX payload format indicator '1'
[INFO ] 2023-07-12 17:22:09.867 MqttConnection - Copied RX message expiry interval 3599
[INFO ] 2023-07-12 17:22:09.867 MqttConnection - Copied RX message response topic '/thing1/response/topic'
[INFO ] 2023-07-12 17:22:09.867 MqttConnection - Copied RX message correlation data 'b'correlation_data''
[INFO ] 2023-07-12 17:22:09.867 MqttConnection - PUBLISH RX user property 'region', 'US'
[INFO ] 2023-07-12 17:22:09.868 MqttConnection - PUBLISH RX user property 'type', 'JSON'
[INFO ] 2023-07-12 17:22:09.869 MqttConnection - Copied RX content type 'text/plain; charset=utf-8'
[INFO ] 2023-07-12 17:22:09.870 GRPCDiscoveryClient - Sending OnReceiveMessage request agent_id 'python-paho-agent' connection_id 0 message b'Hello World!' on topic 'test/topic'
[DEBUG] 2023-07-12 17:22:14.875 GRPCControlServer - Unsubscription: filter test/topic
[INFO ] 2023-07-12 17:22:14.875 GRPCControlServer - UnsubscribeMqtt connection_id 0
[INFO ] 2023-07-12 17:22:14.877 MqttConnection - UNSUBSCRIBE TX user property 'region', 'US'
[INFO ] 2023-07-12 17:22:14.878 MqttConnection - UNSUBSCRIBE TX user property 'type', 'JSON'
[DEBUG] 2023-07-12 17:22:14.880 AsyncioHelper - On socket register write
[DEBUG] 2023-07-12 17:22:14.880 AsyncioHelper - On socket unregister write
[DEBUG] 2023-07-12 17:22:14.977 MqttConnection - Unsubscribed from filters 'test/topic' with message id 3
[INFO ] 2023-07-12 17:22:24.989 GRPCControlServer - CloseMqttConnection connection_id 0 reason 4
[INFO ] 2023-07-12 17:22:24.989 MqttConnection - Disconnect MQTT connection with reason code 4
[INFO ] 2023-07-12 17:22:24.990 MqttConnection - DISCONNECT TX user property 'region', 'US'
[INFO ] 2023-07-12 17:22:24.990 MqttConnection - DISCONNECT TX user property 'type', 'JSON'
[DEBUG] 2023-07-12 17:22:24.993 AsyncioHelper - On socket register write
[INFO ] 2023-07-12 17:22:24.993 GRPCDiscoveryClient - Sending OnMqttDisconnect request agent_id 'python-paho-agent' connection_id 0 error 'None'
[DEBUG] 2023-07-12 17:22:25.005 AsyncioHelper - On socket unregister write
[DEBUG] 2023-07-12 17:22:25.006 AsyncioHelper - On socket close
[INFO ] 2023-07-12 17:22:25.020 GRPCControlServer - shutdownAgent: reason That's it.
[INFO ] 2023-07-12 17:22:25.022 GRPCControlServer - Server termination done
[INFO ] 2023-07-12 17:22:25.023 GRPCLink - Shutdown gPRC link
[INFO ] 2023-07-12 17:22:25.052 Main - Execution done successfully

Mqtt Client Control output

java -Dmqtt_client_id=Python_Paho_Client -Dmqtt_broker_addr=a3t8vwpkw3sltg-ats.iot.eu-west-1.amazonaws.com -Dmqtt_client_ca_file=creds_aws/rootCA.pem -Dmqtt_client_cert_file=creds_aws/thingCert.crt -Dmqtt_client_key_file=creds_aws/privKey.key -jar target/aws-greengrass-testing-mqtt-client-control.jar 47619 true true
[INFO ] 2023-07-12 17:21:42.020 [main] ExampleControl - Control: port 47619, with TLS, MQTT v5
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
[INFO ] 2023-07-12 17:21:43.694 [main] EngineControlImpl - MQTT client control gRPC server started, listening on 47619
[INFO ] 2023-07-12 17:21:50.945 [grpc-default-executor-0] GRPCDiscoveryServer - RegisterAgent: agentId python-paho-agent
[INFO ] 2023-07-12 17:21:50.964 [grpc-default-executor-0] GRPCDiscoveryServer - DiscoveryClient: agentId python-paho-agent address 127.0.0.1 port 62809
[INFO ] 2023-07-12 17:21:50.968 [grpc-default-executor-0] EngineControlImpl - Created new agent control for python-paho-agent on 127.0.0.1:62809
[INFO ] 2023-07-12 17:21:51.022 [grpc-default-executor-0] ExampleControl - Agent python-paho-agent is connected
[INFO ] 2023-07-12 17:21:51.029 [pool-2-thread-1] AgentTestScenario - Playing test scenario for agent id python-paho-agent
[INFO ] 2023-07-12 17:21:54.041 [pool-2-thread-1] AgentTestScenario - Set CONNECT user property: region, US
[INFO ] 2023-07-12 17:21:54.042 [pool-2-thread-1] AgentTestScenario - Set CONNECT user property: type, JSON
[INFO ] 2023-07-12 17:21:54.050 [pool-2-thread-1] AgentTestScenario - Set CONNECT request response information true
[INFO ] 2023-07-12 17:21:54.642 [pool-2-thread-1] AgentControlImpl - Created connection with id 0 CONNACK 'sessionPresent: false
reasonCode: 0
receiveMaximum: 100
maximumQoS: 1
retainAvailable: true
maximumPacketSize: 149504
wildcardSubscriptionsAvailable: true
subscriptionIdentifiersAvailable: false
sharedSubscriptionsAvailable: true
serverKeepAlive: 60
topicAliasMaximum: 8
'
[INFO ] 2023-07-12 17:21:54.642 [pool-2-thread-1] AgentControlImpl - createMqttConnection: MQTT connectionId 0 created
[INFO ] 2023-07-12 17:21:54.645 [pool-2-thread-1] AgentTestScenario - MQTT connection with id 0 is established
[INFO ] 2023-07-12 17:21:59.648 [pool-2-thread-1] AgentTestScenario - Set SUBSCRIBE user property: region, US
[INFO ] 2023-07-12 17:21:59.649 [pool-2-thread-1] AgentTestScenario - Set SUBSCRIBE user property: type, JSON
[INFO ] 2023-07-12 17:21:59.653 [pool-2-thread-1] AgentControlImpl - SubscribeMqtt: subscribe on connection 0
[INFO ] 2023-07-12 17:21:59.765 [pool-2-thread-1] AgentTestScenario - Subscribe response: connectionId 0 reason codes [0] reason string ''
[INFO ] 2023-07-12 17:22:09.768 [pool-2-thread-1] AgentTestScenario - Set PUBLISH payload format indicator true
[INFO ] 2023-07-12 17:22:09.769 [pool-2-thread-1] AgentTestScenario - Set PUBLISH message expiry interval 3600
[INFO ] 2023-07-12 17:22:09.770 [pool-2-thread-1] AgentTestScenario - Set PUBLISH response topic /thing1/response/topic
[INFO ] 2023-07-12 17:22:09.773 [pool-2-thread-1] AgentTestScenario - Set PUBLISH correlation data <ByteString@14fe3672 size=16 contents="correlation_data">
[INFO ] 2023-07-12 17:22:09.773 [pool-2-thread-1] AgentTestScenario - Set PUBLISH user property: region, US
[INFO ] 2023-07-12 17:22:09.773 [pool-2-thread-1] AgentTestScenario - Set PUBLISH user property: type, JSON
[INFO ] 2023-07-12 17:22:09.774 [pool-2-thread-1] AgentTestScenario - Set PUBLISH content type text/plain; charset=utf-8
[INFO ] 2023-07-12 17:22:09.777 [pool-2-thread-1] AgentControlImpl - PublishMqtt: publishing on connectionId 0 topic test/topic
[INFO ] 2023-07-12 17:22:09.868 [pool-2-thread-1] AgentTestScenario - Published connectionId 0 reason code 0 reason string ''
[INFO ] 2023-07-12 17:22:09.873 [grpc-default-executor-0] GRPCDiscoveryServer - OnReceiveMessage: agentId python-paho-agent connectionId 0 topic test/topic QoS 0
[INFO ] 2023-07-12 17:22:09.874 [grpc-default-executor-0] AgentTestScenario - Message received on agentId python-paho-agent connectionId 0 topic test/topic QoS 0 content <ByteString@61afa2ce size=12 contents="Hello World!">
[INFO ] 2023-07-12 17:22:09.874 [grpc-default-executor-0] AgentTestScenario - Message has payload format indicator true
[INFO ] 2023-07-12 17:22:09.876 [grpc-default-executor-0] AgentTestScenario - Message has message expiry interval 3599
[INFO ] 2023-07-12 17:22:09.880 [grpc-default-executor-0] AgentTestScenario - Message has response topic /thing1/response/topic
[INFO ] 2023-07-12 17:22:09.882 [grpc-default-executor-0] AgentTestScenario - Message has correlation data <ByteString@5975763b size=16 contents="correlation_data">
[INFO ] 2023-07-12 17:22:09.884 [grpc-default-executor-0] AgentTestScenario - Message has user property key region value US
[INFO ] 2023-07-12 17:22:09.885 [grpc-default-executor-0] AgentTestScenario - Message has user property key type value JSON
[INFO ] 2023-07-12 17:22:09.885 [grpc-default-executor-0] AgentTestScenario - Message has content type 'text/plain; charset=utf-8'
[INFO ] 2023-07-12 17:22:14.868 [pool-2-thread-1] AgentTestScenario - Set UNSUBSCRIBE user property: region, US
[INFO ] 2023-07-12 17:22:14.868 [pool-2-thread-1] AgentTestScenario - Set UNSUBSCRIBE user property: type, JSON
[INFO ] 2023-07-12 17:22:14.872 [pool-2-thread-1] AgentControlImpl - UnsubscribeMqtt: unsubscribe on connectionId 0
[INFO ] 2023-07-12 17:22:14.980 [pool-2-thread-1] AgentTestScenario - Unsubscribe response: connectionId 0 reason codes [0] reason string ''
[INFO ] 2023-07-12 17:22:24.980 [pool-2-thread-1] AgentTestScenario - Set DISCONNECT user property: region, US
[INFO ] 2023-07-12 17:22:24.980 [pool-2-thread-1] AgentTestScenario - Set DISCONNECT user property: type, JSON
[INFO ] 2023-07-12 17:22:25.002 [grpc-default-executor-0] GRPCDiscoveryServer - OnMqttDisconnect: agentId python-paho-agent connectionId 0 disconnect 'reasonCode: 0
' error ''
[INFO ] 2023-07-12 17:22:25.002 [grpc-default-executor-0] AgentTestScenario - MQTT disconnected on agentId python-paho-agent connectionId 0 disconnect 'reasonCode: 0
' error ''
[INFO ] 2023-07-12 17:22:25.009 [pool-2-thread-1] AgentControlImpl - closeMqttConnection: MQTT connectionId 0 closed
[INFO ] 2023-07-12 17:22:25.023 [pool-2-thread-1] AgentControlImpl - shutdown request sent successfully
[INFO ] 2023-07-12 17:22:25.029 [grpc-default-executor-0] GRPCDiscoveryServer - UnregisterAgent: agentId python-paho-agent reason Agent shutdown by OTF request 'That's it.'
[INFO ] 2023-07-12 17:22:25.048 [grpc-default-executor-0] ExampleControl - Agent python-paho-agent is disconnected

Any additional information or context required to review the change:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

aws-greengrass / aws-greengrass-client-device-auth

feat(uat): implement request response features in python paho client #368