How can I make the "mqtt_demo_mutual_auth" to integrate with custom authentication?

[leozzmc]

Hi experts!

I try to follow the AWS tutorial to implement the mqtt_mutal_tls and it works successfully.

https://docs.aws.amazon.com/iot/latest/developerguide/iot-embedded-c-sdk.html

Screenshot for successfully with mutual TLS

However when i want to try custom authentication with username/password with this SDK, I was facing problems.

My Setting

• Configure a lambda authorizer witht the Nodejs sample code in AWS official documents

  I've test this Lambda authorizer, which works well ! command for testing:

  aws iot test-invoke-authorizer --authorizer-name LambdaAuth \
  --mqtt-context '{"username":"kevin", "password": "dGVzdA==", "clientId":"testclient"}'

• modify the ~/aws-iot-device-sdk-embedded-c/demos/mqtt/mqtt_demo_mutual_auth/demo-config.h

/**
 * @brief AWS IoT MQTT broker port number.
 *
 * In general, port 8883 is for secured MQTT connections.
 *
 * @note Port 443 requires use of the ALPN TLS extension with the ALPN protocol
 * name. When using port 8883, ALPN is not required.
 */
#ifndef AWS_MQTT_PORT
    #define AWS_MQTT_PORT    ( 443 )
#endif

/**
 * @brief The username value for authenticating client to MQTT broker when
 * username/password based client authentication is used.
 *
 * Refer to the AWS IoT documentation below for details regarding client
 * authentication with a username and password.
 * https://docs.aws.amazon.com/iot/latest/developerguide/custom-authentication.html
 * As mentioned in the link above, an authorizer setup needs to be done to use
 * username/password based client authentication.
 *
 * @note AWS IoT message broker requires either a set of client certificate/private key
 * or username/password to authenticate the client. If this config is defined,
 * the username and password will be used instead of the client certificate and
 * private key for client authentication.
 */
 #define CLIENT_USERNAME    "kevin"

/**
 * @brief The password value for authenticating client to MQTT broker when
 * username/password based client authentication is used.
 *
 * Refer to the AWS IoT documentation below for details regarding client
 * authentication with a username and password.
 * https://docs.aws.amazon.com/iot/latest/developerguide/custom-authentication.html
 * As mentioned in the link above, an authorizer setup needs to be done to use
 * username/password based client authentication.
 *
 * @note AWS IoT message broker requires either a set of client certificate/private key
 * or username/password to authenticate the client.
 */
 #define CLIENT_PASSWORD    "test"

• Generate makefile with solely cmake command, and compile the files with make command.

• The I execute the excutable

./mqtt_demo_mutual_demo

The results shows that the MQTT connection failed

My HOST

• EC2, t2.micro
• AMI: debian-12-amd64-20230711-1438

It there any guidance?

[chinglee-iot]

Hi @leozzmc, I don't have experience with custom authentication demo so probably it will take some time to solve the problem with you. At the same time, I will also consult experienced member.

This first problem I saw from your log is that openSSL is not able to receive data. The reason is when CLIENT_USERNAME is defined, client cert and private key setup will be skipped in the demo. Client cert and private key are still required to connect to AWS IoT core. https://github.com/aws/aws-iot-device-sdk-embedded-C/blob/af6b6efc09d98dea8dadc3db1e2992b593abb104/demos/mqtt/mqtt_demo_mutual_auth/mqtt_demo_mutual_auth.c#L629-L632

Can you give the following fix a try and feedback the result?

  // #ifndef CLIENT_USERNAME remove the preprocessor condition
     opensslCredentials.pClientCertPath = CLIENT_CERT_PATH; 
     opensslCredentials.pPrivateKeyPath = CLIENT_PRIVATE_KEY_PATH; 
 // #endif 

[zhengasif]

hi
I meet the same issue and same error msgs can't be resolved now. And besides ,I configured demo-config.h ,I configure lambda function in my iot console also ,but it does't work yet, I can't figure the reasons out. I want to use USERNAME/PASSWORD to log in mqtt server,so CLIENT_CERT_PATH and CLIENT_PRIVATE_KEY_PATH will not provide in my device.

[chinglee-iot]

@zhengasif Thank you for your feedback. We will look into this problem and reply in this thread again.

[chinglee-iot]

@zhengasif I am able to run the mqtt_demo_mutual_auth with custom authentication. Sorry for my previous post. The client cert and private key are not required for custom authentication.

The are couple of things we can check first:

1. The lambda function to be triggered by AWS IoT Core authorizer. This can be checked with the test in lambda.
2. The AWS IoT Core authorizer and grant the AWS IoT Core service permission to invoke the lambda function

After we create the lambda function and the authorizer, we can check the result with test-invoke-authorizer

aws iot test-invoke-authorizer --authorizer-name NAME_OF_AUTHORIZER ...

The json returned by the lambda function should be printed out in your console.

3. Ensure the json contains the required attribute and the policy document in json has the permission for the demo. iot:Connect, iot:Publish, iot:Subscribe and iot:Receive actions are required for this demo.

4. Configure the demo with the following in demo_config.h and run the demo.

   #define AWS_IOT_ENDPOINT         "YourAWSIoTEndpoint"
   #define AWS_MQTT_PORT             ( 443 )
   #define CLIENT_USERNAME           "The client username"
   #define CLIENT_PASSWORD           "The password"

   Now you should be able to connect/subscribe/publish/receive to the AWS IoT Core with the demo.

More detail information can be found in Creating and managing custom authorizers document to run this demo with custom authentication.

Please feedback which steps you are having problem.

[chinglee-iot]

@leozzmc

It looks like you are able to invoke authorizer. Please check the followings and feedback more information:

• AWS IoT Core clouldwatch log. Cloudwatch logs the event of AWS IoT Core. You can reference the Monitor AWS IoT using CloudWatch Logs document to view the log. The log shows the authentication status which can help diagnosis the problem.
• The JSON document returned by your lambda function should contains required value and permission in policy documents. iot:Connect, iot:Publish, iot:Subscribe and iot:Receive actions are required for this demo.

[leozzmc]

Hi @chinglee-iot ,

I can pass the username/password authentication by using 3rd party mqtt client - MQTTX. I only specify

• The name of this connection
• IoT endpoints
• port
• username
• password

Then there are invocations in the Lambda log and AWS IoT Core metrics.

The message can also received by AWS IoT Core.

However, with the mqtt-demo-mutual-auth file in the EC2, the connection still failed.

Here's the Lambda code

// A simple Lambda function for an authorizer. It demonstrates
// How to parse a CLI and Http password to generate a response.

exports.handler = async (event, context, callback) => {

    //Http parameter to initiate allow/deny request
    const HTTP_PARAM_NAME='actionToken';
    const ALLOW_ACTION = 'Allow';
    const DENY_ACTION = 'Deny';

    //Event data passed to Lambda function
    var event_str = JSON.stringify(event);
    console.log('Complete event :'+ event_str);

    //Read protocolData from the event json passed to Lambda function
    var protocolData = event.protocolData;
    console.log('protocolData value---> ' + protocolData);

    //Get the dynamic account ID from function's ARN to be used
    // as full resource for IAM policy
    var ACCOUNT_ID = context.invokedFunctionArn.split(":")[4];
    console.log("ACCOUNT_ID---"+ACCOUNT_ID);

    //Get the dynamic region from function's ARN to be used
    // as full resource for IAM policy
    var REGION = context.invokedFunctionArn.split(":")[3];
    console.log("REGION---"+REGION);

    //protocolData data will be undefined if testing is done via CLI.
    // This will help to test the set up.
    if (protocolData === undefined) {

        //If CLI testing, pass deny action as this is for testing purpose only.
        console.log('Using the test-invoke-authorizer cli for testing only');
        callback(null, generateAuthResponse(DENY_ACTION,ACCOUNT_ID,REGION));

    } else{

        var uname = event.protocolData.mqtt.username;
        var pwd = event.protocolData.mqtt.password;
        var buff = new Buffer(pwd, 'base64');
        var passwd = buff.toString('ascii');
        console.log("pwd: " +pwd);
        switch (passwd) { 
            case 'test': 
                console.log("password: "+passwd);
                callback(null, generateAuthResponse(ALLOW_ACTION, ACCOUNT_ID, REGION)); 
            default: 
              console.log("password: "+passwd);
              callback(null, generateAuthResponse(DENY_ACTION,ACCOUNT_ID,REGION));  
        }

    }

};

// Helper function to generate the authorization IAM response.
var generateAuthResponse = function(effect,ACCOUNT_ID,REGION) {

    var full_resource = "arn:aws:iot:"+ REGION + ":" + ACCOUNT_ID + ":*";
    console.log("full_resource---"+full_resource);

    var authResponse = {};
    authResponse.isAuthenticated = true;
    authResponse.principalId = 'principalId';

    var policyDocument = {};
    policyDocument.Version = '2012-10-17';
    policyDocument.Statement = [];

    var conncectstatement = {};
    var subscribestatement = {};
    var publishstatement = {};
    var receivestatement = {};

    //connect
    conncectstatement.Action = 'iot:Connect';
    conncectstatement.Effect = effect;
    conncectstatement.Resource = full_resource;
    //Subscribe
    subscribestatement.Action = 'iot:Subscribe';
    subscribestatement.Effect = effect;
    subscribestatement.Resource = full_resource;
    // Publish
    publishstatement.Action = 'iot:Publish';
    publishstatement.Effect = effect;
    publishstatement.Resource = full_resource;
    //Receive
    receivestatement.Action = 'iot:Receive';
    receivestatement.Effect = effect;
    receivestatement.Resource = full_resource;

    policyDocument.Statement[0] = conncectstatement;
    policyDocument.Statement[1] = subscribestatement;
    policyDocument.Statement[2] = publishstatement;
    policyDocument.Statement[3] = receivestatement;

    authResponse.policyDocuments = [policyDocument];
    authResponse.disconnectAfterInSeconds = 3600;
    authResponse.refreshAfterInSeconds = 600;

    console.log('custom auth policy function called from http');
    console.log('authResponse --> ' + JSON.stringify(authResponse));
    console.log(authResponse.policyDocuments[0]);

    return authResponse;
}

After executing these command, the policy document will returned, and this can also confirm by checking the Lambda log streams.

aws iot test-invoke-authorizer --authorizer-name TestAuth2 \
--mqtt-context '{"username":"kevin", "password": "dGVzdA==", "clientId":"testclient"}'

policy document

The point is, the same Lambda code and the same configuration in both MQTTX and demo-config.h.

Why the script still failed with connection problems?

And I confirm that the EC2 that I use is in public subnet. It seems that the policy Lambda return may give my mqtt client enough permission to publish and receive mqtt messages.

[chinglee-iot]

@leozzmc

Can you help to check the followings in cloudwatch and feedback the result?

1. Do you see any of the following log in AWSIotLogsV2 group?

   {
   "timestamp": "2023-10-13 18:00:57.244",
   "logLevel": "ERROR",
   "traceId": "6475c73d-d3be-7d1a-b447-843c46917d88",
   "accountId": "......",
   "status": "Failure",
   "eventType": "Connect",
   "protocol": "MQTT",
   "clientId": "testclient",
   "principalId": "xxxxxxxx",
   "sourceIp": ".....",
   "sourcePort": 27487,
   "reason": "AUTHORIZATION_FAILURE",
   "details": "Authorization Failure"
   }

2. Is your lambda triggered when this demo application trying to connect to the MQTT? The basic lambda permission allows this lambda to create log in cloudwatch. If the lambda function is triggered, there will be cloudwatch log.
3. Does the policy document returned by your lambda function contain valid permission?

I am able to run your lambda function with invalid password. The following is the cloudwatch log. I didn't try valid password with base64 encoding. I have not problem connect to the IoT core if I update your lambda to allow any password.

2023-10-16T10:51:20.841Z    .....   INFO    authResponse --> 
{
    "isAuthenticated": true,
    "principalId": "principalId",
    "policyDocuments": [
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "iot:Connect",
                    "Effect": "Deny",
                    "Resource": "arn:aws:iot:us-west-2:....:*"
                },
                {
                    "Action": "iot:Subscribe",
                    "Effect": "Deny",
                    "Resource": "arn:aws:iot:us-west-2:.....:*"
                },
                {
                    "Action": "iot:Publish",
                    "Effect": "Deny",
                    "Resource": "arn:aws:iot:us-west-2:.....:*"
                },
                {
                    "Action": "iot:Receive",
                    "Effect": "Deny",
                    "Resource": "arn:aws:iot:us-west-2:....:*"
                }
            ]
        }
    ],
    "disconnectAfterInSeconds": 3600,
    "refreshAfterInSeconds": 600
}

[chinglee-iot]

@leozzmc In my test, I use the default authorizer with set-default-authorizer.

We are also trying to set the custom authorizer in username and will reply the test result in this thread later.

#define CLIENT_USERNAME "Test?x-amz-customauthorizer-name=TestAuth2"

( edited for typo )

[chinglee-iot]

@leozzmc In the demo, the metrics parameter is already appended to the CLIENT_USERNAME_WITH_METRICS.

#define METRICS_STRING             "?SDK=" OS_NAME "&Version=" OS_VERSION "&Platform=" HARDWARE_PLATFORM_NAME "&MQTTLib=" MQTT_LIB

#define CLIENT_USERNAME_WITH_METRICS    CLIENT_USERNAME METRICS_STRING

Therefore, adding extra parameters in CLIENT_USERNAME would result in invalid format.

For example, with previous example

#define CLIENT_USERNAME "Test?x-amz-customauthorizer-name=TestAuth2"

The resulting string will have two question marks, which is invalid format.

Test?x-amz-customauthorizer-name=TestAuth2?SDK=" OS_NAME "&Version=" OS_VERSION "&Platform=" HARDWARE_PLATFORM_NAME "&MQTTLib=" MQTT_LIB
                                          ^ The extra question mark

We will bring back this problem for discussion and come out a solution to allow user to append parameters in CLIENT_USERNAME. To unblock your development, you may use default authorizer or update the METRICS_STRING with the following code as a temporary workaround.

#define METRICS_STRING               "&SDK=" OS_NAME "&Version=" OS_VERSION "&Platform=" HARDWARE_PLATFORM_NAME "&MQTTLib=" MQTT_LIB
                                      ^ Replace '?' with '&'

[chinglee-iot]

The PR #1893 to fix this issue is merged. We will close this issue. Thank you for creating this issue and feel free to reopen it if any new observation.