Client re-enqueue a packet that causes PAYLOAD_LIMIT_EXCEEDED disconnect event.

chapost1 commented 1 month ago

Describe the bug

Client keeps retrying publish message (re-enqueue) post disconnect, which it's cause is PAYLOAD_LIMIT_EXCEEDED. which cause infinite connect/disconnect events.

Expected Behavior

Result with a puback that indicates a failure and not re-enqueue the message as it's payload won't shrink.

Current Behavior

Client attempts to publish a message with payload larger than AWS max payload size.
Client disconnects.
Client re-connects.
Client re-enqueue packet, and try to send it.
Client disconnects.
Client re-connects.
Client re-enqueue packet, and try to send it.
...

No PUBACK response for the publish method.

Reproduction Steps

Just create a client instance that connects post disconnect events, and attempt to publish a message with payload larger than allowed size.

Possible Solution

Not sure if feasible, because if no puback is returned, then maybe it's on the iot-core service side also, but I think the solution should behave with resulting with failure puback or at least not re-enqueue as it will not be resolved by itself.

Additional Information/Context

No response

SDK version used

1.20.0

Environment details (OS name and version, etc.)

ubuntu:22.04

jmklix commented 1 month ago

~~Can you try using the MQTT5 client? It will handle the PAYLOAD_LIMIT_EXCEEDED, because part of the updated protocol included the server communicating that the payload limit is.~~ The MQTT5 client is still broken, as mentioned in this comment.

chapost1 commented 1 month ago

Can you try using the MQTT5 client? It will handle the PAYLOAD_LIMIT_EXCEEDED, because part of the updated protocol included the server communicating that the payload limit is.

Already using MQTT5 client

I've tested multiple cases now and:

payload with the size of bytes >= 149482 results with the client to actually catch it and throw an error
payload with the size of bytes <= 131072 (128KB) results with no error and no disconnect event, and reaches to the broker safely
payload with the size of bytes >= 131073 (128KB + 1) and <= 149481 results in infinite connect and disconnect events due to the fact that the client keeps re-enqueue the message.

terminal logs

node index.js 
Connack: Connection Success event {
  connack: {
    type: 2,
    sessionPresent: false,
    reasonCode: 0,
    receiveMaximum: 100,
    maximumQos: 1,
    retainAvailable: true,
    maximumPacketSize: 149504,
    topicAliasMaximum: 8,
    userProperties: [],
    wildcardSubscriptionsAvailable: true,
    subscriptionIdentifiersAvailable: false,
    sharedSubscriptionsAvailable: true,
    serverKeepAlive: 120
  },
  settings: {
    maximumQos: 1,
    sessionExpiryInterval: 1200,
    receiveMaximumFromServer: 100,
    maximumPacketSizeToServer: 149504,
    topicAliasMaximumToServer: 8,
    topicAliasMaximumToClient: 0,
    serverKeepAlive: 120,
    retainAvailable: true,
    wildcardSubscriptionsAvailable: true,
    subscriptionIdentifiersAvailable: false,
    sharedSubscriptionsAvailable: true,
    rejoinedSession: false,
    clientId: 'test-issue-535_c4d0021a-96d0-4f43-a321-5d328f15fcb8'
  }
}
Connection established
Disconnection event {
  error: 'Error: libaws-c-mqtt: AWS_ERROR_MQTT5_DISCONNECT_RECEIVED, Mqtt5 client connection interrupted by server DISCONNECT.',
  disconnect: {
    type: 14,
    reasonCode: 151,
    reasonString: 'DISCONNECT:Packet payload is larger than the account limit:f0fbc633-ae95-43bd-9ec6-c18fca17d591',
    userProperties: []
  }
}
Connack: Connection Success event {
  connack: {
    type: 2,
    sessionPresent: false,
    reasonCode: 0,
    receiveMaximum: 100,
    maximumQos: 1,
    retainAvailable: true,
    maximumPacketSize: 149504,
    topicAliasMaximum: 8,
    userProperties: [],
    wildcardSubscriptionsAvailable: true,
    subscriptionIdentifiersAvailable: false,
    sharedSubscriptionsAvailable: true,
    serverKeepAlive: 120
  },
  settings: {
    maximumQos: 1,
    sessionExpiryInterval: 1200,
    receiveMaximumFromServer: 100,
    maximumPacketSizeToServer: 149504,
    topicAliasMaximumToServer: 8,
    topicAliasMaximumToClient: 0,
    serverKeepAlive: 120,
    retainAvailable: true,
    wildcardSubscriptionsAvailable: true,
    subscriptionIdentifiersAvailable: false,
    sharedSubscriptionsAvailable: true,
    rejoinedSession: false,
    clientId: 'test-issue-535_c4d0021a-96d0-4f43-a321-5d328f15fcb8'
  }
}
Disconnection event {
  error: 'Error: libaws-c-mqtt: AWS_ERROR_MQTT5_DISCONNECT_RECEIVED, Mqtt5 client connection interrupted by server DISCONNECT.',
  disconnect: {
    type: 14,
    reasonCode: 151,
    reasonString: 'DISCONNECT:Packet payload is larger than the account limit:c245072b-0f7f-4af9-aa14-2943c023b4b3',
    userProperties: []
  }
}
Connack: Connection Success event {
  connack: {
    type: 2,
    sessionPresent: false,
    reasonCode: 0,
    receiveMaximum: 100,
    maximumQos: 1,
    retainAvailable: true,
    maximumPacketSize: 149504,
    topicAliasMaximum: 8,
    userProperties: [],
    wildcardSubscriptionsAvailable: true,
    subscriptionIdentifiersAvailable: false,
    sharedSubscriptionsAvailable: true,
    serverKeepAlive: 120
  },
  settings: {
    maximumQos: 1,
    sessionExpiryInterval: 1200,
    receiveMaximumFromServer: 100,
    maximumPacketSizeToServer: 149504,
    topicAliasMaximumToServer: 8,
    topicAliasMaximumToClient: 0,
    serverKeepAlive: 120,
    retainAvailable: true,
    wildcardSubscriptionsAvailable: true,
    subscriptionIdentifiersAvailable: false,
    sharedSubscriptionsAvailable: true,
    rejoinedSession: false,
    clientId: 'test-issue-535_c4d0021a-96d0-4f43-a321-5d328f15fcb8'
  }
}
Disconnection event {
  error: 'Error: libaws-c-mqtt: AWS_ERROR_MQTT5_DISCONNECT_RECEIVED, Mqtt5 client connection interrupted by server DISCONNECT.',
  disconnect: {
    type: 14,
    reasonCode: 151,
    reasonString: 'DISCONNECT:Packet payload is larger than the account limit:0d43adad-2e29-4700-935d-7558fc3c6f15',
    userProperties: []
  }
}
^C

code snippet that re-produces the issue.

const iotsdk = require('aws-iot-device-sdk-v2')
const iot = iotsdk.iot
const mqtt5 = iotsdk.mqtt5
const fs = require('node:fs')
const events = require('node:events')
const crypto = require('node:crypto')
const { scheduler } = require('node:timers/promises')

const getClient = () => {
    const port = 8883
    const clientId = `test-issue-535_${crypto.randomUUID()}`

    const cert = fs.readFileSync('./cert.crt')
    const privateKey = fs.readFileSync('./key.pem')
    const endpoint = fs.readFileSync('./endpoint.txt', 'utf-8').trim()

    const config = iot.AwsIotMqtt5ClientConfigBuilder.newDirectMqttBuilderWithMtlsFromMemory(
        endpoint,
        cert,
        privateKey
    )
        .withPort(port)
        .withConnectProperties({
            clientId,
            keepAliveIntervalSeconds: 120,
            sessionExpiryIntervalSeconds: 1200,
        })
        .withSessionBehavior(mqtt5.ClientSessionBehavior.Clean)
        .build()

    const client = new mqtt5.Mqtt5Client(config);

    return client
}

const connect = async (client) => {
    // makes sure process won't die during first connection attempt
    const timer = setInterval(() => { }, 60 * 1000)
    // Allow node to die if the promise above resolved
    clearKeepAliveInterval = () => clearInterval(timer)
    // set up event handlers      
    client.on('error', (error) => {
        console.error("Error event: " + error.toString());
    });

    client.on('connectionSuccess', (eventData) => {
        console.log("Connack: Connection Success event", {
            connack: eventData.connack,
            settings: eventData.settings
        });
    });

    client.on('disconnection', (eventData) => {
        console.log("Disconnection event", {
            error: eventData.error.toString(),
            disconnect: eventData.disconnect || null,
        });
    });

    const connectionSuccess = events.once(client, "connectionSuccess");

    client.start()
    // connect
    await connectionSuccess

    console.log("Connection established")

    return client
}

const run = async () => {
    const client = getClient()
    await connect(client)

    // publish a large payload
    const payload = Buffer.alloc(128*1024 + 1, 'a')

    await client.publish({
        topicName: 'test-issue-535',
        payload: payload,
        qos: mqtt5.QoS.AtLeastOnce,
        retain: false
    })

    await scheduler.wait(1000)
}

run().catch((error) => {
    console.error("An error occurred:", error)
    process.exit(1)
})

bretambrose commented 1 month ago

This is an IoT Core bug that I reported a year ago. Will follow up and see why it has not been addressed yet.

jmklix commented 1 month ago

P162538649

aws / aws-iot-device-sdk-js-v2