KafkaJSNonRetriableError: Cannot change principals during re-authentication

[cion-yatindra]

Describe the bug I have a consumer running on EKS on aws and have set up MSK for kafka, both of them in same VPC. I have used @jm18457/kafkajs-msk-iam-authentication-mechanism library for authentication using IAM roles. I was able to consume messages on EKS, but out of nowhere I got Crash: KafkaJSNonRetriableError: Cannot change principals during re-authentication from IAM.arn:aws:sts::<AppID>:assumed-role/CionEKSPodRole/aws-sdk-js-session-<SessionID1>: IAM.arn:aws:sts::<AppID>:assumed-role/CionEKSPodRole/aws-sdk-js-session-<SessionID2> Error and consumer stopped listening messages, while i am still able to produce messages on other end.

To Reproduce

1. Run a producer that continuously produces messages to a topic
2. Run a consumer that subscribes to that topic.
3. After the consumer has consumed few messages in an hour or sometimes after a day it suddenly stops consuming messages with error mentioned in description.

Code snippet for cosumer:

import { createMechanism } from '@jm18457/kafkajs-msk-iam-authentication-mechanism';
import {  Kafka } from 'kafkajs';
const consumeMessages= async () => {
 try{
        const kafka = new Kafka({
        brokers: ['broker1', 'broker2'],
        ssl: true,
        sasl: createMechanism({
            region: 'us-west-2'
        }),
    });
        const groupId = 'test-cluster-id'
        const kafkaConsumer= kafka.consumer({ groupId:  groupId });
        await kafkaConsumer.connect();
         await kafkaConsumer.subscribe({
            topics: ['some-topic']
        });
         await kafkaConsumer.run({
            // eslint-disable-next-line require-await
            eachMessage: async ({ topic, message }) => {
               // Process message
            }
        });
    } catch(err){
        console.log("Failed to connect kafka")
        console.log(err);
    }
}

Expected behavior Consumer shouldn't crash or retry connecting to the broker.

Observed behavior

1. Was able to consume messages.
2. All of a sudden got error: Crash: KafkaJSNonRetriableError: Cannot change principals during re-authentication from IAM.arn:aws:sts::<AppID>:assumed-role/CionEKSPodRole/aws-sdk-js-session-<SessionID1>: IAM.arn:aws:sts::<AppID>:assumed-role/CionEKSPodRole/aws-sdk-js-session-<SessionID2>

Environment:

• OS: Ubuntu 22.4
• KafkaJS 2.2.4
• Kafka version 2.8.1
• NodeJS version 16.20.0

[adarshbalu]

{"level":"ERROR","timestamp":"2023-08-11T13:24:37.575Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSNonRetriableError: Cannot change principals during re-authentication from IAM.arn:aws:sts::020260453790:assumed-role/msk-full-iam/aws-sdk-js-session-1691756680572: IAM.arn:aws:sts::020260453790:assumed-role/msk-full-iam/aws-sdk-js-session-1691760272714","groupId":"group","stack":"KafkaJSNonRetriableError: Cannot change principals during re-authentication from IAM.arn:aws:sts::020260453790:assumed-role/msk-full-iam/aws-sdk-js-session-1691756680572: IAM.arn:aws:sts::020260453790:assumed-role/msk-full-iam/aws-sdk-js-session-1691760272714\n at /home/node/app/node_modules/kafkajs/src/retry/index.js:55:18\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)"}

[BDeus]

same issue, does anyone as a retriable configuration ?

[cion-yatindra]

Closing this issue as setting AWS_ROLE_SESSION_NAME in environment variable to any static value resolves the error.

[akospaska]

Closing this issue as setting AWS_ROLE_SESSION_NAME in environment variable to any static value resolves the error.

Is there any detailed explanation why it solves the issue?