Authentication fails after ~ 24-30 hours running successfully

[launchiamenterprise]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Below is my setup. For first few hours it is working as expected. After 24 hours - 30 hours i am getting unauthorized exception. I believe token refresh is not being handled properly

`private constructor () { this.configManager = ConfigurationManager.getInstance(); this.kafka = new Kafka({ clientId: this.configManager.getInitialConfig().managementPlane.activationKey, brokers: this.constructBrokers(), ssl: true, sasl: { mechanism: 'oauthbearer', oauthBearerProvider: () => this.oauthBearerTokenProvider() } }); }

constructBrokers (): string[] { const { kafkaServers } = this.configManager.getInitialConfig().dpActivatedResponseDTO; return kafkaServers?.split(',') || []; }

async oauthBearerTokenProvider (): Promise { const { accessID, accessSecret } = this.configManager.getInitialConfig().dpActivatedResponseDTO; process.env.AWS_ACCESS_KEY_ID = accessID; process.env.AWS_SECRET_ACCESS_KEY = accessSecret; const authTokenResponse = await generateAuthTokenFromCredentialsProvider({ region: 'us-west-2', awsCredentialsProvider: fromNodeProviderChain() }); console.log('authTokenResponse', authTokenResponse) return { value: authTokenResponse.token }; }`

aws-msk-iam-sasl-signer-js library version used

1.0.0

Which Node.js version is this issue in?

18.7.1

Operating System and version

mac

Reproduction Steps

Added snippet

Observed Behavior

Token refresh is not handled properly

Expected Behavior

Token should for ever refresh

Possible Solution

No response

Additional Information/Context

No response

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[sankalpbhatia]

Thanks for reporting. Can you share logs on the errors you are seeing here?

[nmlynch94]

Not 100% sure due to the lack of error logs in the original bug, but this seems similar to what we are seeing. Although, we are seeing it roughly once per week as opposed to every 30 hours.

Everything seems to be working smoothly while authenticating with an MSK Kafka cluster using IAM auth, and then our applications log "InvalidIdentityTokenException: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements"

The line that fails is const authTokenResponse = await generateAuthToken({ region });

const kafkaConfig = {
      ...config,
      sasl: {
        mechanism: 'oauthbearer',
        oauthBearerProvider: () => oauthBearerTokenProvider(config.region as string)
      }
    } 

Revelant snippet from oauthBearerTokenProvider

    log.info('Kafka Auth generating authorization token . . .')
    const authTokenResponse = await generateAuthToken({ region });

    log.info('Kafka Auth token successfully generated.')
    return {
      value: authTokenResponse.token
    }

The application appears to recover after, but worried this could be causing an issue we are not anticipating. I can potentially try to generate some more useful debug logs if any are needed and you direct me to which logs you'd like to see.

I'm wondering if this could be related to https://github.com/jmaver-plume/kafkajs-msk-iam-authentication-mechanism/issues/20. I did not test setting a session name but maybe that would be a good next step.

Thanks for any help!

[nmlynch94]

Following up here, setting session name lowered the frequency, but we have still seen this once or twice in the past month.