search

awslabs / eventbridge-kafka-connector

Kafka sink connector for Amazon EventBridge to send events (records) from Kafka topic(s) to the specified EventBridge event bus
https://aws.amazon.com/eventbridge/resources
Apache License 2.0
61 stars 5 forks source link

Leverage default profile supplier for credential reload #359

Closed maschnetwork closed 3 months ago

maschnetwork commented 3 months ago

Description

Leverage Default Profile Supplier for credentials reload as in: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-profiles.html#profile-reloading

Test Steps

TBD

Checklist:

Related Issue

357

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

maschnetwork commented 3 months ago

@embano1 verified locally by mounting credentials file in docker-compose setup via:

- /Users/X/.aws/credentials:/.aws/credentials

Enabled AWS debug logs in config-log4j.properties via:

log4j.logger.software.amazon.awssdk=DEBUG

Modified credentials, issued another request and got:

[2024-08-12 21:00:48,340] DEBUG [eventbridge-e2e|task-0] (ProfileFileSupplier()) Cached value is stale and will be refreshed. (software.amazon.awssdk.utils.cache.CachedSupplier:85)
connect-1     | [2024-08-12 21:00:48,341] DEBUG [eventbridge-e2e|task-0] (ProfileFileSupplier()) Refreshing cached value. (software.amazon.awssdk.utils.cache.CachedSupplier:85)
connect-1     | [2024-08-12 21:00:48,345] WARN [eventbridge-e2e|task-0] (ProfileFileSupplier()) Retrieved value expiration is in the past (2024-08-12T19:00:48.341653877Z). Using expiration of 2024-08-12T19:00:49.345011033Z (software.amazon.awssdk.utils.cache.CachedSupplier:106)
connect-1     | [2024-08-12 21:00:48,346] DEBUG [eventbridge-e2e|task-0] (ProfileFileSupplier()) Successfully refreshed cached value. Next Prefetch Time: +1000000000-12-31T23:59:59.999999999Z. Next Stale Time: 2024-08-12T19:00:49.345011033Z (software.amazon.awssdk.utils.cache.CachedSupplier:85)

Also verified by replacing with invalid credentials and ran into connector error for invalid token during next request.

embano1 commented 3 months ago

Thx a ton!

Regarding

WARN [eventbridge-e2e|task-0] (ProfileFileSupplier()) Retrieved value expiration is in the past (2024-08-12T19:00:48.341653877Z). Using expiration of 2024-08-12T19:00:49.345011033Z (software.amazon.awssdk.utils.cache.CachedSupplier:106)

Is that a concern and does it happen every time the credentials are refreshed? Or just particular to your environment?

embano1 commented 3 months ago

cc/ @ryancuk first fix work in progress, feel free (or your team) to take a look and provide feedback

maschnetwork commented 3 months ago

Is that a concern and does it happen every time the credentials are refreshed? Or just particular to your environment?

It always happens but might be related to the way how the credential file is mounted. With my setup it always keeps the timestamp when it was mounted - even if you change the file. However, I didn't experience any side affects of this approach and the SDK includes extensive test coverage on that behavior.

ryancuk commented 3 months ago

cc/ @ryancuk first fix work in progress, feel free (or your team) to take a look and provide feedback

Thanks very much for the fix. It looks good to me.

embano1 commented 3 months ago

Thx @maschnetwork sounds like this one is good to moved to a ready PR?

maschnetwork commented 3 months ago

@embano1 moved to PR

embano1 commented 3 months ago

LGTM, do you want to squash your commits before I merge?

maschnetwork commented 3 months ago

done

ryancuk commented 3 months ago

hi @embano1, please let me know when the new release is ready for use.

embano1 commented 3 months ago

@ryancuk do you need a Confluent connector release on Confluent hub or are the artifacts here on Github as part of the release sufficient?

ryancuk commented 3 months ago

@embano1, yes we source the connector from confluent. Appreciate for the great help!

embano1 commented 3 months ago

I can cut you a release here and you get all the artifacts, incl. the Confluent zip. There's a manual process which takes couple days for the zip to appear on Confluent Hub. Let me know if that works?

ryancuk commented 3 months ago

@embano1 that works. appreciated!

embano1 commented 3 months ago

Will try to cut a release in the next hours and ping you here.

embano1 commented 3 months ago

@ryancuk new release is here: https://github.com/awslabs/eventbridge-kafka-connector/releases/tag/v1.3.1

The Confluent Hub artifacts are here (zip format): https://github.com/awslabs/eventbridge-kafka-connector/releases/download/v1.3.1/aws-kafka-eventbridge-sink-v1.3.1.zip