AWS SDK 2 plans?

[dbarvitsky]

We are considering the KCL and KPL as a replacement for the homegrown code. I noticed that KCL is now on AWS SDK2 (they have 1.x version, but it seems that 2.x is mainstream), and KPL is on AWS SDK 1.

I am aware of ability to use SDK1 and SDK2 together, but would much rather avoid this option if at all possible.

Question: is KPL planning a new branch with SDK2 dependency as KCL did any time soon? What is the general recommendation for those who want to use both KCL and KPL in the same project?

Thanks in advance!

[Bolshem]

Hello team! Do you have any updates on this?

[code4dc]

Is there an official timeline from the team on this?

[dbarvitsky]

Update from my side: it appears that the homegrown solution is now unlikely to be replaced with KPL. It has grown into many micro-services throughout the company and people are now used to it. We didn't have a single request to move to KPL in a year, and ended up adding a few features like durable publishing, custom enveloping and fine-grained telemetry through Micrometer and NewRelic. So it looks like there is no turning back for us now.

Having said that, and considering that 19 people care about this at the time of writing, would it make sense for us to open-source our library?

[code4dc]

Would definitely be worth it for us especially if you are on AWS SDK v2! Right now it seems that KPL is more trouble than it's worth.

[dbarvitsky]

@code4dc We might or might no clear the bar of the "more trouble than it's worth" too... Ours is purpose built to fit into our architecture and is not a drop-in KPL replacement by any means.

I'd probably do a demo first to see if it is worth it, since releasing the source takes considerable amount of hoop jumping. I don't mind getting behind this, as long as community can benefit from it. My profile has a link to my CV site with contacts in it, please reach out directly.

[jdcaperon]

My guess would be that upgrading to 2.x would likely happen in line with 1.x being moved into Maintenance mode given that the KPL's dependency on the Java 1.0 SDK is purely for authentication AFAICT. Given that both SDK's can be run side by side I imagine that is recommended approach.

I would be interested to hear about the complexity of creating your own "KPL" one-day @dbarvitsky even in a blog post or presentation somewhere.

[dbarvitsky]

@jdcaperon asked my boss about this (and demo as @code4dc has requested). Please reach out directly (my profile has private email) if you are interested, so that we don't confuse people with a tangent discussion.

[softprops]

Is there any update on this?

[dbarvitsky]

@softprops - not from my side (note: we are the folks who rolled up our own library that is SDK2 based, I am not related to AWS or AWS labs in any way). Still happy to do a demo / donate the library if community is interested. I asked for concerned citizens to reach out directly earlier, but didn't get much traction. We continue to use our own library in prod.

[jdcaperon]

When other fires aren't burning one day I will take you up on the offer!

[softprops]

For what it’s worth, as java itself is evolving there are no plans to evolve sdk v1 with it. All of those efforts are going into sdk v2 moving forward https://github.com/aws/aws-sdk-java/issues/1603#issuecomment-540270923

[barryoneill]

I'm a little unclear - Is this project going to remain on the V1 SDK indefinitely? (also unsure if anyone in this thread is an AWS representative)

[junyuc25]

Hi, in 2024, the AWS SDK v1 will transition into maintenance mode where only critical updates will be issued. The process of full deprecation is slated to begin in early 2025. Is there any updates on the migration timelines?

[JuanDGiraldoM]

We did some test to compare with SDK v2 performance, and show it:

Scenario:

• EKS cluster (k8s)
• Spring boot reactive microservices (Spring webflux)
• KPL v. 0.15.7
• SDK v2 (bom 2.20.128)
• Kinesis Data Stream with 100 shards
• Data of ~3KB per request
• Performance Analyzer (up to 1.000 concurrent users)

• With just one pod, and 3 cores

SDK get's better performance at startup, and uses all cpu. After total cpu uses, begins to degrade. Moreover, KPL works better because batch

• With HPA, 1.5 cores up to 10 pods

It was so difficult to scale KPL because of its low cpu consumption, and the scale doesn't improve the performance

And KPL has 3 vulnerabilities: CVE-2023-2976 (score 7.1): Google guava 1.0 to 31.1 https://nvd.nist.gov/vuln/detail/CVE-2023-2976

•CVE-2020-8908 (score 3.3): Google guava, all versions https://nvd.nist.gov/vuln/detail/CVE-2020-8908

•CWE-674 - UNCONTROLLED RECURSION (score 7.5): Apache Commons-Collections  < 4.3 https://devhub.checkmarx.com/cve-details/Cx78f40514-81ff/?utm_source=jetbrains&utm_medium=referral&utm_campaign=idea

They're different purposes (KPL for batch and SDK for real time) If you try KPL with buffer time to 10s, for near real time

The behaviour is similar to SDK

So, I don't have more criteria to use KPL for specific batch use case (that you can also do with SDK) and the code implementation is cleaner. With KPL you need to use Callbacks and Futures, while SDK just use natural reactive flow.

[barryoneill]

Hi - is this project dead? AWS itself has announced EOL for the V1 SDK.

Others are hitting issues with STS conflicts (same as myself) when the producer is present in their application - https://github.com/awslabs/amazon-kinesis-producer/issues/557