search

AutoMQ / automq

AutoMQ is a cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 10x cost-effective. Autoscale in seconds. Single-digit ms latency.
https://www.automq.com/docs
Other
3.57k stars 178 forks source link

[Enhancement] Dependency updates welcome? #1727

Open reneleonhardt opened 1 month ago

reneleonhardt commented 1 month ago

Who is this for and what problem do they have today?

AutoMQ users facing bugs and security vulnerabilities caused by old dependencies and Docker images.

Why is solving this problem impactful?

I guess most code is still coming from Kafka upstream, that would explain the large technical debt (many outdated versions). Updating as many dependencies and Docker images as possible does not only fix security vulnerabilities, but also fixes other bugs and allows using new features. Just look at a current trivy scan of confluentinc/cp-kafka to see how many months they wait until updating a single dependency...

Additional notes

If updates are welcome in this project (for example you updated minJavaVersion 5 months ago) I could provide a first PR with what updates I can find and a Dependabot config to automate this tedious process in the future. Just by looking for a minute through main I saw already dozens of updates (i.e. kafka 3.7.1 and 3.8.0, Guava, JDK, GraalVM, Gradle, Maven, ...).

superhx commented 4 weeks ago

Updating dependencies is not just about updating versions; it is a more complex task that requires additional effort in compatibility adaptation, performance regression, and other preparatory tasks. The dependencies of Apache Kafka upstream have been verified by a large number of users in the community, so AutoMQ currently adopts a dependency version update strategy that follows Apache Kafka upstream.

reneleonhardt commented 4 weeks ago

I see, maybe you want to make these invisible barriers preventing reducing technical debt more transparent by writing tests showing that performance is "100%" now with those old dependencies, so updates that reduce performance can be identified automatically? That all major frameworks are compatible like Spring Boot 3.3 + Spring Kafka 3.2 + Spring Cloud Stream 4.1 and Quarkus 3.13 + SmallRye Reactive Messaging 4.18 and Micronaut 4.6 + Micronaut Kafka 5.5 to automatically show that an dependency update breaks compatibility in one component?

And performance and compatibility which can't be improved by updates outweigh security risks for your users because of years of technical debt? https://www.veracode.com/resources/state-software-security-2024-addressing-threat-security-debt

$ docker pull automqinc/automq:1.1.2-rc0
$ trivy image --scanners vuln automqinc/automq:1.1.2-rc0
automqinc/automq:1.1.2-rc0 (debian 11.3)
========================================
Total: 1451 (UNKNOWN: 0, LOW: 946, MEDIUM: 184, HIGH: 278, CRITICAL: 43)

Updating Debian 11.3 (2.5 years old) to 12.6 (2 months old) would fix hundreds of vulnerabilities alone, leaving Java with 1 easy update fixing 2 vulnerabilities... but it's impossible because netty-codec-http 4.1.108 is slow and incompatible to 4.1.94? 🤔

Library Vulnerability Severity
io.netty:netty-codec-http CVE-2024-29025 MEDIUM
(netty-codec-http-4.1.100.Final.jar)
io.netty:netty-codec-http
(netty-codec-http-4.1.94.Final.jar)
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p HIGH
(netty-codec-http2-4.1.94.Final.jar)