lfrancke
commented
2 months ago I'm fairly certain that there is another similar issue already and there is at least one draft PR for Hadoop on this already.
Open dervoeti opened 2 months ago
lfrancke
commented
2 months ago I'm fairly certain that there is another similar issue already and there is at least one draft PR for Hadoop on this already.
xeniape
commented
2 months ago Hive analysis regarding the Hadoop question in the issue description: The components copied into Hive from the Hadoop-Image are already slimmed down by some degree since we are only building part of the Hadoop components https://github.com/stackabletech/docker-images/blob/main/hadoop/Dockerfile#L66
Hive therefore only contains a subset of Hadoop (contents shown with dive):
Someone with experience and product knowledge might slim it further down by removing jars that are not needed from the shown folders (and under each lib folder in there).
With Hive 4.0.0 the component we are using from Hive (standalone-metastore) is split into multiple separate parts: metastore-common, metastore-server, and metastore-tools, where - depending on the outcome of tests etc. - we might also only include the metastore-server component, further slimming down the attack surface https://github.com/stackabletech/docker-images/pull/818/files#diff-71bbe6452013d0b0b73eca04f83193cb3ece7f3c58310666cbc66c7a954e115aR40
In case it's needed for later:
razvan
commented
2 months ago HBase
The first attempt to remove unused components focused on Phoenix. After realising that > 50% of CVEs in the HBase image come from the jackson-databind 2.4.0 the focus was shifted to removing this dependency.
This PR #820 removes it from the phoenix-server component and the number of CVEs is reduced from 502 to 229.
CI for PR #820 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/hbase-operator-it-custom/
razvan
commented
2 months ago HBase - replace htrace with the noop version
Looked into replacing the htrace dependency with it's no-op version as done in the Omid image by @soenkeliebau . Htrace 3.5.0 brings in the offendingjackson-databind as a transitive dependency into Phoenix.
Unfortunately it is not possible to replace htrace 3.5.0 with htrace-noop 3.5.0 . The noop version is only a drop in replacement for htrace-core4 :(
On the upside, Phoenix will hopefully replace htrace with opentelemetry soon: https://github.com/apache/phoenix/pull/1282
We might have some potential to slim down product images. This can reduce build time, image size and attack surface. For example, the Hive Dockerfile has a comment about Hadoop: https://github.com/stackabletech/docker-images/blob/1965d50dc436552d4c7e06363f4b1ed46deac29c/hive/Dockerfile#L102
Now that we build from source, it might be worth digging into the build processes to: a) Limit which components we build. It doesn't make sense to build stuff that's never copied to the final image. b) Revalidate if all the components that are copied into the final image are really needed in production. With Hive, for example, we switched the build to only build the metastore, which significantly reduced the attack surface. Some products consist of multiple components and plugins, which might not all be needed to run the platform. c) While we're at it, try to generate an SBOM for each component that is copied into the final image (next to the component itself). For most components that should already be the case, see https://github.com/stackabletech/docker-images/pull/814
We want to focus on products that are mostly affected by vulnerabilities right now:
Acceptance criteria: