Cannot build kafka image

[samding01]

Following the README instruction to build a Kafka image, but it gets errors like:

# mvn clean package -Pdocker -DskipTests
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent.kafka-images:kafka-images-parent:6.0.5-SNAPSHOT: Could not find artifact io.confluent:common-docker:pom:6.0.5-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ io.confluent.kafka-images:kafka-images-parent:6.0.5-SNAPSHOT, /root/work/kafka-images/pom.xml, line 22, column 13
...

Tried the difference branch (versions), got the similar errors.

Probably pom.xml needs to change, any idea? Thanks,

[andrewegel]

What branch are you building this on? I'd only recommend building from a -post branch or a tag. The .x branches represent the next iteration of development, and are set to use artifacts that are not yet released.

So try this:

git checkout 6.0.5-post
mvn clean package -Ddocker.registry="" -Ddocker.upstream-tag=6.0.5 -DCONFLUENT_PACKAGES_REPO=https://packages.confluent.io/rpm/6.0 -DCONFLUENT_VERSION=6.0.5 -Pdocker -DskipTests

This should build you docker images with CP version 6.0.5.

I realize this isn't a great user experience as none of this is documented in the Readme. We have something in the backlog to address this.

[samding01]

Thank @andrewegel for your info.

Following your instruction, I got these error msg:

# mvn clean package -Ddocker.registry="" -Ddocker.upstream-tag=6.0.5 -DCONFLUENT_PACKAGES_REPO=https://packages.confluent.io/rpm/6.0 -DCONFLUENT_VERSION=6.0.5 -Pdocker -DskipTests
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/io/confluent/common-docker/6.0.5/common-docker-6.0.5.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent.kafka-images:kafka-images-parent:6.0.5: Could not find artifact io.confluent:common-docker:pom:6.0.5 in central (https://repo.maven.apache.org/maven2) and 'parent.relativePath' points at wrong local POM @ line 22, column 13

seems not able to find common-docker-6.0.5.pom

[andrewegel]

You need to have https://packages.confluent.io/maven as a maven repo somewhere in your maven settings.xml.

I can see this artifact there: https://packages.confluent.io/maven/io/confluent/common-docker/6.0.5/

[samding01]

@andrewegel I created a settings.xml under ~/.m2/ and it is able to donwload common-docker-6.0.5.pom, but hit a new errors then

#  mvn clean package -Ddocker.registry="" -Ddocker.upstream-tag=6.0.5 -DCnfluent.io/rpm/6.0 -DCONFLUENT_VERSION=6.0.5 -Pdocker -DskipTests
[INFO] Scanning for projects...
Downloading from common-docker-6.0.5.pom: https://packages.confluent.io/maven/io/confluent/common-docker/6.0.5/common-docker-6.0.5.pom
Downloaded from common-docker-6.0.5.pom: https://packages.confluent.io/maven/io/confluent/common-docker/6.0.5/common-docker-6.0.5.pom (2.8 kB at 459 B/s)
Downloading from common-docker-6.0.5.pom: https://packages.confluent.io/maven/io/confluent/common/6.0.5/common-6.0.5.pom
Downloaded from common-docker-6.0.5.pom: https://packages.confluent.io/maven/io/confluent/common/6.0.5/common-6.0.5.pom (1.3 kB at 20 kB/s)
Downloading from common-docker-6.0.5.pom: https://packages.confluent.io/maven/io/confluent/common-parent/6.0.5/common-parent-6.0.5.pom
Downloaded from common-docker-6.0.5.pom: https://packages.confluent.io/maven/io/confluent/common-parent/6.0.5/common-parent-6.0.5.pom (54 kB at 534 kB/s)
Downloading from common-docker-6.0.5.pom: https://packages.confluent.io/maven/kr/motd/maven/os-maven-plugin/1.6.2/os-maven-plugin-1.6.2.pom
Downloading from common-docker-6.0.5.pom: https://packages.confluent.io/maven/com/fasterxml/jackson/jackson-bom/2.10.5.20201202/jackson-bom-2.10.5.20201202.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin kr.motd.maven:os-maven-plugin:1.6.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for kr.motd.maven:os-maven-plugin:jar:1.6.2 @
[ERROR] Non-resolvable import POM: Could not transfer artifact com.fasterxml.jackson:jackson-bom:pom:2.10.5.20201202 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [confluent (http://packages.confluent.io/maven/, default, releases+snapshots)] @ io.confluent:common-parent:6.0.5, /root/.m2/repository/io/confluent/common-parent/6.0.5/common-parent-6.0.5.pom, line 235, column 25

[samding01]

@andrewegel In fact, I am trying to build cp-kafka image for s390x. I can build Apache-kafka on s390x based on instruction, but that is package only. Is it possible for me to use the package (jars) for s390x and create a cp-kafka image of s390x? I checked the docker files in this repository, did not find correct one. Can you give me an insight on how to do it in manually? Thanks,

[andrewegel]

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin kr.motd.maven:os-maven-plugin:1.6.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for kr.motd.maven:os-maven-plugin:jar:1.6.2 @
[ERROR] Non-resolvable import POM: Could not transfer artifact com.fasterxml.jackson:jackson-bom:pom:2.10.5.20201202 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [confluent (http://packages.confluent.io/maven/, default, releases+snapshots)] @ io.confluent:common-parent:6.0.5, /root/.m2/repository/io/confluent/common-parent/6.0.5/common-parent-6.0.5.pom, line 235, column 25

This error is now in your environment - https://packages.confluent.io/maven is not a global mirror for all maven dependencies, we only provide io.confluent or org.apache.kafka ones.

I am trying to build cp-kafka image for s390x.

See: https://github.com/confluentinc/common-docker/issues/117 We don't currently support any other architectures other than x86_64. ARM64 is likely the next candidate given its wide spread usage in the industry, but its not on any official roadmap. I've seen internal requests for s390x and ppc, but we're not anywhere near capable of supporting those any time soon.

So from here-on out you're on your own for s390 support. The only guidance I can give you now is to review my draft PR here: https://github.com/confluentinc/common-docker/pull/149 that will help you start building through the docker image "lineage" (so to speak) and working through to this repo's images and its builds.

[samding01]

@andrewegel Thank you for your guidance.

The steps to build cp-kafka image of s390x seems includes: 1) build the base image from your PR, but what is the package in https://github.com/confluentinc/common-docker/blob/dab21dcde641e0a8f3c8f9acee7649a7923d3f5a/base/Dockerfile.ubi8#L113 ? 2) from the docker file: https://github.com/confluentinc/kafka-images/blob/2b25dce700f461c5d27313eb5fd7b79c509f77a2/kafka/Dockerfile.ubi8 to build cp-kafka,

Does the RPMs: -DCONFLUENT_PACKAGES_REPO=https://packages.confluent.io/rpm/6.0/ (you provided in the command) contains apache kafka and confluent stuffs? I checked the RPMs and found they are all in x86_64, so we probably first need to build them for s390x. Is https://packages.confluent.io/archive/6.0/ the src ? How can I find a build instruction?

[andrewegel]

1. That line refers to this java artifact: https://github.com/confluentinc/common-docker/blob/dab21dcde641e0a8f3c8f9acee7649a7923d3f5a/base/pom.xml#L31-L81 - Which is pulling in the utility-belt from the same repo, just under a different sub-project: https://github.com/confluentinc/common-docker/tree/dab21dcde641e0a8f3c8f9acee7649a7923d3f5a/utility-belt

2. My sample PR is based on an un-released branch/version, I'm mostly saying, "Take the example changes, and apply it to your situation" - not "Build the PR" because the PR is based on an unreleased version of CP.

Does the RPMs: -DCONFLUENT_PACKAGES_REPO=https://packages.confluent.io/rpm/6.0/ (you provided in the command) contains apache kafka and confluent stuffs? I checked the RPMs and found they are all in x86_64,

Yes, its confluent's RPMs of the on Prem Confluent Platform code in RPMs.

I checked the RPMs and found they are all in x86_64

The top-level link is a yum repo that contains only noarch packages (Java is noarch), the sub-directories 7 and 8 are RPMs built against EL7 and EL8 (so they work with CentOS or RHEL), and are indeed x86_64 only today and are just the c/c++ clients such as librdkafka, avro-c/cpp libs, etc. Those can likely be ignored for your use case.

so we probably first need to build them for s390x. Is https://packages.confluent.io/archive/6.0/ the src ?

The archives are for situations where you can't/don't want to use RPM or Debian packages, and just want a binary blob of pre-compiled Java services that you launch our of /opt/confluent (or wherever you install the tgz too), see: https://docs.confluent.io/platform/6.0.5/installation/installing_cp/zip-tar.html

How can I find a build instruction?

There are none to build the Confluent Platform as its a mix of open-source projects (kafka, ksql, rest-proxy, schema-registry) and proprietary projects (confluent-server which is a custom form of OSS kafka, metadata-service, control-center).

This project's purpose is simpler than you think, its main goals are just Installing pre-built binaries from CONFLUENT_PACKAGES_REPO into a docker container, and setting up the runtime scripts.

At this point I think I've adequately addressed your issue here, and its now gotten very off topic on something Confluent Doesn't support (Building s390x Docker Images).

I'll leave you with our docs location where I think you should start reading on how to use CP and its components: https://docs.confluent.io/platform/current/overview.html

If you need further assistance, I encourage you to reach out to Confluent's Support, or ask some questions in the Confluent Community Slack: https://www.confluent.io/community/ask-the-community/