search

confluentinc / confluent-kafka-go

Confluent's Apache Kafka Golang client
Apache License 2.0
4.6k stars 655 forks source link

Build fails on mac M1 with Bazel #1059

Open zphoenixrises opened 1 year ago

zphoenixrises commented 1 year ago

Description

I am trying to use the confluent-kafka-go library in my project. The build fails with the following error:

 bazel build //simulation/services/simulationrunner --sandbox_debug
INFO: Invocation ID: b75c6734-928a-46e6-b461-70b91e0bee44
INFO: Streaming build results to: https://app.buildbuddy.io/invocation/b75c6734-928a-46e6-b461-70b91e0bee44
INFO: Analyzed target //simulation/services/simulationrunner:simulationrunner (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_zeeshanhaque/35fb6bb51532907f6ed2291f09a7e5b4/external/com_github_confluentinc_confluent_kafka_go/kafka/BUILD.bazel:3:11: GoCompilePkg external/com_github_confluentinc_confluent_kafka_go/kafka/kafka.a failed: (Exit 1): sandbox-exec failed: error executing command 
  (cd /private/var/tmp/_bazel_zeeshanhaque/35fb6bb51532907f6ed2291f09a7e5b4/sandbox/darwin-sandbox/392/execroot/chaski && \
  exec env - \
    CC=external/local_config_cc/cc_wrapper.sh \
    CGO_ENABLED=1 \
    GOARCH=arm64 \
    GOOS=darwin \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:external/local_config_cc:/bin \
    TMPDIR=/var/folders/kv/dcf86fc90vzc7bx36pjy4zjw0000gn/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_zeeshanhaque/35fb6bb51532907f6ed2291f09a7e5b4/sandbox/darwin-sandbox/392/sandbox.sb /var/tmp/_bazel_zeeshanhaque/install/202e556d99f024ca401d77f7be1827ad/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/private/var/tmp/_bazel_zeeshanhaque/35fb6bb51532907f6ed2291f09a7e5b4/sandbox/darwin-sandbox/392/stats.out' bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-e846b08c7501/bin/external/go_sdk/builder_reset/builder compilepkg -sdk external/go_sdk -installsuffix darwin_arm64 -src external/com_github_confluentinc_confluent_kafka_go/kafka/00version.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/adminapi.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/adminoptions.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/build_darwin_amd64.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/build_darwin_arm64.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/build_glibc_linux.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/build_windows.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/config.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/consumer.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/context.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/error.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/error_gen.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/event.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/generated_errors.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/handle.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/header.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/kafka.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/log.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/message.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/metadata.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/misc.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/mockcluster.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/offset.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/producer.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/testhelpers.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/time.go -src external/com_github_confluentinc_confluent_kafka_go/kafka/glue_rdkafka.h -src external/com_github_confluentinc_confluent_kafka_go/kafka/select_rdkafka.h -embedroot '' -embedroot bazel-out/darwin_arm64-fastbuild-ST-eccb913b7463/bin -embedlookupdir external/com_github_confluentinc_confluent_kafka_go/kafka -arc 'github.com/confluentinc/confluent-kafka-go/kafka/librdkafka_vendor=github.com/confluentinc/confluent-kafka-go/kafka/librdkafka_vendor=bazel-out/darwin_arm64-fastbuild-ST-eccb913b7463/bin/external/com_github_confluentinc_confluent_kafka_go/kafka/librdkafka_vendor/librdkafka_vendor.x' -importpath github.com/confluentinc/confluent-kafka-go/kafka -p github.com/confluentinc/confluent-kafka-go/kafka -package_list bazel-out/darwin_arm64-opt-exec-2B5CBBC6-ST-e846b08c7501/bin/external/go_sdk/packages.txt -o bazel-out/darwin_arm64-fastbuild-ST-eccb913b7463/bin/external/com_github_confluentinc_confluent_kafka_go/kafka/kafka.a -x bazel-out/darwin_arm64-fastbuild-ST-eccb913b7463/bin/external/com_github_confluentinc_confluent_kafka_go/kafka/kafka.x -gcflags '' -asmflags '' -cppflags '-I external/com_github_confluentinc_confluent_kafka_go/kafka -iquote .' -cflags '-U_FORTIFY_SOURCE -fstack-protector -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -DUSE_VENDORED_LIBRDKAFKA -DLIBRDKAFKA_STATICLIB -fPIC' -cxxflags '-U_FORTIFY_SOURCE -fstack-protector -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++0x -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fPIC' -objcflags '-U_FORTIFY_SOURCE -fstack-protector -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -DUSE_VENDORED_LIBRDKAFKA -DLIBRDKAFKA_STATICLIB -fPIC' -objcxxflags '-U_FORTIFY_SOURCE -fstack-protector -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++0x -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fPIC' -ldflags '-undefined dynamic_lookup -headerpad_max_install_names -lc++ -lm kafka/librdkafka_vendor/librdkafka_darwin_arm64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration')
In file included from chaski/external/com_github_confluentinc_confluent_kafka_go/kafka/00version.go:24:
external/com_github_confluentinc_confluent_kafka_go/kafka/select_rdkafka.h:26:10: fatal error: 'librdkafka_vendor/rdkafka.h' file not found
#include "librdkafka_vendor/rdkafka.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
compilepkg: error running subcommand external/go_sdk/pkg/tool/darwin_arm64/cgo: exit status 2
Target //simulation/services/simulationrunner/cmd:cmd.binary failed to build
INFO: Elapsed time: 1.815s, Critical Path: 0.97s
INFO: 64 processes: 13 internal, 51 darwin-sandbox.
INFO: Streaming build results to: https://app.buildbuddy.io/invocation/b75c6734-928a-46e6-b461-70b91e0bee44
FAILED: Build did NOT complete successfully

My bazel build file:

load("@io_bazel_rules_go//go:def.bzl", "go_library")

go_library(
    name = "kafkasr",
    srcs = [
        "consumer.go",
        "producer.go",
    ],
    importpath = "github.com/greaterindustries/chaski/libs/comms/kafkasr",
    visibility = ["//visibility:public"],
    deps = [
        "@com_github_confluentinc_confluent_kafka_go//kafka",
        "@com_github_confluentinc_confluent_kafka_go//schemaregistry",
        "@com_github_confluentinc_confluent_kafka_go//schemaregistry/serde",
        "@com_github_confluentinc_confluent_kafka_go//schemaregistry/serde/protobuf",
        "@org_golang_google_protobuf//proto",
        "@org_golang_google_protobuf//reflect/protoreflect",
    ]
)

Checklist

Please provide the following information:

eko commented 6 months ago

Hi @zphoenixrises,

I have the exact same issue. Have you found any solution?

eko commented 6 months ago

I found a solution by patching the BUILD.bazel generated files for this dependency, as mentioned in the issue here: https://github.com/bazelbuild/bazel-gazelle/issues/773

In the deps.bzl file, update the dependency with the patch:

    go_repository(
        name = "com_github_confluentinc_confluent_kafka_go_v2",
        importpath = "github.com/confluentinc/confluent-kafka-go/v2",
        # Issue source: https://github.com/bazelbuild/bazel-gazelle/issues/773
        patches = ["//bazel:kafka.patch"],  # keep
        sum = "h1:icCHutJouWlQREayFwCc7lxDAhws08td+W3/gdqgZts=",
        version = "v2.3.0",
    )

... and here is the bazel/kafka.patch file for the v2.3.0 release:

--- kafka/librdkafka_vendor/BUILD.bazel
+++ kafka/librdkafka_vendor/BUILD.bazel
@@ -1,5 +1,33 @@
 load("@io_bazel_rules_go//go:def.bzl", "go_library")

+cc_library(
+    name = "librdkafka_static",
+    srcs = select({
+        "@io_bazel_rules_go//go/platform:android": [
+            "librdkafka_glibc_linux_amd64.a",
+        ],
+        "@io_bazel_rules_go//go/platform:darwin_amd64": [
+            "librdkafka_darwin_amd64.a",
+        ],
+        "@io_bazel_rules_go//go/platform:darwin_arm64": [
+            "librdkafka_darwin_arm64.a",
+        ],
+        "@io_bazel_rules_go//go/platform:ios_amd64": [
+            "librdkafka_darwin_amd64.a",
+        ],
+        "@io_bazel_rules_go//go/platform:ios_arm64": [
+            "librdkafka_darwin_arm64.a"
+        ],
+        "@io_bazel_rules_go//go/platform:linux": [
+            "librdkafka_glibc_linux_amd64.a",
+            "librdkafka_musl_linux_amd64.a",
+        ],
+        "//conditions:default": [],
+    }),
+    hdrs = ["rdkafka.h", "rdkafka_mock.h"],
+    visibility = ["//visibility:public"],
+)
+
 go_library(
     name = "librdkafka_vendor",
     srcs = [

--- kafka/BUILD.bazel
+++ kafka/BUILD.bazel
@@ -33,6 +33,7 @@
         "time.go",
     ],
     cgo = True,
+    cdeps = ["//kafka/librdkafka_vendor:librdkafka_static"],
     clinkopts = select({
         "@io_bazel_rules_go//go/platform:windows": [
             "kafka/librdkafka_vendor/librdkafka_windows.a -lws2_32 -lsecur32 -lcrypt32",
@@ -40,28 +41,28 @@
         "//conditions:default": [],
     }) + select({
         "@io_bazel_rules_go//go/platform:android_amd64": [
-            "kafka/librdkafka_vendor/librdkafka_glibc_linux_amd64.a -lm -ldl -lpthread -lrt",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_glibc_linux_amd64.a -lm -ldl -lpthread -lrt",
         ],
         "@io_bazel_rules_go//go/platform:android_arm64": [
-            "kafka/librdkafka_vendor/librdkafka_glibc_linux_arm64.a -lm -ldl -lpthread -lrt",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_glibc_linux_arm64.a -lm -ldl -lpthread -lrt",
         ],
         "@io_bazel_rules_go//go/platform:darwin_amd64": [
-            "kafka/librdkafka_vendor/librdkafka_darwin_amd64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_darwin_amd64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
         ],
         "@io_bazel_rules_go//go/platform:darwin_arm64": [
-            "kafka/librdkafka_vendor/librdkafka_darwin_arm64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_darwin_arm64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
         ],
         "@io_bazel_rules_go//go/platform:ios_amd64": [
-            "kafka/librdkafka_vendor/librdkafka_darwin_amd64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_darwin_amd64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
         ],
         "@io_bazel_rules_go//go/platform:ios_arm64": [
-            "kafka/librdkafka_vendor/librdkafka_darwin_arm64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_darwin_arm64.a -lm -lsasl2 -ldl -lpthread -framework CoreFoundation -framework SystemConfiguration",
         ],
         "@io_bazel_rules_go//go/platform:linux_amd64": [
-            "kafka/librdkafka_vendor/librdkafka_glibc_linux_amd64.a -lm -ldl -lpthread -lrt",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_glibc_linux_amd64.a -lm -ldl -lpthread -lrt",
         ],
         "@io_bazel_rules_go//go/platform:linux_arm64": [
-            "kafka/librdkafka_vendor/librdkafka_glibc_linux_arm64.a -lm -ldl -lpthread -lrt",
+            "external/com_github_confluentinc_confluent_kafka_go_v2/kafka/librdkafka_vendor/librdkafka_glibc_linux_arm64.a -lm -ldl -lpthread -lrt",
         ],
         "//conditions:default": [],
     }),

Then, it will successfully build because it will be able to find the embedded librdkafka pre-built files.

I hope it could help.