`/kafka/librdkafka` does not appear in `vendor` directory when `go mod vendor` run

[lizthegrey]

Description

I can get this to work in my local dev environment (stretch, go version go1.12.7 linux/amd64) but not inside a CircleCI golang docker container (stretch, circleci/golang:1.12.5 and circleci/golang:1.12.7 and circleci/golang:1.12.7-buster).

go mod vendor doesn't actually pull the correct subdirs with the packaged .a library files :(

go.mod contains: github.com/confluentinc/confluent-kafka-go-dev v0.0.0-20190703144503-78a815343f20

How to reproduce

$ go mod tidy
$ go mod vendor
$ ls vendor/github.com/confluentinc/confluent-kafka-go-dev/kafka/librdkafka
No such file or directory

running /bin/bash -c "go" "install" "-mod" "vendor" "-ldflags" "-X main.BuildID=129200 -X main.CommitHash=cfa9164ec5a20ec611273f9c903f966d10002bb4     -X github.com/honeycombio/hound/logging/beelineinit.BuildID=129200     -X github.com/honeycombio/hound/logging/beelineinit.BuildDepth=" "./..."
# github.com/confluentinc/confluent-kafka-go-dev/kafka
vendor/github.com/confluentinc/confluent-kafka-go-dev/kafka/00version.go:24:32: fatal error: librdkafka/rdkafka.h: No such file or directory
 #include <librdkafka/rdkafka.h>
                                ^

Checklist

Please provide the following information:

• [x] confluent-kafka-go and librdkafka version (LibraryVersion()):
• [ ] Apache Kafka broker version:
• [ ] Client configuration: ConfigMap{...}
• [x] Operating system:
• [ ] Provide client logs (with "debug": ".." as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[lizthegrey]

This is a side effect of https://github.com/golang/go/issues/26366, which apparently has workarounds?

[edenhill]

Whoa! That was quick! :100:

We'll make this work with go mods, somehow, thanks for link

[lizthegrey]

https://github.com/golang/go/issues/26366#issuecomment-424199216 is the further detail -- you have to put it in the same directory.

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

[edenhill]

I think I found another workaround by making kafka/librdkafka a sub-package that kafka/ imports, seems to make go mod vendor include it. I pushed the fix to confluent-kafka-go-dev master, please update/reset your vendor version and give it a go.

[lizthegrey]

Trying :)

[lizthegrey]

Solved by https://github.com/confluentinc/confluent-kafka-go-dev/commit/4315eca36bc989d0d3dde9fa5a774ce3e5da536a indeed.

[edenhill]

Awesome! Thank you 🥇