search

obsidiandynamics / goharvest

Transactional outbox harvester for Postgres → Kafka, written in Go
BSD 3-Clause "New" or "Revised" License
184 stars 18 forks source link

Update confluent kafka go dep #1

Closed B3rs closed 3 years ago

B3rs commented 3 years ago

Hi all, First of all I want to say good job in creating this package. I tried to use it as a package inside my application but I got stuck in this https://github.com/confluentinc/confluent-kafka-go/issues/450 confluent kafka go issue that has been fixed in v1.4.2. TL; DR; if you try to vendor confluent kafka go v1.4.0 you end up without a confluent folder and the package won't compile... Can we please update at least to v1.4.2? (i can provide a pr for that) thanks

milan-mnf commented 3 years ago

Hi @B3rs, I had the same issue. Not sure how far you got with this, but I solved it by switching from go's native vendoring to a tool called vend (https://github.com/nomad-software/vend). It appears that go's native vendoring does not copy over non-go files (e.g. c headers), and so go reports that files are missing. vend copies over the remaining files, fixing the compilation process. Hope it helps.

B3rs commented 3 years ago

hi @milan-mnf actually we have been using counfluent-kafka-go > 1.4.2 in production for a while and it is working fine without having to resort to third party vendor scripts... I just think that this library is unmaintained so maybe the best option can be forking it

ekoutanov commented 3 years ago

Let's get this in. (1.4.2)

ekoutanov commented 3 years ago

Looking at the release notes for 1.4.2, they fixed the bundling issue as per @B3rs original comment:

The bundled librdkafka directory (kafka/librdkafka) is no longer pruned by Go mod vendor import.

This was raised with Magnus in early 2020. Glad to see that was finally addressed. There's also release 1.5.2. We might as well go with the latest if we can.

B3rs commented 3 years ago

that whould be super useful thanks... i can provide a pr for it! do you have any contributing guidelines?