search

jeffchao / bottled-water

Kafka Connect + Postgres + Debezium and KSQL on Heroku
MIT License
12 stars 8 forks source link

Documentation on how to set this up #1

Closed amiel closed 6 years ago

amiel commented 6 years ago

Hi @jeffchao,

I am curious about your example. Would you be willing to write up a README explaining how to set this up on Heroku?

Thanks!!

-Amiel

jeffchao commented 6 years ago

Hi @amiel. I'll go ahead and do that. In the meantime, I'll assign this issue to myself. Thanks for reaching out.

amiel commented 6 years ago

Thanks @jeffchao!!! I look forward to trying it out :)

jeffchao commented 6 years ago

@amiel The README is up. I added some context as well. Feel free to file another issue if you have any other questions.

amiel commented 6 years ago

@jeffchao Thanks so much for adding this documentation.

I spent a day trying to get it set up. There were a few changes I had to make and I was planning to put a PR together with those changes when I got it working. However, I haven't been able to get the heroku process successfully running. I was able to connect the kafka cluster with kafkacat, but the heroku process eventually crashes with the following error:

Could not look up partition metadata for offset backing store topic in allotted period. This could indicate a connectivity issue, unavailable topic partitions, or if this is your first use of the topic it may have taken too long to create.

I'm pretty much out of time to experiment with this right now, but if you have any advice, I would appreciate it for the next time I get a chance to experiment with Kafka on Heroku. I will also clean up my changes and submit a PR. Even if you don't merge it, it may help other people trying to see what changes I had to make.

I'd love to get CDC working from Heroku Postgres to Heroku Kafka as I think this would improve the stability of data processing at work!

jeffchao commented 6 years ago

@amiel thanks for the feedback. I look forward to your PR.

Regarding the error, I suspect you are using the basic Kafka plans. Is this right? If so, then in these plans, you'll have to pre-create all internal Kafka topics using the kafka topics create command on the Heroku CLI. This is because Heroku basic Kafka plans are multitenant and provide isolation by leveraging Kafka ACLs. This is the most likely cause.

Another option is to try this on a standard kafka plan, which runs expensive, but fortunately is pro-rated to your usage. That way, you can spin one up, test it out, and spin it down.

amiel commented 6 years ago

@jeffchao Yes, I was on a basic Kafka plan. I added the KAFKA_PREFIX to each topic and created the topics manually. I also changed the replication factor to 3, as it seems that the basic plan does not support a replication factor of 1. Thanks for suggesting I try a standard plan, I should have tried that when I ran in to the issues that caused me to add the prefix. Anyway, I did get the app to boot on the standard plan, but there were some changes I had to make (PR on the way 😄 ).

amiel commented 6 years ago

@jeffchao check out https://github.com/jeffchao/bottled-water/pull/2 with my suggested changes.

jeffchao commented 6 years ago

Closed by #2.