search

RedPillAnalytics / gradle-confluent

Apache License 2.0
22 stars 10 forks source link

Fix pipelineExecute infinite loop #87

Closed jeffreylutz closed 4 years ago

jeffreylutz commented 4 years ago

When the SQL script contains 'SET ...' or 'UNSET ...', the pipeline causes ksql server to spike CPU and the pipeline runs indefinitely.

Support for properly handling the SET and UNSET commands. If the SET 'auto.offset.reset'='earliest'; Is set, then the corresponding flag is set or unset. Also, the docker-compose.yml has been updated with Confluent 5.5.0 / KSQLDB.
Lastly, updated the SQL scripts to work with Confluent KSQL 5.5.0.

jeffreylutz commented 4 years ago

There is an issue with building during the publishing phase. I opened a ticket issue with this. https://github.com/RedPillAnalytics/gradle-confluent/issues/88

However, the build artifact is created locally.

Jeff

stewartbryson commented 4 years ago

Hello @jeffreylutz. I've been on PTO, but am looking at this PR today and tomorrow. Thank you so much for your contribution.

stewartbryson commented 4 years ago

Issue #88 has been resolved with PR #90

stewartbryson commented 4 years ago

@jeffreylutz I upgraded the docker image I use for running tests to 5.5.1 of CP. The docker-compose file I have in the repo is for folks to give it a try, but I don't use that with CI/CD. We run Jenkins in GKE, and docker-compose is problematic because it requires DiD. The pod-template.yml file defines the environment for CI/CD. This PR is getting failed tests for two reasons: the GCS requirement (I'm not really sure why that is happening honestly)... but it doesn't matter, because I removed that from the tests. You can re-merge master into your branch and likely resolve that. The other is because I have a test that expects a particular outcome for what should exist in the ksql-script.sql file. That is failing because of the additions to the testing framework... which I am completely glad you did. Do you want to modify that test (or remove it, if we think it's constraining)? Let's try these two things and see if we get a successful build and test.

Let me reiterate... thank you so much for the contribution!

stewartbryson commented 4 years ago

Also @jeffreylutz... looks like you added a Dockerfile. Not sure if that was an accident, or because you weren't aware I was building a separate image for this. That image is built in the following repo in case you are interested: https://github.com/RedPillAnalytics/docker-confluent. It's a relatively simple build using the whole CP platform in one container, with the ksql-datagen process for adding the testing topics.