CP 5.4.1 compatability?

[CHR-LeeOlsen]

Describe the bug We upgraded our development environment to 5.4.1 and are getting the following error. [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

Have you tried using this with that version of KSQL?

[stewartbryson]

We test using 5.4.0. Let me update our builds to test against 5.4.1 and see what happens. Hang tight.

[CHR-LeeOlsen]

These are the logs immediately proceeding the [ERROR] notice. Very vague error message so I'm not sure what to even look at. Could easily be an issue with the setup from our Infrastructure team.

14:41:50.190 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Execute copy for :pipelineExtract' started 14:41:50.207 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Completing Build operation 'Execute copy for :pipelineExtract' 14:41:50.207 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Execute copy for :pipelineExtract' completed 14:41:50.208 [DEBUG] [org.gradle.api.internal.tasks.execution.DefaultTaskFingerprinter] Fingerprinting property destinationDir (Output) for task ':pipelineExtract' 14:41:50.232 [DEBUG] [org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter] Removed task artifact state for {} from context. 14:41:50.234 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Completing Build operation 'Task :pipelineExtract' 14:41:50.234 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Task :pipelineExtract' completed 14:41:50.234 [INFO] [org.gradle.execution.plan.DefaultPlanExecutor] :pipelineExtract (Thread[main,5,main]) completed. Took 0.142 secs. 14:41:50.235 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] main: released lock on : 14:41:50.235 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1.13 completed (1 worker(s) in use) 14:41:50.235 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] main: released lock on root.1.13 14:41:50.237 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] main: acquired lock on : 14:41:50.237 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1.13 started (1 worker(s) in use). 14:41:50.237 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] main: acquired lock on root.1.13 14:41:50.237 [INFO] [org.gradle.execution.plan.DefaultPlanExecutor] :pipelineDeploy (Thread[main,5,main]) started.

[stewartbryson]

Successfully tested against 5.4.1 with PR #83 . That PR tested successfully, and there is a new release building and publishing. However... it's just for updated dependencies... no updated code.

I believe your issue is likely related to an older Gradle version. Can you test with a newer version of Gradle? Or better yet... provide me with the version you have and I'll see if I can reproduce.

[stewartbryson]

Sorry... didn't mean to close the issue. Still open. ;-)

[CHR-LeeOlsen]

It worked fine the last time I used it (before our KSQL upgrade) but maybe there is another underlying error being hidden by an issue with the gradle versions exceptionReporter throwing an error. I'll try updating to the version being used in this repo.

plugins { id 'com.redpillanalytics.gradle-confluent' version '1.1.20' }

Gradle 5.5.1

---

Build time: 2019-07-10 20:38:12 UTC Revision: 3245f748c7061472da4dc184991919810f7935a5 Kotlin: 1.3.31 Groovy: 2.5.4 Ant: Apache Ant(TM) version 1.9.14 compiled on March 12 2019 JVM: 1.8.0_241 (Oracle Corporation 25.241-b07) OS: Windows Server 2016 10.0 amd64

[CHR-LeeOlsen]

Updated and getting the same output.

Gradle 6.3

---

Build time: 2020-03-24 19:52:07 UTC Revision: bacd40b727b0130eeac8855ae3f9fd9a0b207c60 Kotlin: 1.3.70 Groovy: 2.5.10 Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 JVM: 1.8.0_241 (Oracle Corporation 25.241-b07) OS: Windows Server 2016 10.0 amd64

[CHR-LeeOlsen]

I'm not sure what is causing this output. Running the deployment from my local machine instead of through OctopusDeploy with --debug is just crashing but that used to work. When I changed to --info I'm getting good output. The issue is either with my KSQL or with our ACL's on the topics I'm guessing.

Thanks for your time!

[CHR-LeeOlsen]

The error we were getting was:

error_code: 50000: A Kafka topic with the name 'our-source-topic' already exists, with different partition/replica configuration than required. KSQL expects 2 partitions (topic has 1), and 3 replication factor (topic has 3).

I reopened this in case you wanted to look into why --debug was throwing that other error.

[stewartbryson]

@CHR-LeeOlsen Now building and testing against 5.5.1 of CP. Are you still seeing this issue?

[CHR-LeeOlsen]

We have actually abandoned using a centralized KSQL instance with interactive mode on. We ran into issues where deploying to this single KSQL instance from multiple repos was deleting other teams KSQL.

We are still using this plugin to combine multiple KSQL statements into a single file (for ease of development) which is then bundled within a docker image with KSQL so that we are deploying an instance per repo in headless mode. You can close this issue.

[stewartbryson]

We have actually abandoned using a centralized KSQL instance with interactive mode on. We ran into issues where deploying to this single KSQL instance from multiple repos was deleting other teams KSQL.

We are still using this plugin to combine multiple KSQL statements into a single file (for ease of development) which is then bundled within a docker image with KSQL so that we are deploying an instance per repo in headless mode. You can close this issue.

Are you using the ksql-script.sql generated from the plugin @CHR-LeeOlsen? Just curious.

[CHR-LeeOlsen]

Yes. You don't foresee any problems with doing so, do you?

[stewartbryson]

Yes. You don't foresee any problems with doing so, do you?

No!! I was just happy that functionality was working. I added it, and tests exist for it, but we never used that in production, so I'm glad to hear someone is. ;-)

[stewartbryson]

Closing based on a comment from @CHR-LeeOlsen