java-spring-boot2: add kafka template

yharish991 commented 4 years ago

Checklist:

[x] Read the Code of Conduct and Contributing Guidelines.
[x] Followed the commit message guidelines.
[x] Stack adheres to Appsody stack structure.

Modifying an existing stack:

[x] Updated the stack version in stack.yaml

This PR adds a new kafka template to the java-spring-boot2 stack.

Related Issues:

Related to #695

skoh7645 commented 4 years ago

Holding off on merging this as @yharish991 and @ianpartridge are still working together on it. Their aim is to merge and release the java-spring-boot2 stack by this week

yharish991 commented 4 years ago

I see this in my logs when I run this template:

[Container] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ default-kafka-application ---
[Container] [INFO] Changes detected - recompiling the module!
[Container] [INFO] Compiling 4 source files to /project/user-app/target/classes
[Container] [INFO] /project/user-app/src/main/java/application/config/KafkaProducer.java: /project/user-app/src/main/java/application/config/KafkaProducer.java uses unchecked or unsafe operations.
[Container] [INFO] /project/user-app/src/main/java/application/config/KafkaProducer.java: Recompile with -Xlint:unchecked for details.

I pushed some changes to parameterize the KafkaTemplate, you should not see that in the logs anymore

yharish991 commented 4 years ago

I wasn't able to get appsody run to work with this template. Here's the error I see:

[Container] 2020-04-07 15:52:20.959  INFO 178 --- [  restartedMain] o.a.kafka.common.utils.AppInfoParser     : Kafka version : 2.0.1
[Container] 2020-04-07 15:52:20.959  INFO 178 --- [  restartedMain] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId : fa14705e51bd2ce5
[Container] 2020-04-07 15:53:21.139  WARN 178 --- [  restartedMain] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
[Container] 2020-04-07 15:53:21.147  INFO 178 --- [  restartedMain] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'
[Container] 2020-04-07 15:53:21.177  INFO 178 --- [  restartedMain] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
[Container] 2020-04-07 15:53:21.202  WARN 178 --- [  restartedMain] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [MemoryMXBean notification dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[Container]  com.ibm.lang.management.internal.MemoryNotificationThread.processNotificationLoop(Native Method)
[Container]  com.ibm.lang.management.internal.MemoryNotificationThread.run(MemoryNotificationThread.java:183)
[Container] 2020-04-07 15:53:21.260  INFO 178 --- [  restartedMain] ConditionEvaluationReportLoggingListener :
[Container]
[Container] Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
[Container] 2020-04-07 15:53:21.274 ERROR 178 --- [  restartedMain] o.s.boot.SpringApplication               : Application run failed
[Container]
[Container] org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
[Container]   at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185) ~[spring-context-5.1.13.RELEASE.jar:5.1.13.RELEASE]
[Container]   at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53) ~[spring-context-5.1.13.RELEASE.jar:5.1.13.RELEASE]
[Container]   at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360) ~[spring-context-5.1.13.RELEASE.jar:5.1.13.RELEASE]
[Container]   at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158) ~[spring-context-5.1.13.RELEASE.jar:5.1.13.RELEASE]
[Container]   at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122) ~[spring-context-5.1.13.RELEASE.jar:5.1.13.RELEASE]
[Container]   at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:893) ~[spring-context-5.1.13.RELEASE.jar:5.1.13.RELEASE]
[Container]   at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162) ~[spring-boot-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container]   at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552) ~[spring-context-5.1.13.RELEASE.jar:5.1.13.RELEASE]
[Container]   at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container]   at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744) [spring-boot-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container]   at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391) [spring-boot-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container]   at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) [spring-boot-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container]   at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container]   at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204) [spring-boot-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container]   at application.Main.main(Main.java:10) [classes/:na]
[Container]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_242]
[Container]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_242]
[Container]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_242]
[Container]   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_242]
[Container]   at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-2.1.12.RELEASE.jar:2.1.12.RELEASE]
[Container] Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

Can you try using the below docker-compose.yaml and run the appsody app with this command appsody run --network kafka_default --docker-options "--env KAFKA_BOOTSTRAP_SERVERS=kafka:9092"

version: '2'
services:
  zookeeper:
    image: strimzi/kafka:0.17.0-kafka-2.4.0
    command: [
      "sh", "-c",
      "bin/zookeeper-server-start.sh config/zookeeper.properties"
    ]
    ports:
      - "2181:2181"
    environment:
      LOG_DIR: /tmp/logs

  kafka:
    image: strimzi/kafka:0.17.0-kafka-2.4.0
    command: [
      "sh", "-c",
      "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
    ]
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    expose:
      - "9092"
    environment:
      LOG_DIR: "/tmp/logs"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

matthewpwilson commented 4 years ago

Can you try using the below docker-compose.yaml and run the appsody app with this command appsody run --network kafka_default --docker-options "--env KAFKA_BOOTSTRAP_SERVERS=kafka:9092"

I can confirm this works for me.

rykal-nate commented 4 years ago

Are the kafka tests ran as part of appsody test? I assume not because that would require the kafka server right? As part of the travis CI/CD build appsody test is ran so just wondering how that is handled. Thanks!

ianpartridge commented 4 years ago

They use a Spring EmbeddedKafka to spin up an embedded broker to test against :)

rykal-nate commented 4 years ago

Thanks Ian.

yharish991 commented 4 years ago

@skoh7645 can you merge this pr?

appsody / stacks