Closed jaehyeon-kim closed 1 year ago
iliax
commented
1 year ago Hello @jaehyeon-kim, due to logs serde jar file can't be found on containers fs - you should mount a volume with serde jar (see example https://github.com/provectus/kafkaui-glue-sr-serde/blob/af314c4b4e3a5af38585201b6b8b69bb83d9a622/docker-compose/setup-example.yaml)
jaehyeon-kim
commented
1 year ago Hi @iliax
Thanks for pointing that out. I added the JAR file via volume mapping and tried again. However I have another exception. At this time, I copied from the example compose file and the permission of the JAR file is set to be read/write/execute permissions.
Could you please check the issue?
2023-03-19 21:48:20,580 INFO [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.5.Final
2023-03-19 21:48:20,639 INFO [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.6 on f78725df9f55 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2023-03-19 21:48:20,641 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.7.5, Spring v5.3.23
2023-03-19 21:48:20,643 INFO [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"
2023-03-19 21:48:25,961 DEBUG [main] c.p.k.u.s.SerdesInitializer: Configuring serdes for cluster MyFavoriteCluster
2023-03-19 21:48:26,007 INFO [main] c.p.k.u.s.SerdesInitializer: Loading custom serde GlueSchemaRegistry
2023-03-19 21:48:26,043 WARN [main] o.s.b.w.r.c.AnnotationConfigReactiveWebServerApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesController' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/controller/MessagesController.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/MessagesService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/services/glue/model/EntityNotFoundException
2023-03-19 21:48:26,114 INFO [main] o.s.b.a.l.ConditionEvaluationReportLoggingListener:
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-03-19 21:48:26,180 ERROR [main] o.s.b.SpringApplication: Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesController' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/controller/MessagesController.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/MessagesService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/services/glue/model/EntityNotFoundException
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:66)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
at com.provectus.kafka.ui.KafkaUiApplication.startApplication(KafkaUiApplication.java:24)
at com.provectus.kafka.ui.KafkaUiApplication.main(KafkaUiApplication.java:17)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/MessagesService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/services/glue/model/EntityNotFoundException
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
... 26 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/services/glue/model/EntityNotFoundException
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
... 40 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/services/glue/model/EntityNotFoundException
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:224)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311)
... 54 common frames omitted
Caused by: java.lang.NoClassDefFoundError: software/amazon/awssdk/services/glue/model/EntityNotFoundException
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3373)
at java.base/java.lang.Class.getConstructor0(Class.java:3578)
at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2754)
at com.provectus.kafka.ui.serdes.CustomSerdeLoader.loadAndConfigure(CustomSerdeLoader.java:47)
at com.provectus.kafka.ui.serdes.SerdesInitializer.loadAndInitCustomSerde(SerdesInitializer.java:240)
at com.provectus.kafka.ui.serdes.SerdesInitializer.createSerdeFromConfig(SerdesInitializer.java:161)
at com.provectus.kafka.ui.serdes.SerdesInitializer.init(SerdesInitializer.java:101)
at com.provectus.kafka.ui.service.DeserializationService.<init>(DeserializationService.java:40)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211)
... 56 common frames omitted
Caused by: java.lang.ClassNotFoundException: software.amazon.awssdk.services.glue.model.EntityNotFoundException
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
at com.provectus.kafka.ui.serdes.CustomSerdeLoader$ChildFirstClassloader.loadClass(CustomSerdeLoader.java:123)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 71 common frames omitted---
version: "2"
services:
kafka-ui:
container_name: kafka-ui
image: provectuslabs/kafka-ui:master
ports:
- 8080:8080
depends_on:
- zookeeper0
- kafka0
environment:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN: $AWS_SESSION_TOKEN
kafka.clusters.0.name: MyFavoriteCluster
kafka.clusters.0.bootstrapServers: kafka0:29092
kafka.clusters.0.serde.0.name: GlueSchemaRegistry
kafka.clusters.0.serde.0.filePath: /glue-serde/kafkaui-glue-serde-1.0.0.jar
kafka.clusters.0.serde.0.className: com.provectus.kafka.ui.serdes.glue.GlueSerde
kafka.clusters.0.serde.0.properties.region: $AWS_DEFAULT_REGION #required
kafka.clusters.0.serde.0.properties.registry: ccdak #required, name of Glue Schema Registry
volumes:
- ./glue-serde:/glue-serde
zookeeper0:
image: confluentinc/cp-zookeeper:7.3.0
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 2181:2181
kafka0:
image: confluentinc/cp-kafka:7.3.0
depends_on:
- zookeeper0
ports:
- 9092:9092
- 9997:9997
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper0:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka0:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1In the example compose file, the JAR file seems to include dependencies (kafkaui-glue-serde-1.0-SNAPSHOT-jar-with-dependencies.jar) while those that in release don't include those (eg kafkaui-glue-serde-1.0.0.jar).
I wonder if the error is due to that. If so, could you please inform me how to obtain the JAR file as for the example compose file?
iliax
commented
1 year ago @jaehyeon-kim, thank you for noticing that - we will publish jar with dependencies.
To build it locally you need to execute mvn clean install -DskipTests commands and jar will be in target directory. (prerequisites are mvn and java 17 installed). Or you can wait until proper version will be published (couple of days I think).
https://github.com/provectus/kafkaui-glue-sr-serde/issues/10
jaehyeon-kim
commented
1 year ago Hi @iliax
How are you? I failed to build it with the following error. Attached (fail-messages.txt) includes the full debug messages.
Can you please inform me how to fix it?
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------< org.example:kafkaui-glue-serde >-------------------
[INFO] Building Archetype - kafkaui-glue-serde 1.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.835 s
[INFO] Finished at: 2023-03-28T07:22:38+11:00
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.5: Could not transfer artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.5 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionExceptionIt was a network issue. When I tried at home, I was able to obtain kafkaui-glue-serde-1.0-SNAPSHOT-jar-with-dependencies.jar. Let me try with it.
jaehyeon-kim
commented
1 year ago It's possible to add the jar into docker at least. However it fails to deserialise avro messages. Will create another issue.
Hello,
I tried to follow the example compose file but encountered the following error. Attached includes the compose file and full log dump. Could you please inform me how to fix it?
Cheers, Jaehyeon
docker-compose.txt error.txt