search

apache / eventmesh

EventMesh is a new generation serverless event middleware for building distributed event-driven applications.
https://eventmesh.apache.org/
Apache License 2.0
1.62k stars 642 forks source link

[Question] How to use kafka as a storage layer #4232

Open sun-tao opened 1 year ago

sun-tao commented 1 year ago

Search before asking

Question

When I want to use kafka as the storage layer for eventmesh,the runtime start fail.

The configuration is as follow:

connector plugin

eventMesh.connector.plugin.type=standalone

storage plugin

eventMesh.storage.plugin.type=kafka

Does eventmesh support kafka now? If support,how to start eventmesh with kafka?

github-actions[bot] commented 1 year ago

Welcome to the Apache EventMesh community!! We are glad that you are contributing by opening this issue. :D

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our project, please let us know! You can check out our contributing guide on contributing to EventMesh.

Want to get closer to the community?

WeChat Assistant WeChat Public Account Slack
Join Slack Chat
Mailing Lists: Name Description Subscribe Unsubscribe Archive
Users User support and questions mailing list Subscribe Unsubscribe Mail Archives
Development Development related discussions Subscribe Unsubscribe Mail Archives
Commits All commits to repositories Subscribe Unsubscribe Mail Archives
Issues Issues or PRs comments and reviews Subscribe Unsubscribe Mail Archives
Pil0tXia commented 1 year ago

Kafka storage plugin is supported. There is a Kafka connector PR still WIP. You may look for its progress here: https://github.com/apache/eventmesh/pull/4180.

pandaapo commented 1 year ago

First, it will be better if you can modify this issue title to something concrete. "Question title" is a bad title.

Then, Kafka storage plugin is supported. You can provide the error stack information for community accurately locating your problem.

nguyenminhtuanfit commented 1 year ago

eventMesh.storage.plugin.type=kafka

still facing this issue when run

java -Deventmesh.log.home=/Users/tuan/Desktop/apache-eventmesh-1.9.0 -DconfPath=${EVENTMESH_HOME}/conf -classpath ${EVENTMESH_HOME}/conf:${EVENTMESH_HOME}/apps/:${EVENTMESH_HOME}/lib/ org.apache.eventmesh.runtime.boot.EventMeshStartup

3-09-15 00:31:00,407 DEBUG [grpc-nio-boss-ELG-1-1] DefaultChannelId(DefaultChannelId.java:79) - -Dio.netty.processId: 51938 (auto-detected) 2023-09-15 00:31:00,409 DEBUG [grpc-nio-boss-ELG-1-1] NetUtil(NetUtil.java:135) - -Djava.net.preferIPv4Stack: false 2023-09-15 00:31:00,409 DEBUG [grpc-nio-boss-ELG-1-1] NetUtil(NetUtil.java:136) - -Djava.net.preferIPv6Addresses: false 2023-09-15 00:31:00,411 DEBUG [grpc-nio-boss-ELG-1-1] NetUtilInitializations(NetUtilInitializations.java:129) - Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1%lo0) 2023-09-15 00:31:00,412 DEBUG [grpc-nio-boss-ELG-1-1] NetUtil(NetUtil.java:187) - Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. Default: 128 2023-09-15 00:31:00,414 DEBUG [grpc-nio-boss-ELG-1-1] DefaultChannelId(DefaultChannelId.java:101) - -Dio.netty.machineId: 4c:32:75:ff:fe:92:a4:71 (auto-detected) 2023-09-15 00:31:00,439 DEBUG [grpc-nio-boss-ELG-1-1] ByteBufUtil(ByteBufUtil.java:87) - -Dio.netty.allocator.type: pooled 2023-09-15 00:31:00,439 DEBUG [grpc-nio-boss-ELG-1-1] ByteBufUtil(ByteBufUtil.java:96) - -Dio.netty.threadLocalDirectBufferSize: 0 2023-09-15 00:31:00,439 DEBUG [grpc-nio-boss-ELG-1-1] ByteBufUtil(ByteBufUtil.java:99) - -Dio.netty.maxThreadLocalCharBufferSize: 16384 2023-09-15 00:31:00,452 INFO [main] EventMeshGrpcServer(EventMeshGrpcServer.java:150) - ---------------EventMeshGRPCServer running------------------- 2023-09-15 00:31:00,478 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.rocketmq.admin.RocketMQAdminAdaptor 2023-09-15 00:31:00,479 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.standalone.admin.StandaloneAdminAdaptor 2023-09-15 00:31:00,480 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.rocketmq.admin.RocketMQAdminAdaptor 2023-09-15 00:31:00,480 INFO [main] JarExtensionClassLoader(JarExtensionClassLoader.java:132) - load extension class success, extensionType: interface org.apache.eventmesh.api.admin.Admin, extensionClass: class org.apache.eventmesh.storage.standalone.admin.StandaloneAdminAdaptor 2023-09-15 00:31:00,480 ERROR [main] MQAdminWrapper(MQAdminWrapper.java:40) - can't load the meshMQAdmin plugin, please check. 2023-09-15 00:31:00,485 pool-1-thread-1 DEBUG Stopping LoggerContext[name=1b6d3586, org.apache.logging.log4j.core.LoggerContext@428640fa] 2023-09-15 00:31:00,486 pool-1-thread-1 DEBUG Stopping LoggerContext[name=1b6d3586, org.apache.logging.log4j.core.LoggerContext@428640fa]... 2023-09-15 00:31:00,480 ERROR [main] EventMeshStartup(EventMeshStartup.java:57) - EventMesh start fail. java.lang.RuntimeException: doesn't load the meshMQAdmin plugin, please check. at org.apache.eventmesh.runtime.core.plugin.MQAdminWrapper.(MQAdminWrapper.java:41) ~[eventmesh-runtime-1.9.0-release.jar:1.9.0-release] at org.apache.eventmesh.runtime.admin.handler.TopicHandler.(TopicHandler.java:59) ~[eventmesh-runtime-1.9.0-release.jar:1.9.0-release] at org.apache.eventmesh.runtime.admin.controller.ClientManageController.initClientHandler(ClientManageController.java:129) ~[eventmesh-runtime-1.9.0-release.jar:1.9.0-release] at org.apache.eventmesh.runtime.admin.controller.ClientManageController.start(ClientManageController.java:95) ~[eventmesh-runtime-1.9.0-release.jar:1.9.0-release] at org.apache.eventmesh.runtime.boot.EventMeshServer.start(EventMeshServer.java:169) ~[eventmesh-runtime-1.9.0-release.jar:1.9.0-release] at org.apache.eventmesh.runtime.boot.EventMeshStartup.main(EventMeshStartup.java:40) [eventmesh-runtime-1.9.0-release.jar:1.9.0-release] 2023-09-15 00:31:00,489 pool-1-thread-1 DEBUG Shutting down OutputStreamManager SYSTEM_OUT.false.false 2023-09-15 00:31:00,490 pool-1-thread-1 DEBUG OutputStream closed 2023-09-15 00:31:00,490 pool-1-thread-1 DEBUG Shut down OutputStreamManager SYSTEM_OUT.false.false, all resources released: true 2023-09-15 00:31:00,490 pool-1-thread-1 DEBUG Appender console stopped with status true 2023-09-15 00:31:00,490 pool-1-thread-1 DEBUG Log4j2 ConfigurationScheduler shutting down threads in java.util.concurrent.ScheduledThreadPoolExecutor@69532cf3[Running, pool size = 1, active threads = 0, queued tasks = 1, completed tasks = 0] 2023-09-15 00:31:00,490 pool-1-thread-1 DEBUG Stopped XmlConfiguration[location=/Users/tuan/Desktop/dbs/apache-eventmesh-1.9.0/conf/log4j2.xml] OK 2023-09-15 00:31:00,491 pool-1-thread-1 DEBUG Stopped LoggerContext[name=1b6d3586, org.apache.logging.log4j.core.LoggerContext@428640fa] with status true

Pil0tXia commented 1 year ago

@nguyenminhtuanfit Please use the lastest master branch version of EventMesh. It has been fixed.

Pil0tXia commented 10 months ago

Have you encountered any additional problems related to this issue? Please share any relevant experiences here. If you think your issue has been resolved, please feel free to close this thread.

github-actions[bot] commented 7 months ago

It has been 90 days since the last activity on this issue. Apache EventMesh values the voices of the community. Please don't hesitate to share your latest insights on this matter at any time, as the community is more than willing to engage in discussions regarding the development and optimization directions of this feature.

If you feel that your issue has been resolved, please feel free to close it. Should you have any additional information to share, you are welcome to reopen this issue.