search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.64k stars 598 forks source link

ClassNotFoundException: io.netty.channel.epoll.NativeStaticallyReferencedJniMethods #3170

Open sautran opened 5 years ago

sautran commented 5 years ago

I am using the kubernete deployment and go through the installation step. After deployment, I am seeing the following error in api-server as well as the worker pod

Jan 24, 2019 6:27:39 PM org.apache.distributedlog.impl.BKNamespaceDriver getDefaultEventLoopGroup WARNING: Could not use Netty Epoll event loop for bookie server: java.lang.UnsatisfiedLinkError: failed to load the required native library at dlshade.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78) at dlshade.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:38) at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:259) at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:270) at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:208) at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:238) at com.twitter.heron.downloader.DLDownloader.download(DLDownloader.java:66) at com.twitter.heron.downloader.DownloadRunner.main(DownloadRunner.java:42) Caused by: java.lang.NoClassDefFoundError: io/netty/channel/epoll/NativeStaticallyReferencedJniMethods at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at dlshade.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36) at dlshade.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:278) at dlshade.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:231) at dlshade.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:191) at dlshade.io.netty.channel.epoll.Native.(Native.java:61) at dlshade.io.netty.channel.epoll.Epoll.(Epoll.java:33) ... 7 more Caused by: java.lang.ClassNotFoundException: io.netty.channel.epoll.NativeStaticallyReferencedJniMethods at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 18 more

Jan 24, 2019 6:27:39 PM org.apache.distributedlog.impl.BKNamespaceDriver createBKCBuilder

sautran commented 5 years ago

i am running minikube on mac

sautran commented 5 years ago

Look like heron packaged with old distributedlog core jar (0.5). in this release, it package io.netty with dlshade which will cause issue when loading native library.

sautran commented 5 years ago

Here is the log in the api server [2019-01-25 18:30:18 +0000] [INFO] org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase: ZooKeeper client is connected now.
[2019-01-25 18:30:18 +0000] [WARNING] org.apache.distributedlog.impl.BKNamespaceDriver: Could not use Netty Epoll event loop for bookie server: java.lang.UnsatisfiedLinkError: failed to load the required native library at dlshade.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78) at dlshade.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:38) at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:259) at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:270) at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:208) at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:238) at com.twitter.heron.uploader.dlog.DLUploader.initializeNamespace(DLUploader.java:137) at com.twitter.heron.uploader.dlog.DLUploader.initialize(DLUploader.java:94) at com.twitter.heron.scheduler.SubmitterMain.submitTopology(SubmitterMain.java:420) at com.twitter.heron.apiserver.actions.SubmitTopologyAction.execute(SubmitTopologyAction.java:33) at com.twitter.heron.apiserver.resources.TopologyResource.submit(TopologyResource.java:223) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: io/netty/channel/epoll/NativeStaticallyReferencedJniMethods at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at java.lang.Runtime.load0(Runtime.java:809) at java.lang.System.load(System.java:1086) at dlshade.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36) at dlshade.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:278) at dlshade.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:231) at dlshade.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:191) at dlshade.io.netty.channel.epoll.Native.(Native.java:61) at dlshade.io.netty.channel.epoll.Epoll.(Epoll.java:33) ... 59 more Caused by: java.lang.ClassNotFoundException: io.netty.channel.epoll.NativeStaticallyReferencedJniMethods at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 70 more

sautran commented 5 years ago

adding -D io.netty.packagePrefix=dlshade. didn't solve the problem heron-apiserver --base-template kubernetes --cluster kubernetes -D io.netty.packagePrefix=dlshade. -D heron.statemgr.connection.string=zookeeper:2181