Internal server error when using InstanceDiscovery

mcacker commented 11 years ago

Hi, I have a custom InstanceDiscovery class, which has been deployed on the hystrix-dashboard/turbine server. From the output of catalina.out (we're running on Tomcat), the InstanceDiscovery appears to be getting called successfully, but the hystrix-dashboard is reporting an internal server error:

[11:06:59.395] GET http://10.81.224.161:8080/hystrix-dashboard/proxy.stream?origin=http%3A%2F%2F10.81.224.161%3A8080%2Fturbine-web%2Fturbine.stream%3Fcluster%3Dalpha [HTTP/1.1 500 Internal Server Error 38ms]

I can't see any additional information in the server logs, though perhaps i'm not looking in the right place. I see output in logs/localhost.2013-03-20.log and log/catalina.out, but no errors.

I'm using the latest 0.5 snapshot code.

thanks, Mitchell

opuneet commented 11 years ago

Hi @mcacker

Can you try connecting to the turbine stream directly?

try this

curl "http://{turbine host name} : {tomcat port} / turbine.stream?cluster={cluster name}"

You should get the hystrix data on the cmd line.

Thanks, -Puneet

mcacker commented 11 years ago

i'm now seeing this error:

INFO: Deploying web application archive /opt/apache-tomcat-7.0.34/webapps/turbine-web.war Mar 21, 2013 3:45:32 PM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(/opt/tomcat/webapps/turbine-web/WEB-INF/lib/servlet-api-2.5.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. log4j:WARN No such property [httpclient] in org.apache.log4j.ConsoleAppender. Mar 21, 2013 3:45:36 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Error listenerStart Mar 21, 2013 3:45:36 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Context [/turbine-web] startup failed due to previous errors

according to the servlet spec (version 2.5, section 9.7.2 - not version 2.3 as indicated above) it looks like the webapp should not include servlet-api.jar:

"As described in the Java EE license agreement, servlet containers that are not part of a Java EEproduct should not allow the application to override Java SE platform classes, such as those in the java.* and javax.* namespaces, that Java SE does not allow to be modified. The container should not allow applications to override or access the container’s implementation classes."

I'm not sure if that is the reason that turbine-web is not starting, as there are a couple of other errors, but i'll see if i can build turbine-web without it.

mcacker commented 11 years ago

ok, removed servlet-api.jar from the build, and now that error is gone, however, turbine-web still will not start:

INFO: Deploying web application archive /opt/apache-tomcat-7.0.34/webapps/turbine-web.war SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. log4j:WARN No such property [httpclient] in org.apache.log4j.ConsoleAppender. Mar 21, 2013 4:13:26 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Error listenerStart Mar 21, 2013 4:13:26 PM org.apache.catalina.core.StandardContext startInternal SEVERE: Context [/turbine-web] startup failed due to previous errors

Any suggestions? guess i'll see if i can include "org.slf4j.impl.StaticLoggerBinder" and remove httpclient from log4j.properties.

mcacker commented 11 years ago

I see this error in another log file. it's puzzling as the jar which contains the class is included in the war:

SEVERE: Exception sending context initialized event to listener instance of class com.netflix.turbine.StartTurbineServer java.lang.NoClassDefFoundError: org/codehaus/jackson/PrettyPrinter at com.netflix.turbine.data.meta.AggDataMetaInfoAdaptor.(AggDataMetaInfoAdaptor.java:27) at com.netflix.turbine.monitor.cluster.AggregateClusterMonitor.(AggregateClusterMonitor.java:113) at com.netflix.turbine.monitor.cluster.AggregateClusterMonitor.(AggregateClusterMonitor.java:88) at com.netflix.turbine.monitor.cluster.AggregateClusterMonitor.findOrRegisterAggregateMonitor(AggregateClusterMonitor.java:80) at com.netflix.turbine.plugins.DefaultAggregatorFactory.initClusterMonitors(DefaultAggregatorFactory.java:72) at com.netflix.turbine.init.TurbineInit.init(TurbineInit.java:54) at com.netflix.turbine.StartTurbineServer.contextInitialized(StartTurbineServer.java:18) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:977) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1655) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.PrettyPrinter at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 21 more

opuneet commented 11 years ago

Hi @mcacker

opuneet commented 11 years ago

Hi @mcacker

Sorry sent the earlier response to soon. I'll try starting the webapp myself and see if I run into similar problems.

Thanks, -Puneet

opuneet commented 11 years ago

Hi @mcacker

opuneet commented 11 years ago

I'm unable to reproduce any error when running turbine-web.

I tried these steps

Ensure that the build.gradle for turbine-web looks like this, this will help run a local server

apply plugin: 'java' apply plugin: 'war' apply plugin: 'jetty'

dependencies {
    compile project(":turbine-core")

    compile 'log4j:log4j:1.2.17'
    compile 'org.slf4j:slf4j-log4j12:1.6.1'
}

jettyRun { httpPort = 7979 } build.gradle (END)

Add the 2 required properties to your server, you can directly edit the code if you have it downloaded, else just provide the properties

ConfigurationManager.getConfigInstance().setProperty("turbine.ConfigPropertyBasedDiscovery.default.instances", "PUT YOUR HOST NAME"); ConfigurationManager.getConfigInstance().setProperty("turbine.instanceUrlSuffix", "PUT YOUR HOST CONNECTION SUFFIX e.g :8080/mystream "); --> this is the hystrix stream

Then start turbine

cd turbine-web ../gradlew jetyRun

Check the Turbine stream

curl "http://localhost:7979/turbine-web/turbine.stream?cluster=default"

I got data from my server. Not sure what is wrong with your installation, you will have to give me more details so that I can help you here.

Thanks, -Puneet

mcacker commented 11 years ago

I am able to start Turbine if I don't configure my own Instance Discovery. When I do that I end up with java.lang.NoClassDefFoundError: org/codehaus/jackson/PrettyPrinter

It looks like i'm having to include all the jars in my jar file that are already included in Turbine. For example PrettyPrinter is already in the Turbine war.

I'm adding my jar to the Tomcat classpath in catalina.sh as follows:

CLASSPATH=$CLASSPATH:$CATALINA_HOME/lib/BobTurbine-jar-with-dependencies.jar

where the jar contains my Instance and dependencies. Is there a better place to add it to the classpath, where our jar won't have to include all the dependencies that Turbine already includes?

thanks, Mitchell

mcacker commented 11 years ago

btw, i have the properties you mention above set in tomcat's setenv.sh:

JAVA_OPTS="$JAVA_OPTS -Dturbine.aggregator.clusterConfig=dev,alpha" JAVA_OPTS="$JAVA_OPTS -DInstanceDiscovery.impl=com.pgi.bob.cache.BobServerInstances" JAVA_OPTS="$JAVA_OPTS -Dturbine.instanceUrlSuffix=:8080/BobApp/hystrix/hystrix.stream"

opuneet commented 11 years ago

I think that you are facing issues with packaging turbine-web, turbine-core along with your own tomcat based application.

The turbine-web module that is packaged and shipped with the Turbine release is more for demo purposes so that one can start playing with Turbine and seeing how it works. The build.gradle file dictates what dependencies to package in the turbine.war file and hence if you use turbine-web, then you have to ensure that you package all you dependencies properly along with turbine.war, which can make things pretty messy.

turbine-web module is a very thin wrapper around turbine-core. Hence you can easily duplicate it's logic in your own application ( war file ), instead of using the existing turbine.war file. I'll add documentation to the wiki to make this more clear.

Here is the entire code base in turbine-web.

import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

import org.apache.log4j.PropertyConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.netflix.config.ConfigurationManager;
import com.netflix.turbine.init.TurbineInit;

public class StartTurbineServer implements ServletContextListener {

    private static final Logger logger = LoggerFactory.getLogger(StartTurbineServer.class);

    @Override
    public void contextInitialized(ServletContextEvent sce) {
        logger.info("Initing Turbine server");
        TurbineInit.init();
    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        logger.info("Stopping Turbine server");
    }

Here is the entire web.xml content

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" version="2.5">

  <listener>
    <listener-class>com.netflix.turbine.StartTurbineServer</listener-class>
  </listener>

  <servlet>
    <description></description>
    <display-name>TurbineStreamServlet</display-name>
    <servlet-name>TurbineStreamServlet</servlet-name>
    <servlet-class>com.netflix.turbine.streaming.servlet.TurbineStreamServlet</servlet-class>
  </servlet>

  <servlet-mapping>
    <servlet-name>TurbineStreamServlet</servlet-name>
    <url-pattern>/turbine.stream</url-pattern>
  </servlet-mapping>

</web-app>

Here is what I suggest you should do

It will be easier for you to just package your own application and build your own turbine.war file. Follow these steps and everything should "just work"

Create your own web application, which also includes your own InstanceDiscovery implementation
Copy the StartTurbineServer code to your application.
Edit your web.xml to point to your StartTurbineServer
Build your war and deploy with tomcat along with the properties that you mentioned.

mcacker commented 11 years ago

thanks Puneet, I followed your instructions, built my own war, and it's all working now.

Mitchell

opuneet commented 11 years ago

Cool, closing this issue.

Netflix / Turbine