General Questions for Tomcat Webapps

hrzbrg commented 7 years ago

Hi,

I've been looking through the documentation for quite some time in order to get an idea of how I can fit stagemonitor into our Java Microservice environment. We mostly run Spring Boot applications in Docker. I already got from the docs, that there is no single jar (like NewRelic) that I can just load as a javaagent (apart from the byte-buddy that I would need for Java 8). Thats a pity, because we would have to touch all services manually instead of just providing a new base container that already has the javaagent inside. Nevertheless, a problem that can be solved.

What I did not get completely from the documentation are the following questions:

We mostly do not serve websites from our services. Is stagemonitor-web still the right dependency for that or will it make problems?
How does stagemonitor behave when the configured Elasticsearch cluster is not available? Does it interfere with the main application in a negative way?
Multiple containers of a service will run in our backend. How does stagemonitor group them?

Regards hrzbrg

felixbarny commented 7 years ago

Hi and thanks for your questions

stagemonitor-web is the correct dependency for servlet-based applications
I've added a section in the wiki about what happens if Elasticsearch goes down: https://github.com/stagemonitor/stagemonitor/wiki/Elasticsearch#what-happens-if-elasticsearch-goes-down
Stagemonitor by default groups by the tags application host and instance. See the wiki about how the hostname is resolved and manually set: https://github.com/stagemonitor/stagemonitor/wiki/Configuration-Options#host-name

hrzbrg commented 7 years ago

Thanks for your quick answers. Just got two follow up remarks.

Whats the timeline for the distributed tracing? I read that a new release is upcoming. It would be really cool to have that feature. Would it be possible that you push a snapshot so we could put it in a testing environment?
It would be great if one could modify the index where the Kibana dashboards are stored. The default is .kibana but we use a custom one.

felixbarny commented 7 years ago

Distributed tracing will be released in June. How do you modify the Kibana index name? In the Kibana advanced settings?

felixbarny commented 7 years ago

I've just uploaded a snapshot in the sonatype snapshots repo:

        <repository>
            <id>snapshots-repo</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases><enabled>false</enabled></releases>
            <snapshots><enabled>true</enabled></snapshots>
        </repository>

Feedback is much appreciated!

Could you create an issue in the stagemonitor repo about customizing the Kibana index?

hrzbrg commented 7 years ago

We just set it in the Kibana config kibana.index: "internal-kibana" so Kibana just looks in there for Settings.

Thanks for the Snapshot. We will try to fit a trial into this week 👍

hrzbrg commented 7 years ago

So I got the 0.80 snapshot in a service via the following dependencies:

        <!-- Stagemonitor -->
        <dependency>
            <groupId>org.stagemonitor</groupId>
            <artifactId>stagemonitor-web</artifactId>
            <version>0.80.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.stagemonitor</groupId>
            <artifactId>stagemonitor-tracing</artifactId>
            <version>0.80.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.stagemonitor</groupId>
            <artifactId>stagemonitor-tracing-elasticsearch</artifactId>
            <version>0.80.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.stagemonitor</groupId>
            <artifactId>stagemonitor-jdbc</artifactId>
            <version>0.80.0-SNAPSHOT</version>
        </dependency>

In Elasticsearch I can see stagemonitor-metrics and stagemonitor-spans but the stagemonitor-spans data is pretty unspectacular:

----------------------------------------------------------------------
Selftime (ms)              Total (ms)                 Method signature
----------------------------------------------------------------------
000021.43  100% ██████████ 000021.43  100% ██████████ GET /v1/fixedfares

I saw that you deprecated the requestmonitor. Where might I find that tracing data now thats also depicted in https://github.com/stagemonitor/stagemonitor/wiki/Request-Analysis-Dashboard#request-details ?

felixbarny commented 7 years ago

The dependencies are correct.

This data should be in the stagemonitor-spans- index. Is only the call tree empty or are also other fields like header. missing?

If only the call tree is empty, you probably have to set stagemonitor.instrument.include.

hrzbrg commented 7 years ago

stagemonitor.instrument.include is set to my package, however there is no call trace data.

felixbarny commented 7 years ago

I'm sorry to hear that. There is another user which has problems with the call tree: https://github.com/stagemonitor/stagemonitor-mailinglist/issues/41. Did you try the suggestions I made there?

felixbarny commented 7 years ago

So the other information is there as expected? Does the Kibana Request Analysis dashboard work?

hrzbrg commented 7 years ago

I got the snapshot to work now. The stupid issue was, that the byte buddy agent was not loaded because of a typo. Traces are now visible in Kibana. Thanks for your help so far. Will report more once we get into distributed tracing.

felixbarny commented 7 years ago

Do you specify the byte buddy agent via -javaagent? Actually, that should not be needed as stagemonitor attaches the agent itself at runtime.

Sebastian notifications@github.com schrieb am Mi., 31. Mai 2017 19:23:

I got the snapshot to work now. One stupid issue was, that the byte buddy agent was not loaded because of a typo. Traces are now visible in Kibana. Thanks for your help so far. Will report more once we get into distributed tracing.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/stagemonitor/stagemonitor-mailinglist/issues/44#issuecomment-305257699, or mute the thread https://github.com/notifications/unsubscribe-auth/ACEDCNRp1R0QPgYyFZIfGo5eslz00VP9ks5r_aIVgaJpZM4NoaGf .

hrzbrg commented 7 years ago

It only worked after I specified it correctly as -javaagent in the Java Opts (Java 8).

felixbarny commented 7 years ago

There are no known issues with runtime attachment and Java 8. Did you get any error?

josh-cain commented 6 years ago

I'm running into the same thing. My setup is a little wonky, as the deployment is a JBoss module. However, I've tried allowing runtime attachment and manually specifying byte buddy via a -javaagnet param, and neither apporach works.

As described above ^, I can see all the data hitting the stagemonitor-spans-* index, but the trace just winds up like this:

---------------------------------------------------------------------- 
Selftime (ms) Total (ms) Method signature 
---------------------------------------------------------------------- 
000014.00 100% ██████████ 000014.00 100% ██████████ GET /myUrl

In an effort to get something going, here are my properties:

stagemonitor.profiler.active=true stagemonitor.web.widget.enabled=true stagemonitor.internal.monitoring=true stagemonitor.instrument.debug=true stagemonitor.elasticsearch.configurationSource.deactivateStagemonitorIfEsIsDown=false stagemonitor.instanceName=mybox stagemonitor.applicationName=myapp stagemonitor.profiler.logCallStacks=true stagemonitor.instrument.include=my.package stagemonitor.elasticsearch.url=http://localhost:9200 stagemonitor.profiler.minExecutionTimeNanos=500

Any thoughts?

felixbarny commented 6 years ago

Which versions of Java, stagemonitor and JBoss are you using? Any framewoks? Is /myUrl a static resource or a Spring MVC controller method or something else?

josh-cain commented 6 years ago

openjdk version "1.8.0_91"
stagemonitor 0.81.0
No frameworks, but web app is deployed as a JBoss module
/myUrl is a RestEasy Webapp Root

felixbarny commented 6 years ago

And the resteasy endpoint is inside the my.package package?

josh-cain commented 6 years ago

That's correct.

felixbarny commented 6 years ago

Could you post the # stagemonitor status checks which are logged on startup?

josh-cain commented 6 years ago

[2017-08-04 20:49:37,963+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) # stagemonitor status
[2017-08-04 20:49:37,965+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - CorePlugin (CorePlugin is initialized)
[2017-08-04 20:49:37,965+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - ElasticsearchTracingPlugin (ElasticsearchTracingPlugin is initialized)
[2017-08-04 20:49:37,965+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - JdbcPlugin (JdbcPlugin is initialized)
[2017-08-04 20:49:37,965+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - JvmPlugin (JvmPlugin is initialized)
[2017-08-04 20:49:37,966+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - OsPlugin (OsPlugin is initialized)
[2017-08-04 20:49:37,966+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - ServletPlugin (ServletPlugin is initialized)
[2017-08-04 20:49:37,966+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - TracingPlugin (TracingPlugin is initialized)
[2017-08-04 20:49:37,966+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) OK   - elasticsearch 
[2017-08-04 20:49:37,966+0000] INFO  [org.stagemonitor.core.Stagemonitor] (MSC service thread 1-8) FAIL - instrumentation_datasource (No DataSource instrumented)

felixbarny commented 6 years ago

There is a spelling error in -javaagnet. Could that be the problem?

Otherwise, its strange that there is not log entry like that: INFO Stagemonitor - OK - instrumentation

Could you start debugging here: https://github.com/stagemonitor/stagemonitor/blob/0.81.0/stagemonitor-core/src/main/java/org/stagemonitor/core/instrument/AgentAttacher.java#L76

And see why it does not at least get to here: https://github.com/stagemonitor/stagemonitor/blob/0.81.0/stagemonitor-core/src/main/java/org/stagemonitor/core/instrument/AgentAttacher.java#L104

josh-cain commented 6 years ago

@felixbarny Thanks for taking a look at that! I've got to move on to some other things today, but will try to get back to it when I can.

yucigou commented 6 years ago

Seems that the Call Tree is empty for the live demo: https://stagemonitor-demo.isys-software.de/. I tried changing the setting of Included packages dynamically to de, net, org, but the Call Tree is always empty.

felixbarny commented 6 years ago

How embarrassing 😨 thx for reporting

felixbarny commented 6 years ago

its fixed now

stagemonitor / stagemonitor-mailinglist

General Questions for Tomcat Webapps #44