Realtime User Monitoring

[lukegalea]

We'd like to use RUM, but it appears that the node client doesn't currently support this.

Can we manually add the JS in?

[othiym23]

Unfortunately, no. The tricky part of adding end user monitoring to Node applications is not the client JS, but figuring out how to do the necessary header manipulation and script tag insertion transparently, without breaking streaming or your existing middleware stack (or your apps, for that matter - RUM adds significant complexity). As such, RUM has never been slated to be a 1.0 feature for the Node agent, and will probably be added (at first) as an explicitly invoked middleware somewhere down the line.

That said, you're one of the first testers to ask for RUM, so if you could tell us a little about your stack and your requirements, that would help us plan for the addition of RUM and RUM-like features down the road. Thanks!

[lukegalea]

Thanks for the quick response.

Our app is a backbone client, connecting to an expressjs server side.. so the middleware wouldn't work for us. We'd need a way to manually bundle RUM into our client distribution.

We aren't using yeoman or brunch, but plan to migrate eventually. If there was a middleware I bet that would cover many users. But a solution that worked with Yeoman would be more inline with what we need.

Alternatively, a middleware that served the js in a similar manner to how socket.io serves the js required for it's client would work. Then we could just put a script tag in referencing /rum-client.js which would be generated by the middleware.

Or am I missing something?

[joeblau]

@othiym23 I was interested in RUM as well. I'm running http://gitignore.io and the stack is an express app with a static jade web page.

[dkleehammer]

I too would like RUM. I'm running expressjs and EJS templates. With connect middleware being the most common, you could probably wrap the middleware you'd cover a lot of users. You could also do it as a use function in custom frameworks, wrapping the middleware stack.

I'd be in for testing this functionality.

[rtgibbons]

This is big one for us, we are using it for several clients. We primarily are using Geddy with swig template processing.

I would be happy with a global namespaced function that would provide us the script tags so we can place them in our templates.

[othiym23]

@rtgibbons You highlight one of the main issues with bringing RUM to Node: you're the first person I've encountered who wants to use New Relic with Geddy (you are among the first people I've ever encountered using Geddy who don't work at Yammer ;). The Node ecosystem is huge, and we do have to come up with someway of adding features to the agent that follows the 80/20 rule, which, based on our internal metrics, means prioritizing Express.

Your idea's a good one, though, so thanks! There's a bunch of supporting work that would have to be done first (to extract RUM data from HTTP headers and / or cookies, to ensure that the RUM transaction is visible to the agent at the point at which you're requesting the script tags, and to do a bunch of stuff specific to New Relic's current implementation of RUM), but that's certainly a simpler and more general interface than offering a middleware. I'll keep it in mind at such point as we decide to implement RUM for Node.

[rtgibbons]

No problem; That's kinda what I figured, which is why I decided to comment.

If I think of anything else, I'll let you know.

[jsilveira]

+1. Using Express+Jade

[EdJ]

I have been thinking about this, and I'm not sure full transparency is really achievable in node.js. I've attempted to write a library that overwrote the http object with some additional transparent functionality in the past, and it proved difficult to cover all the edge cases, leading me to abandon that project. I also believe that the fragmentation in the node community and libraries means there's not truly a one-size-fits-all method for implementing anything.

However, I do believe that a method could be provided with a function signature of something like:

// request and response are the standard http.request and http.ServerResponse.
void addRum(request, response) {}

This would allow a simple implementation of a connect middleware module that could do the relevant header parsing, as well as providing an understandable API for those who choose not to use connect.

This obviously leaves the issue of injecting the RUM tag. As the view engines are so numerous, and not guaranteed to be used, I again propose that a simple method would be appropriate:

string getRumTags(request, response) {}

The output of this could then be manually added to a view in many cases, and in many cases could be automatically injected into the HTML view using some conditional logic, or a direct method call. I think you'd find that industrious node community members quickly integrate these two functions into their libraries of choice :)

Obviously, this may be a little too intrusive for many users, and someone smarter than me may be able to work out how to automatically inject the RUM script tag to the response, but I don't believe that sits nicely with the level of control node.js gives over the HTTP server.

I was interested to attempt to implement something like this over the weekend, already being very familiar with Compuware's Gomez browser RUM solution, however, there doesn't seem to be an open-sourced version of the New Relic RUM API. Is there a spec out there somewhere so I could knock together a test / example version?

[pkorac]

+1 Using express and ejs

[othiym23]

This is definitely one of the top items on the list of features to add to New Relic for Node. Just so anyone who comes across this ticket knows, though, we're probably going to do it piecewise, and the early implementation will probably be pretty manual; most likely it will be an API call that returns the necessary headers and cookie values for you to use with whatever middleware / templating stacks you have handy.

More to come!

[pkorac]

That would be just fine for the time being. Hope it gets released soon. Keep up the good work guys!

[wallies]

Not sure if this relates to the issue or not, but I have been trying to get RUM working with my Nodejs agent, as I have angularjs on the client side. Ive been trying to use this https://github.com/uken/newrelic-timing, I dont know whether newrelic is just not enabling the browser tab or Im not sending the right data. Any help

[othiym23]

That module was designed to be used with our Rails integration (or one of the other agents that currently has support for RUM built in). There are some changes we'll have to make to the agent just to get the basic pieces in place for modules like that to work.

Basic support for RUM will be coming soon! I promise!

[traviskaufman]

Big +1 to this as well and thanks @othiym23 for your active participation in this issue!!

[jperl]

Any update on the RUM support for New Relic? At work (Angie's List) we just added a node express app, and we would love to have RUM data.

[brunkle]

Hi Jperl, We are working on RUM support and hope to have it in your hands soon! Stay tuned!

[jperl]

That is great news. Thanks!

[shawn-crossley]

@brunkle - We are looking forward to this Build.com

[jfeust]

Another big +1. Business is starting to require RUM metrics and doesn't understand why our Node app can't have them. @brunkle - any ETA on this?

Stack: Node, Express, Jade

[nuc]

+1 (node, express, jade)

[henridf]

We'd love to be able to turn this on at sessionbox.com too.

[felipemorais]

I'd love to be able to migrate ouro java server to node. But rum data is a high level priorit.

[jperl]

Any update? I'm scared they are going to make me investigate Gomez XF soon.

[chalkup]

+1 (node, express, jade, less)

[othiym23]

Sooooon. Less than a month, and maybe much sooner than that. It will be pretty hands-on to start, though!

[chalkup]

Thats great news. Thanks for keeping us updated!

[Reggino]

+1 awesome

[nuc]

The new browser monitoring is really helpful!!

It would be super awesome to have it on our node / js apps..

[othiym23]

Stay tuned! Not long now...

[othiym23]

Manual RPM / browser monitoring stub generation has been added in New Relic for Node.js, which is available today! :tada: Check the README for details.

[robbiet480]

@othiym23 Woo! Just integrated it, using your exact instructions in README, but no dice. Seeing <!-- NREUM: (4) --> in header, but no script loads. I can open another issue if you think it's relevant, but I wasn't sure if you may want to keep initial implementation issues in this ticket instead.

[alicegoldfuss]

Hi @robbiet480 that message means our agent hasn't finished its handshake with the collector. So, your app was started and a user/you hit the site right away before our collector could talk to our agent. Give it a few minutes and it should resolve.

[groundwater]

Hi @robbiet480, the header generation has a circuit-breaker that will output a semi-informative comment in the event there was not enough information to generate the header properly.

(4) indicates there was a problem handshaking with our server and receiving a valid application id.

This could either be:

1. you loaded the browser page before the agent made its initial connection to New Relic
2. you have an invalid license key
3. another problem that would result in a lack of application id

If you place your logs into trace level, and send them to support@newrelic.com we can help sort that out for you.

[robbiet480]

@alicegoldfuss @groundwater 10 points for replying to me within 5 seconds of each other!

Yeah, I increased the log level to trace and realized that I wasn't setting a license key on localhost, because normally that's done by Heroku. Adding a key fixed this issue.

Thanks for the quick assist.

[groundwater]

@alicegoldfuss has her finger on the trigger!

@robbiet480 glad that worked for you

[shawn-crossley]

@othiym23 and @groundwater Thanks for the hard work. Can't wait to see the results after we let it run all weekend

[groundwater]

Thanks @shawn-crossley, and I encourage everyone to share any interesting traces with us, you can email jacob@newrelic.com with a link to your trace and I'll pass it along to the team.

[jfeust]

Possible slow memory leak when using with cluster. Our master cluster process usually hangs around 30 MB, after upgrade and changes it was up to 130 MB in a few hours. Got heroku dyno R14's. Reverted for now. Will try to look into more tomorrow.

We have the var newrelic = require('newrelic'); outside of all express-cluster blocks at the top of the app.js file.

[othiym23]

Keep us posted, @jfeust. The module creates a lot of ephemeral objects, but it shouldn't be retaining more than a few MiB over the long term, as the metrics are aggregated in real time and it doesn't retain many / any references to large resources.

[jfeust]

Well I tried only upgrading to 1.4.0 (not implementing RUM in the views or any other changes) and we're definitely seeing a slow but steady memory leak.

In 8 hours our dynos that are normally 300mb are up to 480 and growing. Our cluster master worker that's normally 30mb is up to 120mb.

Anyone else using this with cluster? Things were perfectly stable on 1.3.2.

[groundwater]

Hi @jfeust, I definitely will check this out. We do not want to be causing memory leaks!

Is there any chance you can give me a snapshot of your dependencies in package.json, or even better send support@newrelic.com an email with a link to you account.

Where are you requiring the newrelic module? Any details that can help run our own test will help immensely!

[jfeust]

@groundwater here you go.

"dependencies": {
"analytics-node": "0.5.0",
"connect-flash": "0.1.1",
"express": "3.4.6",
"jade": "~1.1.5",
"logfmt": "0.18.1",
"mongodb": "1.4.x",
"mongoose": "3.8.3",
"request": "2.30.0",
"underscore": "1.5.2",
"newrelic": "1.3.2",
"express-cdn": "https://registry.npmjs.org/express-cdn/-/express-cdn-0.1.9.tgz"
},

We have the require at the very first line of our app.js file, outside of any cluster code

// New Relic require needs to be at the top to capture everything
var newrelic = require('newrelic'),
  _ = require('underscore'),
  cluster = require('cluster'),
  Log = require('./app/tools/log'),
  settings = require('./app/settings');

// Code to run if we're in the master process
if (cluster.isMaster) {
    var workers = {};
    ...

[groundwater]

Thanks @jfeust, I'll get back to you soon.

[jfeust]

Hey @groundwater. Any update? Have you able to recreate the leak?

[groundwater]

@jfeust nothing as dramatic as you're experiencing. I created a sample app, trying to include the modules you listed, and ran it on SmartOS for several hours, hitting it with requests constantly.

I used gcore to dump the running core, and searched through the ruins with mdb and ::findjsobjects. There was definitely a small, but growing collection of Array objects that referred to components in the async-listener module, but they were small, and no other objects were retaining them. After several hours they were on the order of 1mb only.

I have a few more ideas, and am definitely on the case, but nothing so far.

If you feel like sharing more of your application setup, I would be appreciative. You can email me at jacob@newrelic.com. We treat customer privacy very seriously, and it would not be shared outside of our node team. I certainly understand if you're not comfortable with that, and will still look to reproduce the issue regardless.

If you're interested in trying out mdb spelunking, I can also give you some basics.