rschoessler commented 5 years ago

99 seems to have a similar problem, but that solution is not working for me.

Here's my setup:

boomerang -> boomcatch -> telegraf -> influxdb

My webpage is pretty simple and includes the following in the <head>:

 <script src="assets/js/boomerang/boomerang.js"></script>
  <script src="assets/js/boomerang/plugins/rt.js"></script>
  <script src="assets/js/boomerang/plugins/painttiming.js"></script>
  <script>
    BOOMR.init({
      beacon_url: "http://localhost:8080/beacon",
      PaintTiming: {
        enabled: true,
        clearOnBeacon: true
      }
    });
  </script>

It doesn't matter if I use painttiming.js or navtiming.js or usertiming.js, the only data that is ever sent BY boomcatch is rt.firstbyte, rt.lastbyte and rt.load:

2019-03-28 10:44:15 INFO boomcatch: sending rt.firstbyte:315|ms
rt.lastbyte:3159|ms
rt.load:3159|ms

2019-03-28 10:44:15 INFO boomcatch: sent 56 bytes

If I debug the browser/sources in developer tools and break on boomerang:3372:

BOOMR.sendBeaconData(varsSent);

I can see that the timings that I am requesting are being sent:

The rt data gets into my database, but nothing else.

Ideally I want to get usertiming as I have created custom user timing marks that I am most interested in

What am I missing??? Thanks in advance.

cc: @josebolos

rschoessler commented 5 years ago

@josebolos would you be able to assist me with this? Asking directly as I see that you are the most (only?) contributor to this project of late. Thanks if you are able to help.

josebolos commented 5 years ago

Hi @rschoessler,

You haven't attached boomcatch's config so I'm just guessing here. What mapper are you using?

If you're using the default mapper, which is statsd, I think that this mapper only translates certain data, as you can see in the statsd mapper docs.

Each boomcatch user has different needs in terms of what needs to be mapped and sent, so the recommended course of action is to write your own mapper. It's quite easy to do!.

Before starting to write your own, I would suggest to check the unmapped mapper just to confirm that this is indeed the problem.

Hope that it helps!

rschoessler commented 5 years ago

@josebolos I'm running boomcatch from command line. Essentially, I was using the default mapper which I think is statsd. That is ONLY sending the rt data from my original post over to telegraf (which forwards to influxdb). I've tried the unmapped mapper and that gets an error on the telegraf side:

Error: splitting '|', Unable to parse metric: {"data":

...which I assume is because the format is not statsd which telegraf is expecting.

Here's my command (for statsd...have switched to unmapped, doesn't work):

H:\>boomcatch --host localhost --port 8080 --maxSize 2048 --mapper statsd

...which sends:

2019-04-05 10:06:47 INFO boomcatch: sending rt.firstbyte:311|ms
rt.lastbyte:2144|ms
rt.load:2144|ms

2019-04-05 10:06:47 INFO boomcatch: sent 56 bytes

I've tried to use statsd but seems like it has issues sending to my version of influxdb. I'm still testing things out, so I'm open to other suggestions on what boomcatch should send to that will ultimately get it into influxdb...would you have any suggestions?

Thanks!

rschoessler commented 5 years ago

Also based on the statsd mappers documentation it would seem that the navtiming data should be supported by my current setup/approach (i.e. rt is making it, but navtiming is not).

rschoessler commented 5 years ago

@josebolos ,

I've changed my configuration to use boomerang -> boomcatch -> statsd/graphite docker.

I can see that the navtiming data is going from the browser and into boomcatch by adding in a little logging:

2019-04-11 16:31:55 INFO boomcatch: state.body: rt.start=navigation&rt.bmr=401%2C92%2C67%2C44&rt.tstart=1555014709659&rt.bstart=1555014710324&rt.end=1555014715403&t_resp=312&t_page=5432&t_done=5744&t_other=t_domloaded%7C4165%2Cboomerang%7C5095%2Cboomr_fb%7C665&rt.tt=297186&rt.obo=4&ntnav st=1555014709659&nt_fet_st=1555014709663&nt_dns_st=1555014709667&nt_dns_end=1555014709667&nt_con_st=1555014709667&nt_con_end=1555014709969&nt_req_st=1555014709969&nt_res_st=1555014709971&nt_res_end=1555014709972&nt_domloading=1555014710005&nt_domint=1555 014713822&nt_domcontloaded_st=1555014713822&nt_domcontloaded_end=1555014713823&nt_domcomp=1555014715393&nt_load_st=1555014715393&nt_load_end=1555014715403&nt_unload_st=1555014709988&nt_unload_end=1555014709988&nt_enc_size=1706&nt_dec_size=1706&nt_trn_siz e=210&nt_red_cnt=0&nt_nav_type=1&u=http%3A%2F%2Flocalhost%3A4200%2F&r=http%3A%2F%2Flocalhost%3A4200%2F&v=%25boomerang_version%25&rt.si=u0w74xfvwk-ppt6n7&rt.ss=1555006435083&rt.sl=42&vis.st=visible&ua.plt=Win32&ua.vnd=Google%20Inc.&pid=dglw42zw&n=1&sb=1

-- log that I added for mappedData 2019-04-11 16:31:55 INFO boomcatch: mappedData: rt.firstbyte:312|ms rt.lastbyte:5744|ms rt.load:5744|ms

-- what is actually sent 2019-04-11 16:31:55 INFO boomcatch: sending rt.firstbyte:312|ms rt.lastbyte:5744|ms rt.load:5744|ms

So, this confirms for me that boomcatch seems to be either filtering out or ignoring the navigation timing data that is sent in.

Again, the command that I run from my terminal is: boomcatch --host localhost --port 4201 --mapper statsd --path /beacon

What am I missing?

vedranf commented 5 years ago

Hello,

I think i have the same/similar issue. With unmapped, I see that data is being received:

2019-04-19 13:48:13 INFO boomcatch: sending {"data":{"restiming":"{\"https://\":{\"www.\" .... 2019-04-19 13:48:13 INFO boomcatch: sent 6543 bytes

but upon switching to statsd, this is all I get:

2019-04-19 13:50:21 INFO boomcatch: referer=https://www.../ user-agent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36 address=127.0.0.1[...] method=POST url=/ 2019-04-19 13:50:21 INFO boomcatch: sent 0 bytes ...

I run it like this:

$ ./src/cli.js --host 127.0.0.1 --port 2222 --workers 1 --mapper statsd --path / --forwarder console

Also, with waterfall I get the same error as in #86. Is it possible that boomcatch fails to parse the data? Perhaps a version mismatch with boomerang? Or it's sending invalid json? This "restiming" key appears to be json embedded as a string with quotes escaped. I'm not sure if that is expected.

Thanks, Vedran

vedranf commented 5 years ago

Hello,

I just found examples under client/ directory. That worked, however, format being use there doesn't match what's described in: https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.ResourceTiming.html

That boomerang uses json and trie structure, while boomcatch seems to uses application/x-www-form-urlencoded key-values pairs.

Regards, Vedran

aftabnaveed commented 5 years ago

Before forwarding the data you probably need to decompress it via

var ResourceTimingDecompression = require('resourcetiming-compression').ResourceTimingDecompression
ResourceTimingDecompression.decompressResources(JSON.parse(rt)));

That kind of works but I am having an issue with it which is specified here https://github.com/nicjansma/resourcetiming-compression.js/issues/28

genesis-teddy commented 4 years ago

Before forwarding the data you probably need to decompress it via
var ResourceTimingDecompression = require('resourcetiming-compression').ResourceTimingDecompression
ResourceTimingDecompression.decompressResources(JSON.parse(rt)));
That kind of works but I am having an issue with it which is specified here nicjansma/resourcetiming-compression.js#28

Which files should have this? Does anyone have a patch?

nevesing commented 4 years ago

I am also having the same problem. Any help would be appreciated.

genesis-teddy commented 4 years ago

I am also having the same problem. Any help would be appreciated.

https://github.com/genesis-teddy/boomcatch/wiki/Notes

nevesing commented 4 years ago

@genesis-teddy - Thanks for the info. So I reinstalled Boomcatch using your repo and I am able to see metrics from RT and ResTiming BUT not from NT yet:

npm install -g https://github.com/genesis-teddy/boomcatch

Startup cmd: boomcatch --prefix Jun17 --port 7000 --host 0.0.0.0 --path / --mapper statsd --forwarder udp --forwardHost mystatsd_host

BOOMR.init():

e.detail.BOOMR.init({ beacon_url: "https://myboomcatch_host", beacon_type: "GET", ResourceTiming: { enabled: true, clearOnBeacon: true }, RT: { enabled: true, cookie: "Cookie-Sample" }, NavigationTiming: { enabled: true } });

Console output: It shows metrics related to RT and ResTiming only. Tried disabling ResTiming or RT and I dont see metrics for NT. It is not the beacon issue as the 'unmapped' mapper shows all the metrics.

genesis-teddy commented 4 years ago

@genesis-teddy - Thanks for the info. So I reinstalled Boomcatch using your repo and I am able to see metrics from RT and ResTiming BUT not from NT yet:

npm install -g https://github.com/genesis-teddy/boomcatch

Startup cmd: boomcatch --prefix Jun17 --port 7000 --host 0.0.0.0 --path / --mapper statsd --forwarder udp --forwardHost mystatsd_host

BOOMR.init():

e.detail.BOOMR.init({ beacon_url: "https://myboomcatch_host", beacon_type: "GET", ResourceTiming: { enabled: true, clearOnBeacon: true }, RT: { enabled: true, cookie: "Cookie-Sample" }, NavigationTiming: { enabled: true } });

Console output: It shows metrics related to RT and ResTiming only. Tried disabling ResTiming or RT and I dont see metrics for NT. It is not the beacon issue as the 'unmapped' mapper shows all the metrics.

Glad it helped. I have not tried NT on my POC, so I have nothing to add, other than the documentation indicates it is available in the statsd mapper. Kindly share what you find out 🙂

nevesing commented 4 years ago

I dont have much experience in development or javascript so I am not sure.

genesis-teddy commented 4 years ago

I dont have much experience in development or javascript so I am not sure.

I’m curious to test, but too busy to help right now. Could you share a screenshot of grafana tree?

nevesing commented 4 years ago

Thanks for showing Interest. I am not sure what do you mean by grafana tree screenshot. NT shows NavigationTiming API metrics.

genesis-teddy commented 4 years ago

Thanks for showing Interest. I am not sure what do you mean by grafana tree screenshot. NT shows NavigationTiming API metrics.

648BFFC8-15B9-429D-AAB7-42C1FDCCD76D

nevesing commented 4 years ago

Oh ok. This screenshot is from Graphite for RT and ResTiming. Similarly I should see for NT as well once it is fixed. Also if I capture metrics from 10 different applications, how does Graphite differentiate metrics by each app I dont know.

genesis-teddy commented 4 years ago

Oh ok. This screenshot is from Graphite for RT and ResTiming. Similarly I should see for NT as well once it is fixed. Also if I capture metrics from 10 different applications, how does Graphite differentiate metrics by each app I dont know.

Yes, Graphite. I got the two mixed up 😅 Somewhere in boomcatch, there’s a reference to a prefix.

springernature / boomcatch

boomcatch sends only `rt` to telegraf (statsd) #111

99 seems to have a similar problem, but that solution is not working for me.