Need resource timings in EUM

gopi-nath commented 6 years ago

In EUM request that is sent to backend, even the time taken to fetch resources(like .css, .js etc) also goes in "res" parameter.

Sample EUM Request: https://domain.com/stagemonitor/public/eum?ty=pl&r=1512468205524&t=cda31cbaf2d26c17&bt=a3d8f66db36e0572&u=https%3A%2F%2Fdomain.com%2Fdashboard%2Fhealth&m_bs=a3d8f66db36e0572&ts=-2624&d=3290&t_unl=0&t_red=0&t_apc=0&t_dns=0&t_tcp=0&t_req=288&t_rsp=12&t_pro=2985&t_loa=0&t_fp=2383&res=%7B%22https%3A%2F%2F%22%3A%7B%435353312323d760877a9d-2859881f2927fb4a16ba2601e4f1470b.ssl.cf1.rackcdn.com%2Fdist-uat%2F%22%3A%7B%22%2Fmyweb-%22%3A%7B%225d4c2635c1dce9b9adf6c1c7faf434ac.min.css%3Ffef434323234aa2cb298%22%3A%5B%22-2279%2C564%2C2%22%5D%2C%22d910242052d92606b131cdcb8f122abe.min.css%3Ffef434323234aa2cb298%22%3A%5B%22-2279%2C1618%2C2%22%5D%7D%2C%22vendor-4896ce53355a91ef67f2.bundle.js%3Ffef434323234aa2cb298%22%3A%5B%22-2278%2C2639%2C3%22%5D%2C%22r-caf7218e12db5701c968.bundle.js%3Ffef434323234aa2cb298%22%3A%5B%22-2277%2C949%2C3%22%5D%2C%22myweb-8c9f808fd59e2f9ebd01.min.js%3Ffef434323234aa2cb298%22%3A%5B%22-2277%2C1916%2C3%22%5D%7D%2C%22fonts.googleapis.com%2Fcss%3Ffamily%3D%22%3A%7B%22Roboto%3A300%2C400%2C500%2C700%22%3A%5B%22-2283%2C339%2C2%2C3%2C1081%22%5D%2C%22Lato%3A400%2C700%2C400italic%2C700italic%26subset%3Dlatin%22%3A%5B%22-458%2C175%2C4%2C3%2C546%22%5D%7D%2C%22cdnjs.cloudflare.com%2Fajax%2Flibs%2F%22%3A%7B%22semantic-ui%2F2.2.12%2Fsemantic.min.css%22%3A%5B%22-2283%2C1808%2C2%22%5D%2C%22noUiSlider%2F9.2.0%2Fnouislider.css%22%3A%5B%22-2282%2C531%2C2%22%5D%2C%22toastr.js%2Flatest%2Ftoastr.min.css%22%3A%5B%22-2281%2C523%2C2%22%5D%7D%2C%22domain.com%2F%22%3A%7B%22css%2Freact%2Fdashboard.css%22%3A%5B%22-2281%2C329%2C2%2C3%2C739%22%5D%2C%22stagemonitor%2Fpublic%2Feum.js%22%3A%5B%22-280%2C277%2C3%2C3%2C2964%22%5D%7D%7D%7D

We need this data also in the backend(ES).

How do we get this also in ES ?

felixbarny commented 6 years ago

Currently, this data is not set to Elasticsearch. Could you describe you use case a bit more? How do you want to analyze the data and what is it you are interested in?

gopi-nath commented 6 years ago

Use-case: In case the page load time is high, I want to see which resource consumes more time generally in that request. I can have a percentile table for all such resources in Kibana and then filter out one particular request url in that. which would show me all the resources that were fetched for the request url and their time percentiles.

gopi-nath commented 6 years ago

@felixbarny : any updates ? can it be done? I tried to put the string as it is in ES. To get a better version of this, I would have to revert the stringify of the metrics which comes from the js

felixbarny commented 6 years ago

To be honest, this is not currently on our near term list. But you could totally to this on your own. I'd be more than happy to guide you as well.

I'd start with adding a custom org.stagemonitor.web.servlet.eum.ClientSpanTagProcessor which reads the res parameter from the pageload beacon.

You can parse the Json with JsonUtils.getMapper().readValue(String content) and modify it as you like.

Another approach would be to store the string as is but to enhance the stagemonitor Kibana plugin to visualize the resource timings accordingly.

The disadvantage of the above approaches is that you cant draw statistics across different page loads, like load time percentiles of a particular script.

To do that, you'd have to create individual spans for each resource and attach them to the pageload span by setting their parent_id to the span_id of the pageload span.

gopi-nath commented 6 years ago

@felixbarny sure, I'll try to do it.

Currently the issue is structure of the data coming from the eum.js:

res: {
  "https://": {
    "2d5de7a683d760877a9d-.ssl.cf1.rackcdn.com/dist-uat/": {
      "/myweb-": {
        "5d4c2635c1dce9b9adf6c1c7faf434ac.min.css": [
          "-2279,564,2"
        ],
        "d910242052d92606b131cdcb8f122abe.min.css": [
          "-2279,1618,2"
        ]
      },
      "vendor-4896ce53355a91ef67f2.bundle.js": [
        "-2278,2639,3"
      ],
      "r-caf7218e12db5701c968.bundle.js": [
        "-2277,949,3"
      ],
      "myweb-8c9f808fd59e2f9ebd01.min.js?": [
        "-2277,1916,3"
      ]
    },
    "fonts.googleapis.com/css?family=": {
      "Roboto:300,400,500,700": [
        "-2283,339,2,3,1081"
      ],
      "Lato:400,700,400italic,700italic&subset=latin": [
        "-458,175,4,3,546"
      ]
    },
    "cdnjs.cloudflare.com/ajax/libs/": {
      "semantic-ui/2.2.12/semantic.min.css": [
        "-2283,1808,2"
      ],
      "noUiSlider/9.2.0/nouislider.css": [
        "-2282,531,2"
      ],
      "toastr.js/latest/toastr.min.css": [
        "-2281,523,2"
      ]
    },
    "emyweb/": {
      "css/react/dashboard.css": [
        "-2281,329,2,3,739"
      ],
      "stagemonitor/public/eum.js": [
        "-280,277,3,3,2964"
      ]
    }
  }
}

The name of the resource file itself is nested in this. The actual file names are:

gopi-nath commented 6 years ago

@felixbarny Is there any way I could get the data as : res: { https://2d5de7a683d760877a9d-.ssl.cf1.rackcdn.com/dist-uat/myweb-5d4c2635c1dce9b9adf6c1c7faf434ac.min.css : [xx,xxx,xxxx] ,

https://2d5de7a683d760877a9d-.ssl.cf1.rackcdn.com/dist-uat/myweb-d910242052d92606b131cdcb8f122abe.min.css : [yy,yyy,yyyy], . . }

felixbarny commented 6 years ago

You'll have to transform the data and concatenate the path according to the nesting. If the format seems a bit strange, it's to save bandwidth when sending this information back to the server.

gopi-nath commented 6 years ago

@felixbarny This is done.

I have looped through the string, transformed the data and concatenated the paths accordingly. Each string is assigned its corresponding duration.

For now this should be enough. This would give an insight on which resource has consumed more time in the request. I can check this whenever the processing time is more than a threshold.

stagemonitor / stagemonitor-mailinglist

Need resource timings in EUM #62