Propose Page speed reports, Load time analytics

robocoder commented 14 years ago

In Piwik 1.12 we are shipping Page Speed Report in Piwik! A new column "Average Response time" is displayed in the Pages Urls and Page title reports. This column is the time it takes for server to generate the response and for the visitor to download it.

The historical view of the time can be seen via Row evolution feature.

The average time across all pages is plotted on Visitor > Overview.

Time tracking works for all modern browsers, who most implement the Dom timing API.

http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html

Enjoy this new simple page speed report in Piwik.

robocoder commented 13 years ago

Attachment: changes to piwik.js 1700.patch

anonymous-matomo-user commented 12 years ago

Attachment: Site Performance plugin (work in progress) SitePerformance.zip

mattab commented 11 years ago

Attachment: showing requests with wrong performance time output.png

mattab commented 13 years ago

See also #1985

mattab commented 13 years ago

This was released in GA: http://www.google.com/support/analyticshelp/bin/answer.py?hl=en&answer=1205784&topic=1120718

We could do something similar, by doing:

Custom variables per page (in development), use a custom var per page to track the page load time for this page. Report the page load time for all URLs and Page Titles (as a new metric)
Custom Variables per visit. Track, as a Custom Var, the running average page load time of all pages in the visit.

Regarding "how to track the page load speed", we could either fire a different beacon (like GA does), or maybe approximate the page load speed by stopping the timer when the asynchronous beacon is fired? But, it might be not accurate enough to do this approximation.

robocoder commented 13 years ago

Reference http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/timing-overview.png

We wouldn't capture every perf metric. I think we can condense it two main ones (similar to "drive/road" and "curb" time metrics):

time for the network request and response (ie responseEnd - navigationStart)
time for the page content to load resources and render (ie loadEventStart - domLoading)

The second metric can be approximated using the naive method shown in the spec example 1.

mattab commented 13 years ago

It would be great to tackle this before Piwik 2.0... increasing priority so we can figure out the implementation plan

robocoder commented 13 years ago

I'll post my idea later tonight.

robocoder commented 13 years ago

I've attached a patch for proposed (minimum) changes to piwik.js.

I used a separate tracking request to record the timing because the trackPageView may be called before the dom has completed, and it may too early to pass the timings as custom variables.
This could be extended to sample a percentage of page views, eg

// theoretically, sample 10% of page views
if (Math.random() % 10 != 0) { return; }

Usage:

<head>
<script type="text/javascript">
var domLoading = new Date().getTime();
...
piwikTracker.enableTiming(domLoading);

or asynchronous method:

<head>
<script type="text/javascript">
var _paq = [];
_paq.push['enableTiming', new Date().getTime()];
</script>

mattab commented 13 years ago

it looks very interesting!

For user friendliness and ease of use and maintenance, I'm not sure about asking users to add the "var domLoading = ..." at the top of the page.

If we don't pass this value, timing will only work for recent browsers. I think that's good enough, rather than introduce an API input that we may not want to maintain in the future. What do you think?

For the php side, I'm working on Custom Var per page, so we could create a "fake pageview" with 2 custom vars that could maybe "update" the previous page view row rather than create a new one... That would make reporting easier I think

robocoder commented 13 years ago

Replying to matt:

For user friendliness and ease of use and maintenance, I'm not sure about asking users to add the "var domLoading = ..." at the top of the page.

If we don't pass this value, timing will only work for recent browsers. I think that's good enough, rather than introduce an API input that we may not want to maintain in the future. What do you think?

Sure, timing for newer browsers only would be easier.

For the php side, I'm working on Custom Var per page, so we could create a "fake pageview" with 2 custom vars that could maybe "update" the previous page view row rather than create a new one... That would make reporting easier I think

Are you talking about page-level custom vars?

mattab commented 13 years ago

Are you talking about page-level custom vars?

Yes, #2432

mattab commented 13 years ago

OK for limiting this to new browsers implementing timing APIs. These are the browsers we want to spend most of our time on since these are the ones we can easily time and profile.

Aim to provide new interesting reports:

Average page load time overall, and overall Bar chart of loading times
For each Page URL, (and/or page title?), Average load time breakdown (Bar graph), min/max
Show response time for each Country also? What other dimensions are relevant?

Implementation proposal:

Piwik.js
- select a sample of visits, eg. 10%?
- Send the timing request, in a different Piwik request, with a custom variable of type page, with a custom name eg. _pkspd for load time value
- the PHP client also support this feature via trackTiming (similar to GA)

For example if the jQuery library took 20 milliseconds to load from the Google Content Network, and you wanted data to be sent for 50% of visitors, you would call:
_gaq.push([_trackTiming, jQuery, Load Library, 20, Google CDN, 50]);

PHP Tracker code modifications
- Modify Action logging mechanism to UPDATE the previous page view row, rather than insert a new page view for this "fake action". We do not want to change bounce rates. The request for an action with this custom variable set is the indicator
PHP Archiving Code & API Output
- Modify the Actions report, to also count the total time loading page for all visits on the page, using the custom variable value for these rows that had it. Add the metrics (or count + sum)
UI: Add a new report in UI Actions, to show the Page Speed report page
- Hide load time column from other the Actions reports

I think this would be a fantastic new feature to have.

mattab commented 13 years ago

This feature is becoming increasingly important for Web Analytics and Piwik should have this very interesting report :)

mattab commented 12 years ago

New metrics in the site speed reports in GA: it looks very interesting! http://analytics.blogspot.com/2011/12/greater-insights-from-site-speed-report.html

Tracking Javascript Can we track the following metrics as custom variables scope page?


    Avg. Redirection Time - the time spent in redirection before fetching this page. If there are no redirects, the value for this metric is expected to be 0.
    Avg. Domain Lookup Time - the average amount of time spent in DNS lookup for this page.
    Avg. Server Connection Time - the time needed for the user to connect to your server.
    Avg. Server Response Time - the time for your server to respond to a user request, including the network time from the users location to your server.
    Avg. Page Download Time - the time to download your page.

Note: we don't have to track them all, maybe we should only track the general aggregated value for all visits rather than per-page value since it'd be too much data overhead to store in the Actions reports.

robocoder commented 12 years ago

All part of the Navigation Timing spec (linked above in ticket description). Not in my patch, but could be added.

The GA implementation has site and per page averages.

anonymous-matomo-user commented 12 years ago

Hello, have there been any more developments on this? At the company I work for I have been asked to develop a Piwik plugin which will track and display page loading metrics. The plugin will most likely become open-source once it is done. So, in order not to duplicate effort, apart from the attached patch is there any more info or code on the ticket subject?

robocoder commented 12 years ago

Banezaklan: feel free to develop and contribute this feature

mattab commented 12 years ago

This feature is just too important and too good not to assign the highest priority ;)

anonymous-matomo-user commented 12 years ago

Anybody working on this? I am new to piwik but would be happy to contribute if someone could point me in the right direction. We need this functionality ourselves. Is the best place to start the example plugin?

anonymous-matomo-user commented 12 years ago

My bad! Educated myself a little and figured out where it stands.

Replying to nnnnathann:

Anybody working on this? I am new to piwik but would be happy to contribute if someone could point me in the right direction. We need this functionality ourselves. Is the best place to start the example plugin?

anonymous-matomo-user commented 12 years ago

Hi,

I have applied the patch uploaded by vipsoft and it seems to work, at least the part where 'nettime' and 'domtime' values are sent to piwik server. My question is what now? How can I view this data that in the Dashboard UI ? Is a new plug-in needed for that?

Thanks, Horatiu

anonymous-matomo-user commented 12 years ago

Hi Horatiu,

I am about 1000% naive when it comes to Piwik, and the first thing I did was try to hack a page speed number into the Actions section using the implementation proposal by matt as a guide.

I didn't want to submit a patch to SVN as this solution is about as hacky as it gets, but it might give you some insight into where things happen, especially if you are new like me:

https://github.com/nnnnathann/piwik/compare/master...page_load

Hope that helps!

Would love to spend some time to get this working the right way, Piwik is pretty awesome

anonymous-matomo-user commented 12 years ago

Hi nnnnathann,

Thanks for your info, I have applied your patch and it works! However, as you said, it's kind of hard to interpret the data since the values of the Custom Variable don't have the URL of the page analyzed. I will try to find some solution to this.

Thanks again, Horatiu

anonymous-matomo-user commented 12 years ago

Hello all, Some time ago we started to work on a plugin to enable piwik to record the page loading metrics. I am uploading what we did so far. Note that, this is still work in progress and I'm attaching the plugin as it is now. It is not ready for production yet! What remains to be done is proper archiving of the data from the custom table which plugin creates and displaying of analyzed data preferably as part of standard Piwik reports. At the moment, there is only one simple analysis of the data and is displayed as a part of the plugin. It is pulling directly from the custom table so it will get slow soon on very busy websites. We're kind of stuck with the archiving part as we don't understand completely what and how to archive in order get the best reports later. All ideas and help is welcome!

Notes: Example JS script to be added to monitored website is in the archive.

anonymous-matomo-user commented 11 years ago

Any progress on this?

mattab commented 11 years ago

No progress yet, we are waiting for a company to sponsor the development on this feature! please contact us for more info

halfdan commented 11 years ago

Anthon's patch to piwik.js is great. I think however that the API should rather be piwikTracker.enableTiming(0.1) to track 10% of page views. We should also use the HTML5 Web Performance API to get the data. Supporting newer browser only is ok for this feature. http://www.html5rocks.com/en/tutorials/webperformance/basics

timo-bes commented 11 years ago

In 65458d099ae4a0ddb1923b7f6e110c5c0a9d3207: refs #1700 basic performance analytics: handle server generation time for each page and page title

CORE

formatting sub-second times
getColumn() method on data table array (in order to behave the same as the regular data table class)
data tables can store in their meta data, which columns are empty (this is used in order to dynamically hide the new "generation time" column)
ViewDataTable and Api.getProcessedReport act according to the empty column meta data

SCHEMA

new column custom_float_1 in log_link_visit_action
new version to apply the change

TRACKER

Piwik_Tracker::setGenerationTime
tracking parameter "generation_time_me"
value is stored in new custom_float_1 column
the log importer can handle a group "generation_time_micro" which can be used in a custom log format. _micro is used because apache can log the time in microseconds but piwik processes milliseconds.
note: extension of JS tracker still missing

ACTIONS PLUGIN

for pages and page titles, add new columns sum_time_generation and nb_hits_with_time_generation to the blob archives
if they are set, compute avg_time_generation on the API level. if not, remove the columns and mark them as empty in the data table meta data.
show new column "avg. generation time" in the pages and page titles reports

plus TESTS for everything

timo-bes commented 11 years ago

In bdf12d27b0eada820acd5210c94b21ab26466f95: refs #1700 fixing an error introduced in the previous commit (thanks peter for the report). added a little debugging output to datatable_manager.

timo-bes commented 11 years ago

In f56c9d6548778aea63e872ec125f4867e62def1c: refs #1700 performance tracking in piwik.js

by default, send performance.timing.responseEnd - performance.timing.requestStart as generation time with the tracking request
if the performance.timing API is not supported by the browser, don't send the metric
new method disablePerformanceTracking()
new method setGenerationTimeMs(generationTime) to set the generation after measuring it on the server side

timo-bes commented 11 years ago

A minimal version of this feature that supports only the generation time is complete at this point.

Here's a proposal for part of the documentation:

By default, Piwik uses the performance timing API that is available in most modern browsers. If it is available, piwik.js will measure the time between the start of the request and the end of receiving the response and track it as generation time. This number is the best approximation for the server generation time that is available on the client side. It is only approximate because it includes the transfer time.
Since the API is not available in all browsers, the performance data is not set on all pageviews. The value shown in Piwik is the average of all pageviews where a value for the time is set.
If you want to analyze the real server generation time, you have to measure it on the server and use piwikTracker.setGenerationTimeMs(generationTimeInMilliSeconds) to tell Piwik the real value. The call to setGenerationTimeMs() has to take place before calling trackPageView().
You can also disable the entire feature by using piwikTracker.disablePerformanceTracking(). This method also has to be called before calling trackPageView().
If you want to import the generation time from the server logs, specify a custom log format with the group "generation_time_micro" for the generation time in microseconds.

For more details about browser support, see http://caniuse.com/nav-timing

timo-bes commented 11 years ago

In 611922c647f8dfefffb472bd077d5c77ab99509b: refs #1700 better time formatting for milliseconds

mattab commented 11 years ago

In b10464e230fe19fc4dc3b1746fb2448b25b43308: * Small tweaks to translation refs #1700

rename custom_float_1 DB field to custom_float (further time tracking can be tracked as events)
fix regression missing auth in Live
remove extra comma in javascript which breaks IE parsing

mattab commented 11 years ago

In 276e8aee268062c7f905fcda1ea4c4f608a943c7: Typo refs #1700

mattab commented 11 years ago

Great work Timo on this one (code + tests look nice)!

It's a simple implementation, approximating the Page Speed to the Server generation+Response download, but it's a agreat approximate of the server speed as "experienced by visitors".

Feedback/Review:

Can you add general "Avg. generation time" in Visitors>Overview as sparkline + archived metric? Overall per-website site speed will be of huge value!!
Can you update the doc at: https://github.com/piwik/piwik/blob/master/misc/log-analytics/README.md Maybe add a new section and showcase an example of a custom log + Custom command line used to import the time.
import_logs script should support micro (your use case) but also milli (the std use case will be expected to work). Please also add a new test case for milli... ;)
Thx for for the doc for JS. I will update the JS doc shortly before release.

mattab commented 11 years ago

In db0dab9235a5ac0dd1216708a58f17483e3b4213: Refs #1700 Fix notice when there is no data Delete sum_time_generation column from output + Fix tests. This data point can be obtained by AVG_TIME * HITS_WITH_TIME so not so useful in output

mattab commented 11 years ago

Couple suggestions to make reports better:

in the Reports, I think it's better to show 2 decimals so make two changes:
- show 3.456s as 3.46s
- show 876ms as 0.87s This will allow easy visual comparison and highlighting of "bad" columns. 2 decimals avoids clutter.
in the Datatable for Page Urls/Page Titles, when hovering on the avg. response time value, show a tooltip "Average based on 198 hits" showing the nb_hits_with_response_time. This helps explaning sometimes some very high values, based on 1 or 2 hits.

timo-bes commented 11 years ago

In fb5f11a5ca82b15eb419803e6e00ce6dfba515d0: refs #1700 performance analytics

adding avg_time_generation to Actions.get + integration tests
adding sparkline for average generation time to Visitors > Overview
changing number formatting to 0.XXs instead of XXXms + test cases
tooltip for reports with avg. generation time: "average based on X hit(s)"
log import: support generation_time_milli (not only generation_time_micro)
example for importing generation time from logs in read me

timo-bes commented 11 years ago

All review comments are implemented now.

OK to close the ticket?

timo-bes commented 11 years ago

In 730bbbe74e98bc0d6cf675f93a28bd6ebc88b22f: refs #1700 fixing test failure

mattab commented 11 years ago

In 1cfa989fd1bb9324a8a329652fe7f069fe91396a: Beautify the time in Metadata reports + tests Small update to log analytics readme Fixes #1700

Awesome work Timo!

timo-bes commented 11 years ago

In 30f06e28d6ff1a1d366f598f657fb8484a9ae062: refs #1700: new metrics min_time_generation, max_time_generation

DataTable_Row::sumRow has a new parameter $aggregationOperations which allows treating columns as min or max (not only sum)
The aggregation operations can be set on any data table or passed to ArchiveProcessing_Period::archiveDataTable()
The Actions plugin uses the mechanism to aggregate the new metrics as min and max
The metrics are in the API output but not in processed reports
The min/max values are shown in the tooltip that appears when hovering the average generation times
Integration test updates

mattab commented 11 years ago

In f61c0196e33d8ecc5c4e967ce1a8199a11477de2: Very nice commit!

Refs #1700

Refactoring + adding missing parameters to other sumRow() calls
Fixing tests on PHP 5.4 / 5.5

timo-bes commented 11 years ago

Thanks, Matt. I think now we're done here...

mattab commented 11 years ago

Looking good.

But on the demo running latest code, the tooltip on hover, are not displayed, not sure why... I re-generated the archive for today but didn't help. The data is there in the XML: http://demo.piwik.org/index.php?module=API&method=Actions.getPageUrls&format=XML&idSite=7&period=day&date=2013-04-04&flat=0&filter_limit=100&token_auth=anonymous

timo-bes commented 11 years ago

Did you delete tmp/assets/*?

mattab commented 11 years ago

I just did to be sure, but this is not necessary as it's done during the upgrade process. still not working. I checked and the tooltip PHP code is the latest one.

timo-bes commented 11 years ago

The JS code is there too. But the span.cell-tooltip is missing. It sould be rendered in datatable_cell.tpl. Can you check that the template starts with {assign var="tooltipIndex" value=$column|cat:"_tooltip"} and that the template cache has been flushed?

mattab commented 11 years ago

flushing the templates worked... Strange I think it's the first time I had to manually delete compiled templates on upgrade on the demo... maybe we introduced a regression somewhere else.

It's working anyway, very nice!

matomo-org / matomo

Propose Page speed reports, Load time analytics #1700