Static PNG Graph: plot several metrics & multiple row evolutions

[mattab]

It would be very nice of the Static Image Graph API could plot several metrics at once, similarly to the JS graphs.

For example we could plot Visits and Pageviews on the same graphs with: index.php?module=API&method=ImageGraph.get&idSite=3&apiModule=VisitsSummary&apiAction=get&token_auth=anonymous&graphType=evolution&period=day&date=previous30&width=500&height=250&columns=nb_actions,nb_visits

• For consistency we could rename parameter column to columns
• Accept comma separated list of columns (consistent with other APIs)

The Static Image Graph API should also support a new parameter named &labels to plot multiple row evolutions.

• Accept comma separated list of row labels
• when $labels == false, plot the top N labels. N should be by default equal to 5 and can be overridden with the value of filter_limit
• new metadata field : imageGraphEvolutionUrl

[julienmoumne]

Attachment: VisitFrequency with multiple metrics.png

[julienmoumne]

Attachment: Referers.getRefererType with multiple metrics.png

[julienmoumne]

Attachment: UserSettings.getPlugin with multiple metrics (mobile size).png

[julienmoumne]

Attachment: &labels=Search Engines,Direct Entry,Websites&columns=nb_actions Referers.getRefererType row evolution.png

[julienmoumne]

Attachment: http://dev.piwik.org/trac/ticket/3158#comment:3 getRowEvolution now supports $idGoal Goals getVisitsUntilConversion ecommerceOrder last5 months LABEL=1 VISIT.png

[mattab]

This feature would be very useful for:

• Piwik Mobile to plot several metrics at once on the graphs
• But the ability to draw multiple lines of values on a same graph will also be reused for another feature, plotting a same metric for different rows:
  • Include in Scheduled reports graphs plotting visits from Search VS Websites VS Campaigns VS Direct entry #3014

[julienmoumne]

This ticket seems to be more than plotting multiple metrics.

3014 requires the ImageGraph plugin to also plot row(s) evolution.

The first question we need to answer is whether we :

• add a new &labels parameter to ImageGraph.get, or
• add a new API method ImageGraph.getRowEvolution

If the first option is selected, we need to define when should the Row Evolution API be solicited.

Here are the main two cases I could think of :

• when at least one row label is provided via the &labels parameter
• when &period &date correspond to multiple periods and the requested report has a dimension (such as Referrers, UserSettings, UserCountry and so on). Currently, ImageGraph.get generates an error see The graph cannot be drawn for this combination of 'date' and 'period' parameters.
• should both cases above be required to solicitate the Row Evolution API or should one be sufficient ? For example, if no labels are supplied, maybe we could automatically plot ALL labels?

This evolution will also require to update the Metadata imageGraphUrl field.

The current behavior of the imageGraphUrl is as follow :

• if &period &date correspond to multiple periods and the requested report has a dimension, &period &date will be rewritten in the imageGraphUrl field. As an example, in API.getMetadata Referers.get period=day date=last30 the '&period' parameter is rewritten to 'range'.
• however, if the requested report has no dimension, &period &date are left intact as in API.getMetadata VisitsSummary.get period=day date=last30.

After updating the ImageGraph API to support row(s) evolution, we need to determine the value of the imageGraphUrl field for the first case (API.getMetadata Referers.get period=day date=last30).

Instead of rewriting &period &date, the imageGraphUrl would contain an API call which returns row evolution. However, we need to decide which label(s) should be drawn. Ideally, we could plot all labels. This gets complicated when the report does not have a constant number of labels (constantRowsCount!=1) or when the number of rows is too big.

[mattab]

Replying to JulienM:

This ticket seems to be more than plotting multiple metrics.

3014 requires the ImageGraph plugin to also plot row(s) evolution.

Thank you for your great analysis :)

The first question we need to answer is whether we :

• add a new &labels parameter to ImageGraph.get, or

I think this solution is nice indeed!

Here are the main two cases I could think of :

• when at least one row label is provided via the &labels parameter
• when &period &date correspond to multiple periods and the requested report has a dimension (such as Referrers, UserSettings, UserCountry and so on). Currently, ImageGraph.get generates an error see The graph cannot be drawn for this combination of 'date' and 'period' parameters.
• should both cases above be required to solicitate the Row Evolution API or should one be sufficient ? For example, if no labels are supplied, maybe we could automatically plot ALL labels?

Agreed that when no labels are supplied, we could plot the evolution of the top 5 labels for example ?

After updating the ImageGraph API to support row(s) evolution, we need to determine the value of the imageGraphUrl field for the first case (API.getMetadata Referers.get period=day date=last30).

Instead of rewriting &period &date, the imageGraphUrl would contain an API call which returns row evolution. However, we need to decide which label(s) should be drawn. Ideally, we could plot all labels. This gets complicated when the report does not have a constant number of labels (constantRowsCount!=1) or when the number of rows is too big.

Plotting all labels it not so possible, because above 5 lines the graph becomes hard to read.

• I propose that we plot the top 5 labels
• the imageGraphUrl is rewritten with a new parameter &filter_limit=5 -- this will be used to limit the data from API but also the user could easily increase the value if he wishes to plot top 10 labels on a wider graph.
• Even we could have fun with Thomas and for example plot top 10 lines when the graph is full screen on a tablet ;-)

--> &labels and &filter_limit support sounds like perfect API evolution for this feature!

There is also an open question about how to plot Ecommerce metrics, Goal metrics, on the graph. Maybe in a different ticket we could propose to have &enhanced=1 parameter that would return the goal/ecommerce metrics as well.

[julienmoumne]

Pending comment:3:ticket:3158.

If it can't be fixed, we can throw an exception when trying to plot row evolutions for Goal reports when Piwik_ImageGraph_API.get $idGoal is not empty.

We would also need to alter the imageGraphUrl construction logic to switch back to non-evolution graph.

[julienmoumne]

The logic to retrieve the top 5 row evolutions should maybe be part of #3158, see comment:4:ticket:3158.

[mattab]

Very nice graphs!!

If it can't be fixed, we can throw an exception when trying to plot row evolutions for Goal reports when Piwik_ImageGraph_API.get $idGoal is not empty. You meant "when idGoal is empty" I suppose?

We would also need to alter the imageGraphUrl construction logic to switch back to non-evolution graph. +1

• As noted before, the row evolution should not require the &labels= parameter (for example to plot evolution of referrer types, &labels is not required)
• This new ability to plot multiple values at once will be great when we implement "Compare the monthly report to the last month" #291 and we could show, on the graph, the values for the current and previous month...

[julienmoumne]

(In [6918]) refs #3013

• plot multiple metrics using $columns (renamed from $column)
• plot row evolutions using optional $labels, will plot top 5 labels when omitted
• imageGraphEvolutionUrl added to report metadata (only for reports without constant row count and Referers.getRefererType)
• $showMetricTitle renamed to $showLegend

refs #3014

• new report parameter 'Show Historical Graphs for the top 5 values'
• better looking evolution graph on landcape pages

refs #3158

• $label is now optional, will return top N=filter_limit row evolutions sorted with filter_sort_column when omitted
• support for $idGoal
• integration tests updated

constantRowsCount set on

• Goals_getDaysToConversion
• Goals_getVisitsUntilConversion
• Referers_getRefererType

merges

• r6874

IDEA

• in Piwik UI, add a link 'display top 10 row evolutions' using API.getRowEvolution &label=false&filter_sort_column=xyz&filter_limit=10

TODO

• fix scheduled report integration tests
• review @reviews

[mattab]

Great patch and improvements! :)

• //@review without adding this, I can not set [...]

  That's expected, OK.

• // The other way to solve this issue is to add the following exceptions in API.getRowEvolution: module=Goals AND (method=getVisitsUntilConversion OR getDaysToConversion)

--> This other way to solve would be better than the committed fix: sometimes it is important to delete the rows that have visits=0 but some conversions, because sometimes the "Keywords" or "websites" used in previous visits can be credited a "conversion" if they are the last referrer.

So not deleting the conversion==0 row in this datatable would result in the UI showing rows with 0 visits which was a bug we fixed with this code to delete rows :)

• Regarding code:

    1135                            // retrieve top labels 
    1136                            $dataTableArray = $this->loadRowEvolutionDataFromAPI( 
    1137                                    $idSite, 
    1138                                    $period, 
    1139                                    $date, 
    1140                                    $apiModule, 
    1141                                    $apiAction, 
    1142                                    null, 
    1143                                    $segment, 
    1144                                    $idGoal 
    1145                            ); 
    1146     
    1147                            // get the last, ie. most recent, datatable 
    1148                            $dataTables = $dataTableArray->getArray(); 
    1149                            $mostRecentDataTable = end($dataTables); 
    1150     
    1151                            $labels = $mostRecentDataTable->getColumn('label'); 

Here it would be more efficient to directly call the API using Piwik_API_Request() and http_build_query() to build the request string. The advantage will be better performance since you can only request the last period's worth of data.

You should be able to use the functions: Piwik_Archive::isMultiplePeriod and Piwik_Period_Range($dateString) to get the last date of the range.

let me know if there is any problem doing this change? it will speed up graphs.

• I don't think this change is necessary? why adding the clause here?

1244    1269                    // add "processed metrics" like actions per visit or bounce rate 
1245                        if ($apiModule != 'Actions') 
    1270                    if ($apiModule != 'Actions' && $label) 

• would you be able to prepare a documentation updates, at least new examples to show the cool new features, for the page: http://piwik.org/docs/analytics-api/metadata/#toc-static-image-graphs -- this will help users discover the very cool features and capabilities of the graph API! :-)

[julienmoumne]

(In [6925]) refs #3158

• adding more report exceptions from being filtered by AddColumnProcessedMetrics filter
• I forgot to manage an edge case when $label=false and there are no data, what should the API return ? see @review
• only retrieve the last period when looking for top n labels

refs #3013

• better error message when specifying invalid metrics in $columns
• same image height for all graph types in all page orientations for pdf reports

[mattab]

paste due to misclick on my new laptop ;)

[julienmoumne]

(In [6947]) refs #1721 refs #3013

• show label icons in evolution graph
• smaller legend font size
• draw colored shadows in legend
• vertical legend for evolution graphs
• enhanced label skipping for daily evolution graph

[julienmoumne]

(In [6951]) refs #1721 refs #3013

• better legend font sizes
• legend shadow only when plotting multiple series
• more shadow padding

[julienmoumne]

(In [6969]) refs #3013

• reduce shadow width to clear off series' icons
• fix shadow on big font sizes
• increased line icon width in evolution graphs

[julienmoumne]

(In [6970]) refs #3013

• forgot to update Piwik_ImageGraph_StaticGraph_Exception

[julienmoumne]

(In [6982]) refs #3013

• horizontal shadow padding
• top & left margin added for vertical legends

[julienmoumne]

(In [7009]) refs #3013

• fix pie charts

[mattab]

This is a test comment but a real comment too, awesome work & can't wait to see the next improvements ;-)

[julienmoumne]

(In [7026]) refs #3013

• remove value markers on evolution graphs

[julienmoumne]

(In [7077]) refs #3013

• truncate legend labels in evolution graphs

[julienmoumne]

(In [7078]) refs #3013

• rounding top scale value to the next multiple of 10

[julienmoumne]

(In [7079]) refs #3013

• more labels displayed on vertical bars and evolution graphs

[julienmoumne]

(In [7086])

refs #3158 refs #3013

• remove metric names from metadata with new API parameter $legendAppendMetric == false (defaults to true)

refs #3158

• return label values instead of metadata url with new API parameter $labelUseAbsoluteUrl == false (defaults to true)

  refs #3013

• set API.getRowEvolution $labelUseAbsoluteUrl == false

[julienmoumne]

(In [7088]) refs #3013

• truncate vertical legend labels at 70% of total image width
• reduce default legend font size

[julienmoumne]

(In [7102]) refs #3013

• fix label skipping algorithm when plotting data using period=day&date=previous|last7

[mattab]

(In [Refs #3013 I'm in a rush, but I had to add this code as the $periodsData$i) was NULL in my case, not sure why it hasn't been needed until now, so maybe there is an underlying problem in Site Search, or maybe this change is acceptable...?

[mattab]

(In [7303]) Fixes #3474

• sometimes width or height are not defined, in particular when the API fails before the $width and $height are defined in line 216-217 of ImageGraph/API.php so we always check the error graph has a width/height set.

Ping Refs #3013

[mattab]

This ticket is getting long, What is the status of #3013?