Transitions: reports about previous and following actions for each page URL

[timo-bes]

Awesome feature: Analytics Previous / next pages

For each page URL, this plugin reports the top X

• previous pages
• following pages
• outlinks
• downloads
• campaigns
• search engine keywords
• external websites

The information is given in a popover which can be launched via a table action on the pages report.

[mattab]

Attachment: Example Transition Aug 22nd Transitions forum page.png

[mattab]

Attachment: Icon on hover could have colors and show a flow ? or something else, like a button and a text label ? Transitions open.png

[mattab]

Attachment: merged JS file showing errors firebug error.png

[mattab]

Attachment: too much data, direct entry is partly drawn.png

[mattab]

Attachment: Example blog post page with all previous-next types set.png

[mattab]

Attachment: Missing incoming traffic, does not sum to 61155 pageviews.png

[mattab]

Attachment: No data for URL.png

[mattab]

Attachment: Web Analytics Reports - piwik.org - Over 1 year - took 20s to generate.png

[mattab]

Attachment: missing direct entries.png

[mattab]

Attachment: 03.png

[mattab]

Attachment: unicode not displayed.png

[mattab]

Attachment: unicode URL displayed correctly in report.png

[timo-bes]

The plugin fakes an instance of Piwik_ArchiveProcessing_Day in order to use archiving code for live querying (later pre-archiving might be added). The plugin uses Piwik_ArchiveProcessing_Day::query*ByDimension. These methods have been extended in #3330.

From the returned data, only nb_actions is used in Transitions. In particular, count(distinct idvisit) and count(distinct idvisitor) is processed in MySQL and not used in the report. Count distinct takes significantly more time and memory than other aggregation functions. Should we extend the query*ByDimension methods to limit the columns in the result? The signatures are already rather long after #3330. I'm not sure what's best...

[timo-bes]

(In [6811]) refs #3332: Preparations for Transitions

• advanced period factory
• fake archive processing for live querying

[timo-bes]

(In [6812]) refs #3332 Transitions

• Archiving code that doesn't hook into the actual archiving
• API code that fakes an archive processing instance and calls archiving
• PHPUnit test

[timo-bes]

(In [6813]) refs #3332: properties for previous commit

[timo-bes]

In case you're wondering: I can't commit properties from git svn. I have to commit with git svn, then update, propset & commit with svn. That's why I always set properties in a separate commit.

[timo-bes]

(In [6843]) refs #3332 row actions registry: preparations for Transitions UI

• row actions are no longer in the HTML templates
• they are generated in JS
• plugins can register their custom actions in a registry
• actions can be available on a subset of tables and rows

[timo-bes]

(In [6844]) refs #3332: Transitions UI (row action for pages report + popover)

[timo-bes]

(In [6845]) refs #3332 properties

[timo-bes]

So here's the current state of the Transitions UI. The plugin adds a new row action to the pages report which opens the Transitions popover.

It would be great if some of you could give it a try and report back to me. I'm especially interested in performance. How long does it take to load the popover on how many pageviews per day?

Please don't take too much time writing suggestions for the UI itself. We already have lots of changes planned. Once that's done and you still have suggestions, please let me know.

The icon that launches the popover is just a placeholder. I'd like to have an icon that looks a bit like the popover with curvy bars going in and out of a box. Unfortunately, everything I came up with didn't look good. Do we have someone who could design the icon?

[timo-bes]

(In [6846]) refs #3332 more work on Transitions

• highlight the internal traffic as a group (light blue background + headline)
• close popover on overlay click
• slight color changes
• refactoring to remove duplication in JS
• "Others" not clickable anymore
• center box contains more info and is a little higher

[timo-bes]

Even though we did URL normalization recently, we still have some problems with non-normalized URLs in this plugin. URLs with different domains or different strings after the hash have different idactions but end up in the same row in the pages report. This causes inconsistency because the numbers from the pages report and the ones derived on the basis of the idaction don't match. Since the Transitions report mixes both, that's bad.

One possible solution would be to add two options to the configuration:

• One to replace the tracked domain with the main domain as provided in the website configuration. I'm not sure whether we should replace all domains or just the ones configured as alternative domains.
• Another option would normalize a little more, removing the hash part and other stuff that causes two idactions to end up in the same row in the actions report.

When both these options are enabled, the problem should be gone. When enabling one of the options would break the links for a user, he can disable them. Honestly, I think most users wouldn't care very much about broken links. But with the options, it's up to them.

An alternative solution would be this: in order to avoid conflicting numbers from the pages report and the Transitions plugin's archiving, we don't reuse the pages report and have to get the main metrics from the logs. In particular, we're talking about #hits, #entries, #exits. it should be possible to get #hits by summing up the other numbers.

So what do you think? Which option's better? Are there other alternatives?

[timo-bes]

(In [6853]) refs #3332 Transitions

• other groups (websites, search engines, campaigns) can be opened
• refactored some redundant JS code
• translations

[timo-bes]

(In [6854]) refs #3332 updated expected test output for Transitions

[timo-bes]

(In [6855]) refs #3332

• reset open group when closing the popover or changing the url
• fixed some lint

[mattab]

From the returned data, only nb_actions is used in Transitions. In particular, count(distinct idvisit) and count(distinct idvisitor) is processed in MySQL and not used in the report. Count distinct takes significantly more time and memory than other aggregation functions. Should we extend the query*ByDimension methods to limit the columns in the result? The signatures are already rather long after #3330. I'm not sure what's best...

If it's one more parameter and not too tricky, I'd say it's better to put it in. It might earn a few milliseconds here and there, which would tremendously help for pre-archiving (where milliseconds quickly make minutes ;))

So please add the parameter, and unit test, and all good. It's complicated but it has to be.

Even though we did URL normalization recently, we still have some problems with non-normalized URLs in this plugin. URLs with different domains or different strings after the hash have different idactions but end up in the same row in the pages report.

Hash tags: good point, I put a feature request in: #3232 - comment there if you have to add info.

Different domains: I think we can leave with this limitation for now. Users can use setCustomUrl in javascript to correct input URL if necessary... Later we could also implement a filter mechanism to modify input URLs and define rules in the piwik UI, but seems too edge case for now - UI and usability is IMHO more useful than perfect data!

[timo-bes]

Hash tags: good point, I put a feature request in: #3232 - comment there if you have to add info.

Not recording the hash part in the database by default (as you suggested in #3232) would help lots.

Thanks for creating such a lovely icon by the way - lol! Seriously, there must be someone who is good at this kind of thing and can design an icon for us.

Also, why are my commits not showing up in the ticket anymore? I made four today referring this ticket and none are shown here.

[timo-bes]

Replying to matts E-Mail:

Popover Title: could have a one line title such as "Transitions for /blog/hello-world... - April 15th 2012" indicating 2 important elements: date, and the page we are looking at.. would add real value without using more pixels, always appreciated.

The popover already conatins a lot of text and the title would make it look really packed. Also the title is way too big compared to the other text in the popover. Imho it looks quite ugly. Besides, the page we are looking at is already indicated in the center box.

So I'd rather not set the title. OK?

[timo-bes]

(In []) refs #3332

• aligning outgoing traffic right, pageviews centered in the main box
• fixing last imprefect translation
• removing some duplication from JS

[sgiehl]

(In [6868]) refs #3332, refs #3227 fixed possible fatal if no row is available for page metrics; do not run transition tests by default; ensure that all plugins are loaded before integration tests

[mattab]

Timo, WOOW! Excellent feature, awesome potential and I really like the first implementation you made!

Here is my feedback regarding the User Interface.

It took 3-4 seconds max (often 1s) to load for the homepage with 1500 pageviews. It is definitely super responsive ons mall websites, very nice speed! Benefits of RankingQuery and URL normalization are huge.

Very well done.. :-) impressed by your work & the performance side of it.

Here is full review of using the feature review & making it more accessible:

• Popover Title: could have a one line title such as "Transitions for /blog/hello-world... - April 15th 2012" indicating 2 important elements: date, and the page we are looking at.. would add real value without using more pixels, always appreciated.
• Loading: on click to load a new page, would be nice to show a white background with opacity, and a loading message in the middle, as to avoid the 'flash' that occurs
  • Ideally the loading would say "Loading Transitions for $url_truncated_with_tooltip" as to make waiting user fully aware of what's happening during these long seconds ;-) Hopefully only for high traffic piwik. So basically add "Loading Transitions" above the loading spin?
• About the Central Panel
  • Important UX/data improvement: on hover on the central panel, the absolute numbers could be displayed with opacity, on left and right outside to the central div, in their respective flow? It would give a nice flashy/halo effect and would also display the numbers on the bar itself, avoiding user/brain to read & parse the text in the middle. I think this would also look quite awesome, especially if the border to the central DIV would slightly illuminate or something like that?
  • Could "Outgoing traffic" text section block be aligned on the right as to visually show it refers to data on the right?
  • Could Exit bar be orangy/grey so it doesn't give bad feeling, but green is too positive for exits?
  • The number of page views for zoomedin page should be written much bigger, font 16pt.
  • Below the big page view metric should be "This page is N% of all pageviews" in grey, as to give context of this page within the whole.
  • Could the H2 have a font-size: 15pt or so? It looks nice.
• A few times clicking on the labels, I got a "The label '/feed.php' could not be found in Actions.getPageUrls". In this case, could you instead leave the previous view displayed, and simply write a message inline below the clicked label, saying "No data found for $label_clicked", instead of popup?
• When clicking from one Flow group to another (from Websites to Search Engines) would it be possible to see the transition animation of the groups in 500ms in the UI, so to show users how the ui works? I guess it's non trivial. ;)
• When a "group" is hovered, it would be nice that all groups on the same side, would display the number of clicks in the bar
  • Note: whenever we draw numbers in bars, we should only draw for numbers > X% so that there is enough space for the text + some white space to look nice. Small occurence referers groups will not have the metric displayed.

Other review:

• It would be great to have URL with hash tags to open the Transitions view (idem for Row Evolution)
• Transitions & Row Evolution feature access:
  • I like the idea of icons, even Row Evolution and Transition icons could be colored on hover.
  • On hover on the Transitions icon, I have an idea: the current grey T letter, could on hover show the next/previous flow on left / right of image? I guess we'd have to design an icon for it to look ince, I crafted a preview (see screenshot) but it's not ideal is it ;)
  • Also we could extend the 2 icons as buttons, and write "[ See Evolution ] [ See Transitions! ]" or similar. On hover, the icons could also show a Label a make like a little button? These icons+labels in a button would represented for what they really are: "Actions" for a given row.

More reviews to come later

[mattab]

(In [6987]) Removing temporarily Transitions plugin from release Refs #3332

on the demo, when enabling the plugin, it was creating a JS error in the merged JS file

[mattab]

(In [6988]) Removing temporarily Transitions plugin from release Refs #3332

[timo-bes]

Some general comments about the reviews (also the ones received via e-mail)

• Thanks for all the suggestions, they were very useful!
• I'll start with what I won't do: we can't use larger fonts because urls can get very lengthy and it's good to have as much of them as possible. Also, I won't implement an animation when opening groups because it would be very hard to do and result in even more complex code (which is quite complex already).
• The conntection between the flows and the metrics in the center were hard to see. Many of your comments resulted from this. In the upcoming commit, I have implemented something to make it easier.
• We need some general popover code: loading messages, changing between popovers and using the history. Many of the comments related to that were almost identical to comments about RowEvolution.
• Outlinks and page URLs should be clickable, I agree. I think we should use a redirect proxy to hide the current URL from the HTTP referrer. As far as I can see, that doesn't exist so far. Right?

[timo-bes]

(In [7032]) refs #3332 adding Transitions again

[timo-bes]

(In [7033]) refs #3332 highlighting groups on hover in Transitions

[timo-bes]

After the last commit, when you hover a goup title, it highlights the metric in the center. Also, the group is highlighted when you hover the metric in the center box. This makes the mapping between the groups and the center box easier.

[timo-bes]

(In [7035]) refs #3332 nicer loading message for transitions, introducing a general popover helper

[mattab]

(In [7037]) will release 1.9 B to test transitions on demo and bring back Row evolution on demo Refs #3332

[mattab]

When Transitions is enabled, the Merged Javascript file returns the error:

SyntaxError: missing } after property list
index.php?module=Proxy&action=getJs&cb=0703c62885101c6797656014692f23aa
Line 1122

See attached screenshot showing error in firebug.

[mattab]

Wow very cool changes! They make a big difference indeed :)

I think we should use a redirect proxy to hide the current URL from the HTTP referrer. As far as I can see, that doesn't exist so far. Right?

Yes it's there, you can use: href="?module=Proxy&action=redirect&url='.urlencode($URL)

After the last commit, when you hover a goup title, it highlights the metric in the center. Also, the group is highlighted when you hover the metric in the center box. This makes the mapping between the groups and the center box easier.

Very nice indeed!

Some feedback after testing this feature for 1 hour:

• Tooltip for a given previous/next page, on hover on left/right percentages, says: "22 of 45 from search engine", could you slightly clarify and write "$URL \n 22 visits (out of 45 visits) from search engines"
  • I think that, on hover on the label, and the %, the same tooltip should be shown which would contain URL + detail. This way we don't have 2 different tooltips (one for URL, one for the %) in the same block.
• currently the orange is displayed/highlighted when the label is hovered. Could you also color in orange, when the mouse hovers on the blue/green curbs on left/right of central box? It will really improve usability as it's natural for me to hover on the curb (ie. closer to central box) rather than the actual group's label
• small detail: when the box is hovered orange, it would be nice to show the + icon in orange as well (rather than blue)
• Could you add a tooltip, on the main "1834 pageviews" in the middle. The Tooltip would say "In July 2012, This page had 1834 pageviews (5% of all pageviews)"
• when there are zero from internal page, from websites, or zero to internal page, etc. write the label in light grey to maximise focus on data points
• Transitions works for Actions>Pages report. Could you also enable it for Actions>Dowloands and Actions>Outlinks? ie. see the transition report for a specific downloaded file, would be useful.
  • For example It would be very interesting to see for "piwik.org/latest.zip" which pages and keywords were used as previous interaction to the download!
  • Also it would really interest me as I'd like to show Transitions for Internal search keywords :)
• the center box top blue link (ie. the name of the page looked at) should link to the actual page (with target=_blank, via proxy)

Transitions is such a great feature!! :)

[timo-bes]

(In [7043]) refs #3332 fixing JS error in merged file

[timo-bes]

(In [7045]) refs #3332 Transitions remembers the popover state in a second hash part of the URL => using direct links and the browser history is possible now!

[timo-bes]

(In [7046]) refs #3332 Transitions: main url, external referrers, outlinks and downloads open in new tab on click

[timo-bes]

• Tooltip for a given previous/next page, on hover on left/right percentages, says: "22 of 45 from search engine", could you slightly clarify and write "$URL \n 22 visits (out of 45 visits) from search engines"

I played a bit with that idea and decided to do only a minor change

• We can't write "visits" because the base is not visits but transitions. One visit can do multiple transitions to an internal URL for example.
• Writing the URL in the tooltip is a bit much and distracts from the actual info we want to show.
• I like "X (out of Y)" better than "X of Y", the change will be included in the next commit.

• currently the orange is displayed/highlighted when the label is hovered. Could you also color in orange, when the mouse hovers on the blue/green curbs on left/right of central box? It will really improve usability as it's natural for me to hover on the curb (ie. closer to central box) rather than the actual group's label

Unfortunately, that's a limitation of canvas: we can't attach events to the shapes. I know it would be nice but it's nearly impossible.

• For example It would be very interesting to see for "piwik.org/latest.zip" which pages and keywords were used as previous interaction to the download!

It would be nice to have downloads, campaigns and external websites in the center. It's on my list but I won't do it for the first version.

• Also it would really interest me as I'd like to show Transitions for Internal search keywords :)

It is planned to have internal search keywords on the left so you can see which searches lead to the page. Having internal searches in the center is interesting but that kind of info should be included in the internal search plugin.

[timo-bes]

• small detail: when the box is hovered orange, it would be nice to show the + icon in orange as well (rather than blue)

Actually, I think it's good that it's blue because the orange background highlights the blue icon. If we make it orange on orange background, it's easier to miss it and expanding the groups is one of the coolest parts about the feature so I'd like to make it visible.

[timo-bes]

(In [7050]) refs #3332 Transitions:

• tooltip on title showing share of all pageviews and date range
• metrics with value 0 are greyed out in the center box
• adjusted dimensions for wider popover
• re-clicking the current menu item reloads the page again
• transitions is available on entry and exit pages as well

[timo-bes]

Matt, in your screenshot the percentage labels on the curves sometimes had two lines (becuase you don't have Arial). Can you please check whether that's fixed now?

[timo-bes]

(In [7051]) refs #3332 Transitions:

• Showing referrals from campaigns in the center box
• Proper error message when URL was not found or truncated from the actions report (introducing general popover error method)
• Stylish icon for row-action
• Minor JS and CSS fixes