Transitions: reports about previous and following actions for each page URL

[timo-bes]

Awesome feature: Analytics Previous / next pages

For each page URL, this plugin reports the top X

• previous pages
• following pages
• outlinks
• downloads
• campaigns
• search engine keywords
• external websites

The information is given in a popover which can be launched via a table action on the pages report.

[mattab]

(In [7052]) Refs #3332 will test latest Transitions update on demo :)

[mattab]

(In [7054]) Refs #3332 Fixing error when merged JS file, broadcast.js should be loaded before other files eg. it is used in datatable_rowactions.js which was loaded first on the demo

[mattab]

very nice changes!

I put some screenshots to show examples of the latest of Transitions.

Feedback:

• The hash URL just works :)
• The tooltip on the focused page "This page had N views (M% of all visits)" Here the % displayed is not valid, it shows 0% or 0.2% when should be more. In the link above page had 3353 / 55538 = 6% but displays 0.1%
• There is a scrolling bar on the right of popover, Cf. screenshot, which can be removed (maybe popover slightly higher?)
• In the middle box, hovering on an incoming/outgoing traffic type will highlight the flow. Please make these labels also clickable, so they expand the flow, for added efficiency analysing one page transitions. we can then only keep the mouse in the middle block to analyse the page.

Having internal searches in the center is interesting but that kind of info should be included in the internal search plugin.

It isn't planned in SiteSearch to show the most "pages starting a search" or "Most visited pages after search". I think in fact it should use transitions for that, since it's the perfect UI for it.

It would be nice to have downloads, campaigns and external websites in the center.

It would be really great to have... Hopefully you might change your mind before V1 :)

• I still have to review code for security & best practise, will do once I finish sitesearch.
• The % now display correctly on one line.
• OK with your other feedback!

[mattab]

Further feedback:

• Nice that you added Campaigns in middle box, I didn't notice it was missing
• The Loading... and the Hash tag features are really useful & add a lot to usabilit!
• Now that I think of it, it's interesting that it works with the double hash tag in the URLs... I even tested with IE and it works. I'm not sure if it's standard practise, did you try to have a Hash tag as follows?

#module=Actions&action=indexPageUrls&idSite=7&period=month&date=2012-08-23&RowAction=Transitions:http://forum.piwik.org/

instead of

#module=Actions&action=indexPageUrls&idSite=7&period=month&date=2012-08-23#RowAction:Transitions:http://forum.piwik.org/

• [http://issues.piwik.org/attachments/3332/No%20data%20for%20URL.png no data for url] - I think it should have worked in this case, maybe a problem with URLs with query strings?
  • I also tried the URL with the second hash tag URL encoded but it didn't load neither, so not sure.
• When keywords are long, Direct entry is partially drawn
• On this screenshot you can see that the sum of incoming traffic+reload is not equal to the pageviews of the clicked page. 49192+5716+366+116161155 = -4720 pages difference. What do you think are possible explanations for such discrepency ?
• Nice example with plenty of data

==================

Timo's answer:

• Now that I think of it, it's interesting that it works with the double hash tag in the URLs... I even tested with IE and it works. I'm not sure if it's standard practise, did you try to have a Hash tag as follows?

We need to have two separate parts because the change of the second part should not cause a change to be triggered in the first part. This means that when the popover changes the report in the background should not be reloaded. Apart from perfomance that would be really bad if a user has opened several folders on the actions report and the actions report is back to start when he closes the popover.

[timo-bes]

(In [7057]) refs #3332 Transitions:

• if a url isn't found in the actions report, show the rest of the UI even though exits and bounces are not available (previously, there was an error message)
• the number of pageviews is derived from the live queries - makes data more consistent
• sanitize data taken from actions report using the live data
• more robust second hash handling

[timo-bes]

I'm sorry, I changed you last comment (#3332) instead of replying to it. I still have it in my notes so none of your suggestions will be lost :)

[timo-bes]

On this screenshot you can see that the sum of incoming traffic+reload is not equal to the pageviews of the clicked page. 49192+5716+366+116161155 = -4720 pages difference. What do you think are possible explanations for such discrepency ?

There can be multiple reasons:

• The page has been loaded sometimes with a hash part or different domains. In this case, the number from the actions report is too high because multiple idactions map to the same row in the data table. This problem should be gone after the update.
• Previous downloads and outlinks are not shown in transitions. 4720 is a bit much compared to the following downloads so I guess this number will drop after the next update. Let's keep an eye on this!

[timo-bes]

(In [7058]) refs #3332 Transitions:

• clicking the group name in the center opens the group
• fixing percentage of all pageviews
• when all groups on the left are available, draw them a bit more compact in order to fit into the popover
• showing previously removed error message again in a real error case
• removing forgotten debug output

[timo-bes]

(In [7059]) refs #3332 Transitions:

• hover effect and tooltip for row action icons
• more robust hash handling

[timo-bes]

Please update the demo so that we can verify that the problems you reported are fixed.

[timo-bes]

I had a crazy idea... If we really trust the queries i wrote, we can derive the number of exits from pageviews - following pages - outlinks - downloads. If we abandon the number of bounces, we could get rid of using the actions report altogether! That should speed everything up a lot - especially for pages with many distinct URLs. Thoughts?

[mattab]

Replying to EZdesign:

I had a crazy idea... If we really trust the queries i wrote, we can derive the number of exits from pageviews - following pages - outlinks - downloads. If we abandon the number of bounces, we could get rid of using the actions report altogether! That should speed everything up a lot - especially for pages with many distinct URLs. Thoughts?

Getting rid of bounces is not a problem, so if it makes code faster, sounds really good to me!

[mattab]

(In [7066]) Refs #3332 Will test new Transitions on demo (Nice tooltip!!) ;-)

[timo-bes]

(In [7072]) refs #3332 Transitions: the actions blob report is not used anymore - improved performance, consistency problems are gone, bounces are not available anymore

[timo-bes]

(In [7073]) refs #3332 Transitions performance tuning

• load total number of pageviews only once and from JS => archiving is never launched from Transitions API
• load markup only once => when clicking a different URL, only the new data is loaded
• load markup and data in parallel

[timo-bes]

After the last refactoring, the MySQL queries account for almost the entire processing time when switching between pages. Speeding them up will improve the UX a lot.

Here are the indexes that will do the job:

ALTER TABLE  `piwik_log_link_visit_action` ADD INDEX  `transitions_url` (  `idaction_url` ,  `idsite` ,  `server_time` );

ALTER TABLE  `piwik_log_link_visit_action` ADD INDEX  `transitions_url_ref` (  `idaction_url_ref` ,  `idsite` ,  `server_time` );

ALTER TABLE  `piwik_log_visit` ADD INDEX  `transitions` (  `visit_entry_idaction_url` ,  `idsite` ,  `visit_last_action_time` );

The order of the columns is very important. Make sure not to change it.

I'm not saying we should add them by default but maybe we can document that adding them will speep up transitions tremendously. On my Piwik, it brought processing time down from 1-2s to 100-200ms. For larger databases, the factor should be even bigger. Maybe we can try on the demo?

[timo-bes]

(In [7074]) refs #3332 Transitions:

• segmentation works now!
• fixing a js bug related to concurrent ajax requests
• refactoring of the api class
• translation update

[mattab]

Excellent updates, faster & simpler code...

My last request as you know would be to support Transitions for Page Titles... then the feature will be fully complete IMHO!

[mattab]

It seems the "Direct entry" metric is not processed correctly, see two attached example screens of missing direct entry.

[timo-bes]

(In [7091]) refs #3332 Transitions

• support for page titles
• language strings for error messages are loaded if not present
• some core extensions

[timo-bes]

(In [7093]) refs #3332 Transitions:

• center url / page title gets truncated if it's longer than four lines
• main tooltip shows the full URL (including domain, protocol and www, untruncated)
• truncation for boxes on the sides works again
• tolerate url=null passed from php (happens when there are encoding errors in the db)
• don't try to shorten page titles like urls

[mattab]

Kuddos Timo for all these great commits....!

• I LOVE seeing Transitions for Page Titles, this is really very useful & more readable for many websites! :) Very powerful!
• It seems "Page title not defined" page title does not show Transitions but appears in Previous/Next. Does it sometime work? to be tested further
• Transitions does not work for Page titles containing / which are grouped in categories and sub-cats. The icon should be hidden on these expandable groups?

I will update on demo now to test further!

[mattab]

(In [7103]) Refs #3332 Summing rows now keep the latest metadata to ensure the latest format of URLs is used when summing over large date range

[mattab]

After using Transitions further, I have the following suggestion

• Suggestion: Display the Transitions icon, on directories
  • Why? many times, the /sub/dir/ page will actually have transitions, since there is a page at example.org/sub/dir/. This is the case for all wordpress blogs by default and many other CMS, which serve pages at the root of each directory/category/tag.
  • How? When there is no data for this page, we could automatically run the query with the segment pageUrl=@http://example.org/sub/dir/ so Transitions would report what you'd expect: all transitions for all pages in the clicked group.
  • Blocker? I tested it seems transitions does not return data when used with the segment &segment=pageUrl=@http://piwik.org/docs/ -- to be tested further

[mattab]

(In [7113]) Refs #3332

• Transition works for Page titles with hierarchy (eg "$Category / $name")
• URL metadata:
  • is only available on non Summary rows.
  • It is now set to the page with highest number of visits, when page names do overwrite.
• However now, by default we do track keep hash tags and we do not aggregate them in one page URL.
  • Refs #3232 Will be nice to have, to allow to "not track fragment" by default. See stub processUrlFragment(). More to do: Admin UI, Tests w/ and wo/ hash tags, w/ Capital letter in hostname
  • URL being tracked, and URL read from the logs (for backward compatibility) are now cleaned: hostname is lowered, and URL Fragment is kept/removed.

[mattab]

(In [7114]) Refs #3332 Will test latest changes on demo, to check if no more bugs

[mattab]

• I notice a bug: the Transitions icon is displayed on Page titles with hierarchy, could we hide it while we wait for later implementation of: #comment:63

[timo-bes]

(In [7124]) refs #3332

• direct entries are shown last in the center (same order as on the left)
• entries and exits in light blue
• title of open group in orange

[timo-bes]

Replying to matt:

• Suggestion: Display the Transitions icon on directories

If you want to see tranitions for example.org/foo/bar/, just open it on foo > bar > index. Doing it for foo/bar/* (i.e. the sum of everything in the directory) doesn't work by design because Transitions always shows stats for one idaction only. Doing something else would mean major code changes and performance issues. I'm generally not in favor for doing that but one week before the release that's definitely not possible...

[timo-bes]

(In [7125]) refs #3332

• Transitions is not available on page title groups anymore
• JS is slightly more efficient with row actions

[timo-bes]

(In [7126]) refs #3332

• use action_url_category_delimiter setting for hierarchical page titles.
• Piwik_Common::unsanitizeInputValue should not be used on page titles. otherwise titles with an ampersand don't work anymore.

[timo-bes]

It seems "Page title not defined" page title does not show Transitions but appears in Previous/Next. Does it sometime work? to be tested further

It works for me. Maybe I fixed it in [7126]? Please try again.

[timo-bes]

(In [7128]) refs #3332: edges on the sides are a bit straighter

[timo-bes]

(In [7132]) refs #3332

• many cross browser fixes (mainly for internet explorer 7-9)
• removing now unneeded methods Piwik_Popover.prepareContent() and Piwik_Popover.showPreparedContent()
• using different canvases for left and right to make clearing only one half possible with excanvas
• drawing loops on a separate canvas to make proper redrawing possible
• canvases are cached in a similar fashion to the graph canvases in order to prevent memory leaks
• fixing issue where lines in tooltip don't break in firefox and ie
• applied auto code formatting (many whitespace changes)

[timo-bes]

(In [7133]) refs #3332: more cross browser compatibility and other fixes

• the popover location is not put after a second hash anymore because that caused trouble in safari and internet explorer. instead, a popover parameter is introduced.
• special url encoding for the popover location to make sure it's en- and decoded the same number of times.
• minimum width for the label column in tables with row action.
• tooltip behaves slightly nicer in ie7.
• fixing a regression that broke transitions for subtables in the page titles report.
• now transitions works in ie789, safari, firefox and chrome. in ie78, the css3 properties of the center box (border radius and shadow) don't work and excanvas can still be pretty slow sometimes. apart from that, everything's fine.
• tested for memory leaks in all browsers. none found.

Apart from translation and documentation, that's it!

PLEASE TEST AS MUCH AS YOU CAN AFTER THIS COMMIT!

[timo-bes]

(In [7137]) refs #3332

• fixing page titles containing the > character. they used to be interpreted as recursive labels, replacing the > with /. now, js puts together the label correctly. it needs to know the action_url_category_delimiter, so it's added to the global js variables.
• fixing error for "Page URL undefined". since there is no consistent data that can be shown for this row, transitions is disabled on it.

[mattab]

(In [7143]) Refs #3332 Fixing JS error on admin pages

FYI since Piwik_Controller is only used for reports controllers (where as Piwik_Controller_Admin is for admin controllers), the separator wasn't set on admin screens

[mattab]

(In [7148]) Refs #3332 This will set the metadata on the URL with the most page views, for daily reports Note to self: we could add a test case for this...

[mattab]

(In [7160]) Refs #3332 Hopefully, fixing all problems with URL being incorrectly aggregated! Thanks Timo for the hint and bug in previous algorithm

[mattab]

Test message notification?

[mattab]

It seems that all "data bugs" should now be fixed, please notify me if you find any further issue!

Timo, I noticed there is a small problem: unicode characters don't get displayed, instead the URL encoded version is displayed, in Transitions.

For example in this URL

See attached screenshot

[timo-bes]

Replying to matt:

Timo, I noticed there is a small problem: unicode characters don't get displayed, instead the URL encoded version is displayed, in Transitions.

The UTF8 chars are shown in the label of the report but the link is urlencoded. That's why it's urlencoded in transitions. I'm pretty sure the urlencoded version is also what's in the database because Transitions looks for this version and works.

So what's the best behavior? Should we urldecode once before displaying urls in Transitions? What does the actions report do with the label?

[timo-bes]

(In [7165]) refs #3332 Transitions help button linking to doc page

[timo-bes]

(In [7166]) refs #3332: url decode in transitions in order to display utf8 urls

[timo-bes]

Please test the latest commit on the demo. The UTF8 chars should work now.

I tested the code with a url that contains both utf8 chars and an html tag. UTF8 encoding and XSS protection ist working in the center box, side boxes and the tooltip. Nevertheless, please test and review closely.

[timo-bes]

OK guys, this is it.

If you can't find more problems, we can close the ticket! Yay!!!

[timo-bes]

(In [7189]) fixes #3332

• refactoring breakpoint methods from transitions to piwikHelper
• loading message is now displayed properly for utf8 urls

[timo-bes]

(In [7201]) refs #2992, refs #3332: integrating previous and following internal searches into transitions. the code is starting to get a little complicated ;-)

[timo-bes]

(In [7214]) refs #2992, refs #3332 fixing two ui bugs in transitions / site search integration

[timo-bes]

(In [7216]) refs #3332, refs #2992 making transitions center box higher because new lines were added for site search. fixing a js bug when truncating urls.