Page Overlay - Analytics Overlay on your site! (Aka in-site analytics)

[mattab]

In order to show analytics data as an overlay, we have to set up a cross-domain communication protocol. The aim of this ticket is to complete a simple version of the feature that provides extensible mechanisms for the communication. Based on this, we can add more things to it.

Page Overlay displays a block around each link showing the % of clicks on this link. the user can still browse the website. On the left a sidebar displays the key statistics for the currently viewed page. It also easily allows the user to open the Page Transitions report and the Row evolution historical report for this particular page.

See #3530 List of Page Overlay Improvements for more info and the current state of the art!

[mattab]

Replying to EZdesign:

Attachment Overlay - links that are behind hovered menu are still displayed.png added

I don't think there's a way to avoid that. Does anybody know how we could tell (efficiently) when a link is behind another dom element? (By the way, Google's JS devs don't.)

Umh, that's a tricky one... Well, we could keep track of a list of rectangles, that are overwriting others, and use this array as a "blacklist" to hide (bubbles inside the blacklist X-Y but not a children of the DOM node so they must be behind it) ?

[mattab]

What is the "Top Page URL"? The one with most hits? Might be counter-intuitive in many cases. The solution above is the best we could come up with. Maybe there's something we're missing...

Yes the one with most hits. but we don't really need this yet, as I would prefer removing submenu by default (worth keeping on the ideas list to to load the "Top Page URL" as it would be useful for those enabling the menu).

• I was using Piwik correctly: I set "Alias URL" to example.com but I was only tracking example.com/dir -- so when Overlay loaded with example.com it failed and I was stuck and could not use Overlay.

That's the exact problem we were facing with my client's website. That's why I added the config option and they launch it from the actions report. I think it would help to add another config option that would hide Overlay from the menu. With these two options combined you could make it only accessible via the pages report and this way make sure that it is only openend for pages tracked by piwik. What do you think?

Agreed these 2 options are important. Overlay submenu does not feel like it belongs like other "Action types"...

I have thought hard about this and here is what I think would be best functionnality:

• Config Option (later possibly a User preference) to Display the "Actions > Overlay" menu --> Disabled by default (less UI clutter)
• By default, clicking on the Page Name or Page Url should I think open Page Overlay Full Screen!!! YEP!
• The orange icon should still link to the URL itself (since it's the meaning of the icon)
  • On hover on the orange icon, display tooltip "Open this $actionType in a new tab". This way we are consistent and display the tooltip on all possible actions for the row.
• On Hover on a Page name, the tooltip displays "Open Page Overlay for $pageNameOrUrlpath"
  • The page overlay would load in the same tab, opened full screen by default. Back button would work and return to Page report.
  • It is similar to insight_launch_from_pages_report=1 except it would open full screen + show tooltip + orange icon still link to URL in new tab!

What do you think about this proposal?

[mattab]

(In [7443]) Refs #2465 Looks like a bug in the menu code, but 99 works...

[timo-bes]

once the API is final and all other core things are done, it wiill be nice to add integration tests (but better focus on functionnality first)

Acutally Overlay doesn't really have an API. It's just a proxy for other API methods with cookie authentication. I'd rather write good test cases for Transitions.

i would like to see a new discreet option "Show Clicks instead of % clicks"

If you have 13472 clicks, it's too much text for the bubbles. Maybe we can show the total number in the lower right on hover?

Could be fun to have similar a mode "Highlight All Links" that would put a dark grey overlay on the site and only show the bubbles.

The "Color Overlays" would have the same effect with additional benefits. It's on the list of improvements.

Have you tried it on GA?

in full screen, there should be a way to "collapse" the sidebar. This is because, some websites are distorted with the sidebar on the left

Why not use the new tab feature when the sidebar distorts the website?

Attachment Overlay - links that are behind hovered menu are still displayed.png added I don't think there's a way to avoid that. Does anybody know how we could tell (efficiently) when a link is behind another dom element? (By the way, Google's JS devs don't.) Umh, that's a tricky one... Well, we could keep track of a list of rectangles, that are overwriting others, and use this array as a "blacklist" to hide (bubbles inside the blacklist X-Y but not a children of the DOM node so they must be behind it) ?

And how would you build such a list? Also keep in mind that DOM elements can be added, removed and repositioned at any time. I don't see any other way than processing every DOM element every two seconds. That would slow everything down a lot...

I would prefer removing submenu by default / clicking on the Page Name or Page Url should I think open Page Overlay Full Screen

First of all, it won't work for the page name because they are not URLs.

In general, I have to think about this for a while... Launching Overlay only from the pages report would be very different from the way other analytics tools do it. It would have andvantages but it would also make it harder to use for people coming from other tools because the workflow they know is opening it via the menu and then clicking throught the website.

[timo-bes]

(In [7444]) refs #2465

• adding remove() method for following pages
• it can be used to remove dom elements, timeouts and events. afterwards, a new implementation can be pasted into the console. this makes debugging without access to the server possible.

[timo-bes]

Replying to matt:

(In [7443]) Refs #2465 Looks like a bug in the menu code, but 99 works...

Doesn't work on the demo - at least not when signed in.

[timo-bes]

(In [7445]) refs #2465

• Renaming Insight to Overlay
• Includes a new minified piwik.js, generated as described in js/README. Please double-check whether the file is OK.
• Servers that had the Insight plugin installed will need to delete it manually, right?

[timo-bes]

(In [7446]) refs #2465 this was missing in the last commit

[timo-bes]

i would like to see a new discreet option "Show Clicks instead of % clicks" If you have 13472 clicks, it's too much text for the bubbles. Maybe we can show the total number in the lower right on hover?

I just realized that this is nonsense. The absolute number is shown on hover. But the point that absolute numbers are too large for the bubbles is still valid.

[timo-bes]

(In [7447]) refs #2465

• adding url normalizer tests to tests/javascript
• note: two of the other test cases are failing (JSLint and tracking:"ecommerce view multiple categories")

[mattab]

(In [7448]) Refs #2465 Will test on demo + Cleaning up Insight plugin from config file + filesystem on update

[mattab]

(In [7449]) Refs #2465 piwik.js is different but looks like white space difference

[timo-bes]

I tried the patch for tracking only pageviews as previous actions. It does the job but there is a problem:

Transitions relies on the fact that there are as many following actions as pageviews. This is used to derive the number of exits and to determine the height of the connections. Changing the tracking behavior requires a change in the display logic of transitions and (more imprtantly) i think it would require an additional query to derive the number of exits in transitions.

This is a quite though challenge but the data quality for a list of outlinks / downloads is really bad at the moment so we should try to do it somehow...

[timo-bes]

How about we drop the approach to show Overlay inside Piwik completely?

• From the pages report, we link to ?module=Overlay... instead of ?module=CoreHome...#module=Overlay...
• This would always open in a new tab
• The "Open in new tab" link would become "Hide sidebar".
• We add a link box to website that contains "Show sidebar" and links back to ?module=Overlay...
• To launch Overlay on the home page, we could add a widget that only contains this link.

Consequences:

• No more menu item "Page Overlay"
• Always full screen
• Mode 1: piwik-url.com/?module=Overlay... with sidebar
• Mode 2: acutal-website.com/... without sidebar
• Mode 1 and 2 can be toggled, no matter how the session was started (would also work for #overlay later)

[timo-bes]

(In [7453]) refs #2465

• when Piwik is used over SSL, load the website in the frame over SSL too
• when nothing comes back from the iframe, show an error message after a timeout (8 seconds)
• improved handling of overlayUrl a bit (though this should be obsolete soon when we only have full screen mode)

[timo-bes]

(In [7454]) refs #2465: followingpages.js improvements

• don't use a css class to remember which links have been discovered anymore. websites might change the class.
• when a link is displayed inline and has only child, use the child as a reference for the position

[timo-bes]

(In [7455]) refs #2465 minor bugfix for followingpages.js

[mattab]

Could be fun to have similar a mode "Highlight All Links" that would put a dark grey overlay on the site and only show the bubbles.

The "Color Overlays" would have the same effect with additional benefits. It's on the list of improvements.

Have you tried it on GA?

I just did, it's pretty cool! I think the site behind should be even a bit darker. It would be nice to have, and useful help to the analysis

in full screen, there should be a way to "collapse" the sidebar. This is because, some websites are distorted with the sidebar on the left

Why not use the new tab feature when the sidebar distorts the website?

because it opens in a new website, and is much slower (have to wait page load again), than if the sidebar was simply collapsed... +1 for sidebar collapse!

Attachment Overlay - links that are behind hovered menu are still displayed.png added I don't think there's a way to avoid that. Does anybody know how we could tell (efficiently) when a link is behind another dom element? (By the way, Google's JS devs don't.) Umh, that's a tricky one... Well, we could keep track of a list of rectangles, that are overwriting others, and use this array as a "blacklist" to hide (bubbles inside the blacklist X-Y but not a children of the DOM node so they must be behind it) ?

And how would you build such a list? Also keep in mind that DOM elements can be added, removed and repositioned at any time. I don't see any other way than processing every DOM element every two seconds. That would slow everything down a lot...

It would work only on timer, indeed... Ok for not doing it, it's fine as it is!

In general, I have to think about this for a while... Launching Overlay only from the pages report would be very different from the way other analytics tools do it. It would have qandvantages but it would also make it harder to use for people coming from other tools because the workflow they know is opening it via the menu and then clicking throught the website.

Ok I let you think... I don't like "Overlay" in "actions" menu but I also agree that it would be nice to have it in one menu somewhere...

Note: To allow discover this feature further, we could even add row evolution in the "All Websites" dashboard and default the "Overlay" action to open the default Site URL overlay.

[mattab]

Replying to EZdesign:

i would like to see a new discreet option "Show Clicks instead of % clicks" If you have 13472 clicks, it's too much text for the bubbles.

I propose 14.5K or 1.1M (0 or 1 decimal based on space available)

There's a similar prettyfying function in Piwik::getPrettySizeFromBytes() used in DBStats report.

I think this would be awesome! many people like clicks more than % (and both separately are really useful)

[mattab]

Replying to EZdesign:

I tried the patch for tracking only pageviews as previous actions. It does the job but there is a problem:

Transitions relies on the fact that there are as many following actions as pageviews. This is used to derive the number of exits and to determine the height of the connections. Changing the tracking behavior requires a change in the display logic of transitions and (more imprtantly) i think it would require an additional query to derive the number of exits in transitions.

This is a quite though challenge but the data quality for a list of outlinks / downloads is really bad at the moment so we should try to do it somehow...

Personnally I prefer the solutoin patch + wrong "Exit" data in Transitions rather than the current solution...

I agree it will create bit of wrong data in transitions for exits, but it's really cool to not miss clicks from pageA -> PageB because of intermediate downloads. People who need the exit rate for a given page can use the exit_rate from the Actions report. IN fact I'm sure they will...

I am not keen to add new queries to fix this. We would better even query the Page URL metrics and derive exits from there rather than add a new query...

But +1 for applying patch and figuring out what to do next, after ;)

[mattab]

Replying to EZdesign:

How about we drop the approach to show Overlay inside Piwik completely?

• From the pages report, we link to ?module=Overlay... instead of ?module=CoreHome...#module=Overlay...
• This would always open in a new tab

That could be cool, but somehow I feel it's better how you did , and add a user option (bottom of sidebar?) "Start Overlay in Full Screen mode" recorded per user. This way I feel will be most usable and fit everybody (I would set to full screen default but many users prefer not to not have the "context" and whole thing change).

• The "Open in new tab" link would become "Hide sidebar".

I wanted to suggest that too. But, it would also work when looking at piwik menu + overlay: it owuld be nice to hide sidebar there to at least display 100% of width even if piwik menu still take height space.

• To launch Overlay on the home page, we could add a widget that only contains this link.

nice idea, to have a widget "Page Overlay" with an input field containing by default the top Page URL, when click "Open Overlay" button which would load the Overlay Full Screen.

• We add a link box to website that contains "Show sidebar" and links back to ?module=Overlay...

When the overlay is viewed full screen, by default I would propose:

• Show the OVerlay in the "Actions" Menu, maybe on the far right of the menu and with an icon, to show clearly it is a differnt kind of report? we would have to hack the menu but it should be possible... Thoughts about displaying "Page Overlay" on the far right Actions submenu?
• Display the sidebar on the left by default
• "Hide Sidebar" would slide-in the sidebar to the left. On hover on the left 20 pixels or so, an icon would allow to display sidebar again? Otherwise, back button would work too, to come back to previous Piwik reports for example.
  • Ideally the Hide Sidebar feature would also be available on a click on the sidebar itself, which would be a few pixels wide clickable sidebar.

In full screen a mode "Shadow website + Highlight link" will be amazing! I like the bubbles and borders and I think with a nice background color inside, and a black opacity overlay below, could look amazing and inviting for users to really SEE how "clickable" their website is, and what is being used. It's all about picking right colors now, see email ;)

• Mode 1: piwik-url.com/?module=Overlay... with sidebar
• Mode 2: acutal-website.com/... without sidebar
• Mode 1 and 2 can be toggled, no matter how the session was started (would also work for #overlay later)
• Mode 1) is also required for #3536 I believe, which will be awesome to have for Piwik Mobile to view Page Overlay
• Mode 2) what is the different with #overlay or is it the same?

I would add:

• Mode 0: default: Add "Page Overlay" to submenu far right, by default shows sidebar and piwik menu, add footer user setting to open fullscreen by default, URL would be piwik-url/?module=CoreHome..#module=Overlay

Ok it's a long talk here but it's important to get this right.... this could be a super useful analytics feature to allow to discover easy improvements to one's on-sit optimizations!

[mattab]

Bug report:

• When looking at overlay for a site, I realize Overlay is really cool and want to look at another site overlay. The URL still contain the overlayUrl from the previous site, causing the message "You are attempting to open Page Overlay for the URL "http://www.X". None of the URLs from the Piwik settings matches the link. Please add the domain there. ```
  • overlayUrl should be removed when switching websites or clicking on any other menu
  • Message can be slightly improved:
  • If the user is admin for the site Piwik::isUserHasAdminAccess($idSite) then the message could link to: module=SitesManager&action=index with hash tag: #row5 for idsite=5 --> this way they can directly edit the settings and we make the feature pain free!

[timo-bes]

(In [7460]) refs #2465 improved followingpages.js

• remember that links have been discovered for hrefs that are not in the following pages
• check whether using child image for offset fails and use the link instead
• prevent double-tagging of links the hard way (happens in very rare cases)
• remove() method didn't unbind window resize event previously which led to behavior that looked like bugs

[timo-bes]

The previous commit should fix all bugs the have been reported concerning the bubbles. There still are problems with sliders that rely on a container with overflow:hidden. I won't fix that for now because it would be complicated and expensive (in terms of CPU).

I didn't check in all browsers yet so there still might be browser-related problems.

[mattab]

(In [7467]) Refs #2465 Will test on demo

[timo-bes]

• Message can be slightly improved:
  • If the user is admin for the site Piwik::isUserHasAdminAccess($idSite) then the message could link to: module=SitesManager&action=index with hash tag: #row5 for idsite=5 --> this way they can directly edit the settings and we make the feature pain free!

Great idea! The only problem is that #row5 links to fifth row, not the row with idSite=5. So how could else could we link to right row in the table?

[timo-bes]

(In [7470]) refs #2465

overlay has two modes now: full screen with sidebar & full screen without sidebar. for now, it is only accessible via a new row action in the pages report.

because of this change, some mostly not-so-nice core-changes could be removed (mainly that's good because overlay doesn't impact as many things outside the plugin anymore):

• disableHistory option of broadcast
• handling for parameter overlayUrl in broadcast
• openLinksInOverlay() method of html table
• config option overlay_launch_from_pages_report
• hack for data table to launch overlay from the links

other changes:

• error message for wrong domain has nice styles and links to admin page when the user has admin privileges
• when registering row actions, the order can be set
• made row action style a little more compact
• increased min width of label column a bit

[timo-bes]

(In [7471]) refs #2465

• simple date selection for sidebar mode
• code formatter applied (whitespace changes)

[timo-bes]

(In [7472]) refs #2465 iframe dimension fix

[timo-bes]

(In [7473]) refs #2465

• row evolution and transitions can be launched from the sidebar
• when launching overlay, always pick date "current month"
• make sure there never are two scrollbars when the window is not high enough

[timo-bes]

(In [7477]) refs #2465

• remember the location of the iframe in the l parameter using the core broadcast methods => popoverParam works as well => transitions and row evolution can make use of the back button
• row evolution and transitions get a static launch method that is used in overlay to trigger the popovers without any knowledge of their implementation
• broadcast.propagateAjax() gets the disableHistory parameter back but the handling is much easier this time
• remove full screen link for now because we might want to do that differently - i'll add some details to the ticket later
• two fixes for sanitized urls

[timo-bes]

(In [7479]) refs #2465

• support for internet explorer 7 & 8
  • redirect by setting window.location.href loses referrer - use fake link
  • implementing position:fixed for status bar in java script
  • when visibility=inherit, go up the dom to find out whether an element is visible
  • avoid .data() completely in followingpages.js
  • some specific html and css
• adding v parameter to loaded script to invalidate caches
• regenerating piwik.js (i hope it worked properly this time)
• hiding notifications that get stuck after removing a hovered element (only happens sometimes)
• binding window.resize after every broadcast.pageload() - the binding was lost after opening a popover
• fixing mode with frames disabled: didn't work after migration to the l parameter

This version is the first one that I think is stable. Please deploy on the demo so we can do some more testing.

The next step is translation and documentation. After that, we're ready to release as beta.

[timo-bes]

(In [7482]) refs #2465 link new doc page from overlay error message

[mattab]

(In [7483]) Refs #2465 Will test on demo

[timo-bes]

(In [7486]) refs #2465: using Piwik::isHttps()

[mattab]

(In [7493]) Refs #2465 FIxing " The only problem is that #row5 links to fifth row, not the row with idSite=5"

[mattab]

(In [7516]) Refs #2465 Will test on demo

[mattab]

(In [7519]) Refs #2465 fixing error - but i'm afraid it still not loading on https page on demo.

[mattab]

(In [7528]) Refs #2465 Will test on demo - this beta should be fairly stable!

[mattab]

(In [7534]) Fixes #3572 Refs #2465 Overlay work on SSL on demo! NICE!

[mattab]

See #3530 List of Page Overlay Improvements for more info and the current state of the art!

[halfdan]

I have a bug (?) report for Page Overlay, sadly Trac blocked my post so I reposted it on pastebin: http://pastebin.com/DtwLvPXi

[mattab]

Error is: Strict Notice:</strong> <i>Only variables should be passed by reference</i> in <b>/var/www/htdocs/piwik.c0demonkey.com/html/plugins/Overlay/API.php</b> on line <b>116</b>

[mattab]

(In [7712]) Refs #2465 Try fixing notice - does it fix it @halfdan?

[mattab]

(In [7713]) Refs #2465 that's better I think

[mattab]

Marking as fixed. Nice work Timo! Hope to see some improvements in Page Overlay later!