piwik.js enhancements

[robocoder]

With the JavaScript tracking API in #355 completed, we can turn our attention to some internals:

• optional file:// and framebuster
• send heart beat (or ping) when view duration reached on a page (e.g., 30 seconds) to differentiate from a traditional bounce

On the horizon:

• set a milestone for removing the deprecated legacy tracking interface (piwik_log and piwik_track)
• ditto for IE5 support (e.g., arguments.callee)

Deferred:

• to better support multiple trackers, install the link tracking once; not a common use case, so performance hasn't been an issue
• re-work unit tests to not depend on an eval(); we use DEBUG tags to exclude this from the minified piwik.js
• finish and document the hook (plugin) interface (see #1635); depends on use cases

Wontfix:

• serve a leaner version that uses features built into ECMAScript 5th Ed.; we need backwards compatibility to track with older browsers
• move piwik_ignore (and ignore class) filtering to the click handler; not sure this is worth doing as it depends on the number of links on the page

[mattab]

looks good, I just don't think we can set a milestone for removing the deprecated legact tracking: we can't force users to re-tag their websites (think how many thousands websites are concerned).

a great thing that you could do, would be to write a blog post summarizing how we built this cool new functionnality: aggregating user feature requests, bug reports, analyzing how competition works, multiple code reviews, how did you test the code on multiple browsers, writing javascript unit tests, packaging (YUI), etc.

What do you think?

[robocoder]

More ideas (mentioned elsewhere):

• async loading
• middle mouse button

To avoid distorted stats, provide optional support for:

• framebuster
• bing-buster (when page loaded from cc.bingj.com cache)
• file:// buster (when page loaded from a saved copy on user's desktop)

[robocoder]

(In [3163]) refs #739, refs #1694 - instead of a bing cache buster, detect when page is loaded from google or bing cache, and apply a fix to the url

[robocoder]

(In [3164]) refs #739, refs #1694 - fallback to the cache url if we can't parse it

[robocoder]

(In [3165]) refs #1694, refs #739 - make cacheFixup() testable

[robocoder]

(In [3167]) refs #1694, refs #739 - Yahoo's cache result is served from Inktomi allocated ip addresses

[robocoder]

(In [3172]) refs #739 - drop IE5 support (arguments.callee removed in ES5 spec but at present, no browser appears to implement strict mode)

[robocoder]

See http://forum.piwik.org/index.php?showtopic=16661 for Xaver's async mods.

[robocoder]

(In [3284]) refs #739 - add DEBUG markers so that the unit test scaffolding can be removed from the production script

[robocoder]

Note; Compressing piwik.js via yuicompressor currently results in a 9079 bytes long file (using the sed instructions in the README). For comparison, compressing via uglifyjs (which requires nodejs) results in a shorter file, only 8660 bytes long.

[robocoder]

(In [3285]) refs #739 - clarify that these are test hooks ... they were never practical in production due to name mangling by yuicompressor

[mattab]

Currently, the Bing / Google cache fixups are done in piwik.js. This is better than before, however it will result in inconsistent results when the Tracking API is used, as it will not go through piwik.js and therefore Bing/Google cache will not be tracked the same.

[robocoder]

(In [3374]) fixes #1842, refs #739 - implement asynchronous tracking (thanks Xaver); see updated documentation, http://piwik.org/doc/javascript-tracking/

[robocoder]

(In [3375]) refs #1842, refs #739 - implement push functions

[robocoder]

merging suggestions from comments into main ticket description; adding a couple for completeness

[robocoder]

(In [3686]) refs #739, refs 409 - updated to pass jslint 2011-01-06; includes the prototype first-party cookie implementation (not up-to-date with recent discussions with matt)

[robocoder]

Going to sleep on this. Too tired to decide whether this would be a feature or considered a security risk?

// automatically use a subdomain wildcard
if (!configCookieDomain && locationHostnameAlias.substr(0, 4) === 'www.') {
    configCookieDomain = locationHostnameAlias.substr(3);
} else if (isString(configCookieDomain) && configCookieDomain.substr(0, 4) === 'www.') {
    configCookieDomain = configCookieDomain.substr(3);
}

[robocoder]

(In [3783]) refs #409 - first party cookies

• API changes:
  • added: setCookieNamePrefix(cookieNamePrefix)
  • added: setCookieDomain(domain)
  • added: setCookiePath(path)
  • added: setVisitorCookieTimeout(timeout) - defaults to 2 years since last page view
  • added: setSessionCookieTimeout(timeout) - defaults to 30 minutes since last activity
  • added: setReferralCookieTimeout(timeout) - defaults to 6 months from the first visit
  • added: setConversionAttributionFirstReferer(enable)
  • added: getVisitorId()
  • for asynchronous tracking, use:

    var visitorId;

    _paq.push(function () {
        visitorId = this.getVisitorId();
    });

• Cookie notes:
  • The default cookie path is '/'. This might be viewed as a potentially insecure default because it allows cookies to be shared across directories on the same domain. (Again, see the social network example.) This is unfortunately, a necessity. If we leave the path blank, the behaviour is undefined (i.e., browser or browser-version dependent). For example, earlier versions of Firefox would default to '/'; later versions default to the origin path.
  • I was hoping to avoid this, but I added a hash to the cookie content similar to GA's setAllowHash(). This is needed for two reasons:
    1. Cookies are uniquely identified by the tuple (key,domain,path). Hashing only the domain is a bug. (See "social network website" use case.)
    2. There's a long-standing cookie+subdomain bug in Firefox (Gecko) dating back to 1.0 that leaks cookies from "example.com" (not ".example.com") to "xyz.example.com". @see https://bugzilla.mozilla.org/show_bug.cgi?id=363872
• changed internal setCookie() method to take expiry time in milliseconds (was days)

• removed internal dropCookie() method as it was never used

  @todo Missing unit tests and cross browser testing

refs #739 - piwik.js improvements

• jslint 2011-01-09
• new unit tests (integrated jslint, is_a functions, sha1(), utf8_encode(), etc)
• use ECMAScript String.substring() instead of non-standard (although widely supported) String.substr()
• implement domainFixup() so "example.com" and "example.com." are equivalent
• API changes:
  • added: killFrame() - a frame buster
  • added: redirectFile( url ) - redirect if browsing off-line, aka file: buster; url is where to redirect to
  • added: setHeartBeatTimer( delay ) - send heart beat 'delay' milliseconds after initial trackPageView(); set to 0 to disable
  • removed: piwik_log() - legacy tracking code; see trackLink()
  • removed: piwik_track() - legacy tracking code; see trackPageView()
  • removed: setDownloadClass() - deprecated; see setDownloadClasses()
  • removed: setLinkClass() - deprecated; see setLinkClasses()

refs #752 - track middle mouse button clicks (via mousedown+mouseup pseudo-click handler); defaults to tracking true "clicks"

• API changes:
  • modified: addListener( element, enablePseudoClickHandler = false )
  • modified: enableLinkTracking( enablePseudoClickHandler = false )

refs #1984 - custom variables vs custom data

@todo These are just stubs.

• API changes:
  • added: setCustomVar(slotId, key, value, opt_scope) - scope is 1 (visitor), 2 (sesson), 3 (page)
  • added: getCustomVar(slotId)
  • added: deleteCustomVar(slotId)
• API changes for consistency:
  • added: setCustomVar(slotId, obj, opt_scope)
  • added: setCustomData(key, value)
  • for the equivalent of deleteCustomData(), use:

    tracker.setCustomData(null);

[mattab]

Reopening, as I have questions / clarifications

• what is the use of utf8 new function? it seems it is only used in sha1, should it be a function? Im not sure if we need to handle this just for the hash algorithm
• killFrame - what is the use case, should this be part of piwik.js? when enabled, could break functionnality such as google translate etc.
• setHeartBeatTimer - revert for now, as feature is not ready. I created a new ticket: #2041 (right now, it will record one page view for each request every 30 sec, instead, ping requests should have a special flag, and in piwik.php it should only update the visit_total_time, not do anything else)
• how does addClickListener work when enable==true ? it looks like it does the same thing as before, but calling the code twice

addEventListener(element, 'click', clickHandler, false);
addEventListener(element, 'click', clickHandler, false); 

[robocoder]

(In [3788]) fixes #739 - addEventListeners() on mouseup and mousedown; good catch, Matt!

• utf8_encode() - required to convert multibyte char to string of bytes; for unit testing, it needs to be a separate function
• killFrame() - would be called after a trackPageView(); used to prevent one's site from being frame by third-parties; there are tradeoffs (as you point) out, so it is not enabled by default

refs #2041 - setHeartBeatTimer - uses setTimeout, so it happens only once (vs setInterval is recurring); only happens on the landing page; intended to be used to improve bounce rate report; changed it to send ping=1 in the request

[robocoder]

(In [3792]) refs #739 - updated to jslint 2011-01-21 which has coding style checks added (e.g., indentation)

[robocoder]

(In [3793]) refs #739 - use JSON reference implementation given that we included both stringify() and json_parse(); some minor mods for greater browser portability

Also:

• rework ping to address scope problem in IE5.5/IE6
• update QUnit test framework
• remove jQuery dependency from unit tests
• reorder unit tests
• increase repeatability (e.g., asynchronous tracking requests may not be sent before request for results)

[robocoder]

(In [3795]) refs #739 - add Jash - JavaScript shell (GPL) to tests/javascript

[robocoder]

(In [3796]) refs #739 - prevent multiple instances of Jash; use the self-hosted copy of Jash.css; include missing icon