UserCountryMap: zoomable world map of your visitors location

[gka]

This plugin adds a worldmap widget to your dashboard. The map is rendered in flash (swf size: 216kb) and has no dependencies to other services like google maps.

the plugin load the data via API call from UserCountry plugin.

by now, the plugin is able to dsiplay the following data:

• unique visitors
• visits
• actions
• max actions
• visit duration

I built in PNG export and tried to adapt the look and feel of the OpenFlashCharts used in the DataTableView component.

[gka]

Attachment: minor bugfixes: filter_limit=-1, mac/linux font issues, multiline tooltips UserCountryMap.zip

[gka]

Attachment: a few suggestions for the design of the map legend piwik-legend-layouts.zip

[gka]

Attachment: second iteration of legend layout, this time with a vertical bar that has a fixed position at the lower left, which should look fine for all continents piwik-legend-layout-left.zip

[gka]

Attachment: added error message for #1088 UserCountryMap-0.5.zip

[mattab]

Attachment: Map is empty but countries widget works Piwik Web Analytics Reports.png

[mattab]

Attachment: XML returned (seen in firebug) but map is blank index.php.xml

[gka]

Attachment: UserCountryMap-0.6.zip

[gka]

Attachment: updated screenshot, now with legend mapplugin.png

[mattab]

Attachment: Bug on IE.PNG

[gka]

Attachment: version 0.7: added fullscreen button, displaying buttons through flash instead of javascript to get rid of the flashplayers security exceptions on fullscreen UserCountryMap-0.7.zip

[halfdan]

Greg, that thing is beautiful! Well done!

• The small "Export as PNG" button has "General_ExportAsImage" as label.
• If you have the World Map on the leftern side and click on "Add Widget" or "Data range" the flash widget is over the expanded <div>.

Apart from that it looks just great! Did you develop the PiwikMap.swf and if not how is it licensed?

[gka]

thanks :)

"General_ExportAsImage" is part of the german translation and seems to be missing in the english lang file. Maybe this isn't a bug of my plugin.

The second bug is now fixed by setting the flashplayers wmode to "opaque".

Yes, I developed the PiwikMap.swf by myself. The only external library I used is the as3corelib (http://code.google.com/p/as3corelib/), which is released under New BSD license.

[anonymous-matomo-user]

Great Plugin and a real alternative to the buggy Maps plugin. A question: Do you plan to enhance functionality, maybe with use of plugin GeoIP so that it is able to show the users based on cities?

[gka]

Good idea. I haven't looked at the GeoIP plugin yet. If it turns out that it works fine and there some kind of xml api to request city data for a given country, I'll try to include this in the UserCountryMap plugin.

[anonymous-matomo-user]

wow. I really like this new widget! Perfect. Thank you!

[mattab]

Very nice work! Great job also on getting the open source map. I think this is unique as far as I know!

I have a few questions

• Are you interested to have your extension in Piwik core? This would definitely be a great feature to have (default widget in dashboard + map integrated in existing Country report).
• What did you base your map on? How accurate is it? How many countries are registered in the map?
• Can we see code source of SWF? Could you release it with the plugin or in github or SF?
• Code review: the Tpl/JS code should be in a template file
• Suggestion: while I quite like the new icons, I think it is not a good idea to add a new visualization of entities. Instead, it might be easier to read if the Metrics was a simple SELECT box that you could select which metric to plot. This would be more usable yet still as cool.
• Could a click on a non colored country zoom out as well?
• It would be nice to have the scale at the bottom (with min/max number printed out next to scale), this would give a better feel for what the colors mean
• there seem to be 3 level of zoom - but maybe 2 zoon levels would be enough: world + continent with pre-selected 5 or 6 continents (fixed XY boxes)? Great work for sure!

[gka]

Ok, I'll try to get through your questions

• Yes, it would be great to see this plugin in Piwik core!
• The map data comes from [http://finder.geocommons.com/overlays/5603] and the data file contains 197 countries. You can see the full map accuracy if you open the KML file in Google Earth. Since the full data set would be too large (> 2mb) I did some simplification on the map data, which reduces the accuracy. this is something we can do further experimentation, the currently used simplification algorithm is very simple and could be replaced with better ones. Another idea is to load more detailed country data at runtime, eg right after the user started zooming into the map.
• Yes, you can see the source, but I have yet little concerns about releasing the entiere source for free. Basically, the extension consist of three parts: the php-piwik integration, the SWF map renderer and an AIR app which reads the shapefile and performs the simplification steps. The latter generates a binary file (AMF) that contains the projected outlines and the entire map topology (country codes, continents, special zoom regions, etc). The SWF map renderer embeds the binary data during compilation, which allows it to quickly render the map without needing to do heavy calculations. You see that there is a two-staged process, one stage generates a binary map file, the other one does the rendering. It would be no problem to release the complete rendering-code, but still I need to think about releasing the mapdata preprocessing.
• The select box is a good idea. I was just thinking about including more metrics like the bounce rate and that would lead to more confusions about the icons meanings. A select box would be more scalable and easier to maintain.
• The scale at the bottom is a good idea, I'll add this feature later on as well as the zoomout on clicks on non-clickable regions.
• Code review: It was very hard for me to get through the Piwik framework the first time. I'll try to integrate this template stuff.

There is one more thing I would like to add to the plugin. I just looked at the GeoIP plugin and wouldn't it be nice to add a detailed country-view in which all visitor cities are displayed like in GA? This would require more thinking about the zooming interface, but it shouldn't be a big problem.

Do you know if the GeoIP plugin is going into Piwik core as well? I tested the plugin but while it correctly updates the tables with the city/lat/long information, it seems that the rest of the plugin doesn't work right (API call returns no data, etc).

[mattab]

Answers

• there seem to be 3 level of zoom - but maybe 2 zoon levels would be enough: world + continent with pre-selected 5 or 6 continents (fixed XY boxes)?
• I asked for code source to ensure that it is Free Software :)
• GeoIp core support is not on the 1.0 roadmap (it will be priority just after 1.0) so let's not worry about this one just yet - better make the existing functionnality good for core :)
• using templates: I understand there is a lot of code. Check out simple example of how to use a template in ExampleFeedburner: https://github.com/piwik/piwik/blob/master/plugins/ExampleFeedburner/ExampleFeedburner.php#L69
• glad you agree with my feedback. It can be in Piwik 1.0 but we are feature freezing this week (in 3 or 4 days) so if you want to get it in core quickly, this is the time to finish it :) (post 1.0 release including new feature will probably be in a couple months after 1.0 at least)
• country precision sounds OK for V1 as well

Looking forward to your update!

[gka]

Updates:

• replaced the cool icons with the usable selection box
• enabled zoom out after clicking on inactive countries
• fixed zoom region "north asia"
• fixed translation bugs
• put html/js code into templates
• you can now find the source code on my bitbucket account

To answer your question on the zoom levels, by now there are only zwo zoom level: world + region, where region means almost the same as continent, except for africa and asia (which are both split into two parts for better zooming experience).

[mattab]

Excellent updates. Thanks for fixing it quickly :)

• filter_limit=200 should probably read filter_limit=-1 ?
• you can remove images from the build
• The only problem left is that you hardcode metrics and build the SELECT statically. Instead, you can use the new awesome Metadata API to fetch a given report metadata. It will directly return the column names and translations for the given report (in your case, UserCountry.getCountry). You can export this as Json as well if you want so directly do the logic in JS.

For example in trunk

http://localhost/trunk/index.php?module=API&method=API.getProcessedReport&idSite=1&apiModule=UserCountry&apiAction=getCountry&format=xml&token_auth=0b809661490d605bfd77f57ed11f0b14&language=en&date=today&period=year

returns:

<?xml version="1.0" encoding="utf-8" ?>
<result>
    <website>BuyForSeniors</website>
    <prettyDate>2009</prettyDate>
    <metadata>
        <category>Visitors</category>
        <name>Country</name>
        <module>UserCountry</module>

        <action>getCountry</action>
        <dimension>Country</dimension>
        <metrics>
            <nb_uniq_visitors>Unique visitors</nb_uniq_visitors>
            <nb_visits>Visits</nb_visits>
            <nb_actions>Actions</nb_actions>

        </metrics>
        <processedMetrics>
            <nb_actions_per_visit>Actions per Visit</nb_actions_per_visit>
            <avg_time_on_site>Avg. Time on Website</avg_time_on_site>
            <bounce_rate>Bounce Rate</bounce_rate>
            <conversion_rate>Conversion Rate</conversion_rate>

        </processedMetrics>
        <metricsGoal>
            <nb_conversions>Conversions</nb_conversions>
            <conversion_rate>Conversion Rate</conversion_rate>
            <revenue>Revenue</revenue>

        </metricsGoal>

        <processedMetricsGoal>
            <revenue_per_visit>Value per Visit</revenue_per_visit>

        </processedMetricsGoal>
        <uniqueId>UserCountry_getCountry</uniqueId>

    </metadata>
    <columns>
        <label>Country</label>

        <nb_visits>Visits</nb_visits>
        <nb_actions>Actions</nb_actions>
        <nb_actions_per_visit>Actions per Visit</nb_actions_per_visit>
        <avg_time_on_site>Avg. Time on Website</avg_time_on_site>
        <bounce_rate>Bounce Rate</bounce_rate>
        <conversion_rate>Conversion Rate</conversion_rate>

        <revenue>Revenue</revenue>

    </columns>
    <reportData>
        <row>
            <label>France</label>
            <nb_visits>12634</nb_visits>
            <nb_actions>12634</nb_actions>

            <revenue> 0</revenue>
            <conversion_rate>6.28%</conversion_rate>
            <nb_actions_per_visit>1</nb_actions_per_visit>
            <avg_time_on_site>00:00:10</avg_time_on_site>
            <bounce_rate>100%</bounce_rate>

        </row>
[...........]

In trunk currently we don't display anymore the (useless) raw metrics: bounce count, total time spent. but instead we display interesting processed metrics from these (bounce rate and avg visit duration). By using this magic API method instead of direct call, you ensure that all the data is pre-processed and columns are set properly (with the %, the revenue sign, right precision, etc.) all consistent with other reports.

Let me know if you have any question. Appart from that, code looks good and I'll be happy to commit :)

[gka]

Yes, there is a question. I understand that the metadata API is a huge time-saver for many developers, but isn't it a violation of the MVC pattern? As the view code of my plugin would have no more access to the raw data, it would have to re-parse this data out of the formated strings. For example, the SWF map needs the avg_time_on_site value two times: once to calculate the right color for the countries and once to show it in the tooltip. To do the color-calculation, it now would have to parse the total number of seconds out of the HH:MM:SS string. Or even worse, it would have to guess the countries iso-code out of it's translated name. I don't think this is the intention of the metadata API. Is there a way to force it to include both the raw (but preprocessed) values (ids, integers, floats) and the formatted values (translated and formatted strings)? Something like this would be helpful:

<reportData>
    <row>
        <label value="fr">France</label>
        <nb_visits value="12634">12634</nb_visits>
        <nb_actions value="12634">12634</nb_actions>

        <revenue value="0.0"> 0</revenue>
        <conversion_rate value="0.0628">6.28%</conversion_rate>
        <nb_actions_per_visit value="1">1</nb_actions_per_visit>
        <avg_time_on_site value="10">00:00:10</avg_time_on_site>
        <bounce_rate value="1">100%</bounce_rate>
    </row>

[gka]

Another remark (which may be off-topic here): I don't think that the avg. time on website is a sufficient replacement for the total time on website. Both values are very interesting on its own. Also there isn't any other replacement of the total time on the site, neither the number of unique visitors nor the number of actions. Therefore I would suggest to keep the total time on site inside the new metadata API.

[anonymous-matomo-user]

I think the bar design for the legend is better than the block version.

[mattab]

• In the API output, I forgot to include the reportMetadata bit which indeed includes the country ISO, flags,etc. as follows

[...]
    <reportMetadata>

        <row>
            <code>fr</code>
            <logo>plugins/UserCountry/flags/fr.png</logo>
            <logoWidth>18</logoWidth>
            <logoHeight>12</logoHeight>

        </row>

        <row>
            <code>gb</code>
            <logo>plugins/UserCountry/flags/gb.png</logo>
            <logoWidth>18</logoWidth>
            <logoHeight>12</logoHeight>

        </row>
[..]

• There is unique Visitors data, however it only works for Days (as with all reports in Piwik). You need to manually remove the 'Unique Visitors' from the metrics list when period != day. The advantage of using this method is that this is done automatically for you, the 'columns' list contains ID + translation of all displayable columns.
• I agree with you that total actions & total time on site (as well as other metrics) could be in the output and could be interesting. Also agreed that time should also be passed raw (in seconds). I added this to the metadata todo list #1491
• I assume you can parse the following (simply removing non numeric characters)?

<bounce_rate>100%</bounce_rate>
<revenue> 0</revenue>

• maybe there is a function to parse time HH:MM:SS until the API provides the metric? If not trivial, let me know and I will work onincluding the raw metrics in second, in the list of report metadata.

Others

• expressInstall.swf is already in libs/swfs/ - could it be reused?
• img/* can be removed I think
• I see a full screen icon - could that work? that would be fun for sure

Keep up great work and let me know your thoughts

[gka]

• ok, good to have the iso code, this is important
• the other values could also be parsed, it's no problem to get the number of seconds out of an HH:MM:SS string, but I'm concerned that there may be some localization (say chinese) with different time formats
• values like 100% and currencies are no problem
• fullscreen is already working via context menu. unfortunately there is a security fix that forbids setting a flashmovie to fullscreen mode via javascript, so this functionality can't be accessed through an icon
• I'll remove the img folder and update the link to the libs/swfs/expressInstall.swf

[gka]

I think I need a newer version of the API plugin. Mine doesn't include a method called getProcessedReport and it isn't listed in the api reference yet.

[gka]

Ok, this seems to be the right time to checkout the svn repository :)

[mattab]

• pretty time is not localized, so no worries for i18n there
• the map is so pretty it deserves a full screen view - could we have a small discreet icon that would only show (in grey or something) on Hover over the flash window?
• yes you need Trunk for these newer features, it has just been coded :)
• legend looks great!

Are there other bug reports that you can think of? Did you try in different languages containing non standard characters?

[gka]

I updated the repository and the new API now works. One thing I don't understand is that there is a column definition for _nb_uniqvisitors below <metadata><metrics>, but there is neither a corresponding column in <columns> nor is there some data below <reportData><row>. Is it simply missing?

[mattab]

I installed the new update but map won't load, and the error is displayed: error while loading contacts (unknown error #1088)

Does it work for you?

[gka]

error 1088 occurs if the xml document is not well-formed. it works for me.

[mattab]

Error was in the en.php file (missing comma)

But it still doesn't data for me even though I see the successful XML request (check out the file attached, maybe it's too much or something??).

[mattab]

Thought for future: it would be great to have the same refactored SELECT metrics option list in all Piwik graphs (vertical bar, pie chart). Having the graph only plot visits at the moment has always bugged me and your SELECT looks like the perfect solution!

[gka]

I tested your xml file and it worked fine. I also checked out the latest Piwik from SVN and it keeps working. Maybe it has to do something with the Apache or PHP settings. The flash xml parser is very strict and it goes down on the smallest charset issues. We got to fix that bug.

Nevertheless I now finished version 0.6. The main changes are:

• legend: The display of the legend should work now. It is only displayed, if the flash window is large enough, to avoid overlapping of the half map on small screens. The legend is always visible in exported images, since the map is then resized to 1000px width. If you zoom into the world, the legend might overlap some countries. In this cases you just have to move your mouse over the overlapped parts and the legend will disappear.
• optimized image export: There is a cool new feature in flashplayer 10 that allows instant image downloads, so one not need to go over the extra export image page where the user is told to right click the image. Instead, flashplayer will open a standard download dialog where the user can choose a filename and the download directory. However, like the fullscreen-mode, this feature cannot be accessed via javascript due to security reasons. Until we find a solution, the image button will work the old way and the context menu will work the new way.
• bug fixes: the bug in lang/en.php is fixed.

[mattab]

Hi greg, great work. Can you please email me at matt att piwik.org to discuss the next steps?

[mattab]

It's working now for me, not sure what the problem was before. Feedback

• Scale looks perfect
• The list of metrics is missing "Revenue"
• instead of Zend_Registry::get('access')->getTokenAuth() you can use Piwik::getCurrentUserTokenAuth()
• typo in selected="seleleced"
• Don't bother writing translations for it, official translators will see it and translate it when this is released
• is it trivial to add a grey fullscreen icon bottom right of the map, when hover (or clicked?) the map?

[anonymous-matomo-user]

after update to 0.6 I get following error:

Fatal error: Cannot use string offset as an array in /users/jmp/www/piwik/plugins/UserCountryMap/UserCountryMap.php on line 88

[mattab]

• bug in IE: see attached screenshot

[gka]

@matt:

• I wrote you a mail on the legend scale. I think there is more work to do, when considering sites with far more visitors (let's say > 1 mio). Also we could improve the usability by adding the values unit to the legend title, for instance "Bounce rate (%)" or "Visit duration (seconds)"
• The list of metrics is generated out of the xml api. If there is no Revenue in your list, then there is no <revenue> tag inside the <metrics> or <processedMetrics> tags. There is also a bug with the unique visitors metrics. It is listed in the metrics definition but there is no data inside the report
• Ok, I'll add the button in the next version

@beatgarantie:

Do you have the latest version of the API plugin installed? The plugin makes a internal api request on API.getMetadata().

[anonymous-matomo-user]

@greg: so, I have to wait for Piwik 0.6.5 to get the new API. Thank you for your fine widget.

[gka]

I just checked in the plugin into svn :)

[mattab]

Great stuff!

[mattab]

(In [2718]) Refs #1514 Minor updates, comments and Core credits. Also including translations in main translation files.

Also updated http://piwik.org/the-piwik-team/ and listed Gregor Aisch in the list of Contributors

[mattab]

(In [2809]) Refs #1514

[anonymous-matomo-user]

Hello Gregor, can you help me finding the package org.piwik.data.Metric? I need this to compile and test a possible fix to #1634.

[anonymous-matomo-user]

Here's a little patch to make the map show up in the Location&Provider submenu. It requires patching some files from Provider and GeoIP, unfortunately I could not find a way around that. Also, it is suffering from a bug in firefox, where flash content which has both with and height attributes in percent is not displayed after loading, but when the site is rendered anew, for example opening/closing firebug. I tried a few workarounds I found on the net, but none of them seem to work here.

* plugins/UserCountryMap/UserCountryMap.php
  replace function postLoad with:
  function postLoad()
  {
    Piwik_AddWidget('General_Visitors', Piwik_Translate('UserCountry_WidgetCountries').' ('.Piwik_Translate('UserCountryMap_worldMap').')', 'UserCountryMap', 'worldMap');
    Piwik_AddAction('template_headerUserCountry', array('Piwik_UserCountryMap','headerUserCountryMap'));
    Piwik_AddAction('template_footerUserCountry', array('Piwik_UserCountryMap','footerUserCountryMap'));
  }

  modify function worldMap:
  function worldMap($template='worldmap')
  {
    $view = Piwik_View::factory($template);

  add three functions in class Piwik_UserCountryMap:
  static public function headerUserCountryMap($notification)
  {
    $out =& $notification->getNotificationObject();
    $out = '<div id="leftcolumn">';
  }

  static public function footerUserCountryMap($notification)
  {
    $out =& $notification->getNotificationObject();
    $out = '</div>
      <div id="rightcolumn">';
    $out .= '<h2>'.Piwik_Translate('UserCountry_WidgetCountries').' ('.Piwik_Translate('UserCountryMap_worldMap').')</h2>';
    $out .= Piwik_FrontController::getInstance()->fetchDispatch('UserCountryMap','worldMapMenu');

    $PlugIns=Piwik_PluginsManager::getInstance()->getLoadedPlugins();
    if (!array_key_exists('Provider',$PlugIns))
    {
      $out.='</div>';
    }
  }

  function worldMapMenu()
  {
    $this->worldMap('worldmap_menu'); 
  }

* plugins/UserCountryMap/templates/worldmap.tpl
  modify line with swfobject.embedSWF, replace
  Math.round($('#UserCountryMap_content').width() *.55)
  with
  (Math.round($('#UserCountryMap_content').width() *.55)) || "55%"

  copy file to worldmap_menu.tpl, add in wordmap_menu.tpl two lines:

  at the beginning, this being the first line:
  {postEvent name="template_headerUserCountryMap"}

  at the end, this being the last line:
  {postEvent name="template_footerUserCountryMap"}

* plugins/Provider/Provider.php
  replace function postLoad with:
  function postLoad()
  {
    $PlugIns=Piwik_PluginsManager::getInstance()->getLoadedPlugins();
    if (array_key_exists('UserCountryMap',$PlugIns))
    {
      Piwik_AddAction('template_footerUserCountryMap', array('Piwik_Provider','footerUserCountryMap'));
    }
    else
    {
      Piwik_AddAction('template_headerUserCountry', array('Piwik_Provider','headerUserCountry'));
      Piwik_AddAction('template_footerUserCountry', array('Piwik_Provider','footerUserCountry'));
    }

  }

  add function:
  static public function footerUserCountryMap($notification)
  {
    $out =& $notification->getNotificationObject();
    $out='<h2>'.Piwik_Translate('Provider_WidgetProviders').'</h2>';
    $out .= Piwik_FrontController::getInstance()->fetchDispatch('Provider','getProvider');
    $out .= '</div>';
  }

* plugins/GeoIP/Controller.php
  replace
  $view->provider = Piwik_FrontController::getInstance()->fetchDispatch('Provider','getProvider');
  with:
  $PlugIns=Piwik_PluginsManager::getInstance()->getLoadedPlugins();
  if(array_key_exists('Provider',$PlugIns))
  {
    $view->provider = Piwik_FrontController::getInstance()->fetchDispatch('Provider','getProvider');
  }
  if(array_key_exists('UserCountryMap',$PlugIns))
  {
    $view->usercountrymap = Piwik_FrontController::getInstance()->fetchDispatch('UserCountryMap','worldMap');
  }

* plugins/GeoIP/templates/index.tpl
  replace

  <h2>Provider</h2>
  {if isset($provider)}
  {$provider}
  {/if}

  with:

  {if isset($usercountrymap)}
  <h2>{'UserCountry_WidgetCountries'|translate} ({'UserCountryMap_worldMap'|translate})</h2>
  {$usercountrymap}
  {/if}
  {if isset($provider)}
  <h2>{'Provider_WidgetProviders'|translate}</h2>
  {$provider}
  {/if}

* lang/de.php
  add:
  'UserCountry_country_' => 'Unbekannt',
  'UserCountry_continent_' => 'Unbekannt',

* lang/en.php
  add:
  'UserCountry_country_' => 'Unknown',
  'UserCountry_continent_' => 'Unknown',

[robocoder]

The changes to lang/*.php are wontfix. The problem is that GeoIP should return "xx" instead of "".

Note: this ticket is closed. Please open a new ticket (or an existing one for map improvements).

[mattab]

see the ticket to include world map in the country report: #1821