Improve SEO analytics in Piwik

[mattab]

This ticket is a place-holder for all SEO Analytics feature requests.

We've narrowed down scope of this ticket and here are the specifications for Piwik 1.X:

• SEO data points (Pagerank, page indexed,etc.) should be each recorded as "archive_numeric" fields (all of them being "integer" values).
• The "seo archiving" code should not hook on the standard daily archiving, but instead be a scheduled task running once a day and pre-archiving all websites SEO (ie. running with "HIGH" priority so it's pre-processed before scheduled reports (which use SEO data))
• These values vary over time. The evolution and changes are critical to the health of the site. The data will be updated and automatically archived, in a scheduled task running once a day.
• One data point is an exception: "Domain age" should be stored in the Piwik_Option table, once for each website (store, time of domain creation in unix timestamp)
• The API will then query these numeric values using existing functions eg $archive->getDataTableFromNumeric(array('pagerank','pages_google','pages_bings','alexa'))
• Currently the widget(https://demo.piwik.org/index.php?module=Widgetize&action=iframe&widget=1&moduleToWidgetize=SEO&actionToWidgetize=getRank&idSite=1&period=day&date=2013-03-09&disableLink=1&widget=1) loads super slow. After this change, if cron is setup, the widget will render instantly. This is awesome.
• Bonus: with the new code, it should be possible to include the "Row Evolution View" for this report! a new "View Evolution [icon]" link, will open the awesome existing multi row evolution report, in a popover, showing history of seo metrics!
• Add metadata support for SEO plugin, so that the SEO report shows up in Piwik Mobile & Scheduled reports.
• Periods display the stats for last day of period. Show last day in light grey.

[halfdan]

I would very much like to see a widget displaying how a website is ranked on Alexa over time (using a graph maybe). This could be done by hooking on the ArchiveProcessing_Day.compute event.

[mattab]

Social widget (twitter, FB)

Ideas for improvement

• Add a new widget for Social metrics: starting with Facebook and Twitter
• Admin user can edit the IDs.
• UI has a top link 'Edit settings', when clicked, shows fields for Facebook and Twitter IDs
• Social IDs are recorded in the piwik_option table (simpler than having to modify the piwik_site table). Also, there could be several Social IDs per website (several FB fan pages eg.)
• Admin can save FB or Twitter or both, clicks save. The data is saved using Piwik_SetOption function (for the given idSite).
• List of metrics to show
  • Twitter followers

  • Tweets

  • Facebook fans
  • (FB interactions from their api?)

This would be a useful tool for webmasters to build a dashboard of all their social presence. Good to have next to the SEO widget.

Maybe other networks could be integrated later. See for example postrank API (digg, reddit, delicious, etc.).

[robocoder]

aboutus.org recently added some "site metrics" and "online visibility audit" tools to their site. These might be nice as Piwik widgets and/or reports within this plugin.

[robocoder]

WordTracker is one of the better known SEO tools, but I ran across some references to tools (later developing into web services) by Michael Marshall, i.e., "link-reputation-tool", "Theme Master", "OptiRanker", and "SEM Logic".

Theme Link Reputation gives you insight into the quality of your referring sites. You enter a URL and target keywords. The tool uses Google's API ti return a survey of the top 50 backlinks (backlink URL, # inbound links, # outbound links, a "reputation score", and the link text).

ThemeMaster/OptiRanker is a SEO tool. Instead of keyword density analysis, it centers on the idea of "themes", related keywords and word combinations, using a lot of math (see "Latent Semantic Indexing"). Meanwhile, SEM Logic is more about competitive intelligence.

[robocoder]

seopanel.in is gpl v2 but might be an interesting integration play.

[anonymous-matomo-user]

See is what position in the serps (in the search engine results page) the website was when the user clicked (in addition to what the query was).

To easily have the ability to click on a referring page URL, without having to deal with truncation.

Ability to view traffic and statistics by day, for instance, show me traffic on Tuesday, or Friday, without the requirement to specify dates. So you can match days of the week, not dates.

Ability to easily visualize user paths throughout the site.

[anonymous-matomo-user]

To improve the optimization of Piwik I would recommend that we attempt to use the new logo (icon) created and attempt to get people currently using Piwik to embed this into the footer of their home page. This should help with the link popularity of Piwik.

[peterbo]

Some ideas for the SEO-Plugin:

• Request the own website for measuring Page Loading Speed (images, (javascript), etc.), warn if an image/etc. exceeds recommended size
• check robots.txt for errors / wrong restrictions
• check google "site:website.tld"-query for domain visibility (number of results) (also in evolution-graph) (very important)
• check SE ranking for 2 or 3 keywords (with evolution graph)
• check for Domain/IP popularity (how many distinct domains / IPs link on your website)
• How many other Domains are hosted on the IP of my Website (for example: http://www.bing.com/search?q=ip%3A91.121.8.168)
• Server Ping / Traceroute (does this make sense)
• apart from the backlinks from yahoo, get backlink data from the piwik archives (site referer-data), to compare both or sum them or whatever
• analyze quality of backlinks (PR, visibility (site:website.tld query), MOZ rank of a site (seomoz.org, semrush.com))

[anonymous-matomo-user]

The key to this tool would 100% be the tracking over time, much of SEO is about today, and now. This tool would become viral if it kept a log of a website's overall internet presence over time. If an SEO was to set this up whenever a new client started, they would be able to show a solid history of improvements over time.

[robocoder]

I expect we would use scheduledTasks or a cron job to fetch (and record) 3rd-party metrics daily.

[robocoder]

(In [4025]) refs #1459 - domain age parser improvement

[anonymous-matomo-user]

Some additional idea's for SEO plugin:

• Ranking data, this can be queried from various suppliers if for example API connections could be made to f.e. Raventools or others. Also ranking data in many occassions is available in the referrer URL from google (&cd=). Note, universal search results may cause different numbers to appear, apart from regular text links. universal search results are identified by sa=X parameter in referrer. Of course this data should be available over time (as @john pointed out..)
• There are numerous formulas out there to calculate 'potential extra traffic' . would be nice to indicate what that extra potential might be if traffic is X and current position = Y. This could be calculated using CTR data and to be considered raw estimate, yet useful.
• Server uptime / performance integration (downtime correlatates not only with traffic, but also search engine rankings). Can be done using various monitoring tools, most of them provide API's.
• Connection with Google Webmaster Tools data API to provide various insights on current website. For example crawl issues, Messages could also be displayed. Many marketers are daily in their analytics program but not in GWT.
• what percentage of pages listed in sitemap is receiving traffic (match with URL's receiving traffic).
• Duplicate content detection , a tool to calculate or show URL's that are most likely to be considered duplicate content.
• integrate CopyScape Premium API to detect per URL if there are copycats for your content.
• visualize Link neighborhood (unsure if possible, depends on data availability)

Q: - What SEO's use alexa rank? (outside US?)

Also a summary or action pane would be cool; algorithmically show suggestions based on 'what has been found'. Considerations might be taken if you were to be able to supply the type or niche of the websites as 'actions' might vary accordingly. f.e. Content websites differ heavily in terms of goals and KPI's opposed to ecommerce websites.

[anonymous-matomo-user]

I think most SEO's use Alexa as a very quick reference to the overall traffic/authority and value of a website or domain. I think a way to help Piwik offer basic SEO information and tools might be to partner with companies, offer some basic versions of the data set in many tools like majesticseo, seomoz or others. I can open the door to discussions. This would help Piwik get in the door with these guys, as they would then support Piwik and help drive traffic to Piwik. In return Piwik would market the paid versions of these tools. Piwik could even get a commission on any sales of these guys premium tools that are made from the basic use within Piwik.

[mattab]

martijn, we now show the search engine ranking (via &cd) in the real time Live! widget and in the Visitor Log. I like the idea of integrating with other APIs for sure, but if they only offer a premium paid for version, it is difficult for us to integrate it since it wouldn't provide value for most users (not willing to pay). John, maybe you have more ideas on how we could work on this? Do majesticseo or seomoz also provide interesting things for non paid users?

[anonymous-matomo-user]

@peterb: You say: check google "site:website.tld"-query for domain visibility

Please use site:website.tld/folder to get statistics for single websites hosted on the same server.

[anonymous-matomo-user]

what a about bot metrics? i dont exactly know if its possible but maybe someone else.

• how often are bots visiting my page
• which bots
• which sites do they visit

[robocoder]

webdevoase: see BotTracker in #2391

[mattab]

A simple improvement: cache the result of the API calls for 1 day, so that loading the widget again and again is fast. Currently, it does the Requests to all services on every widget load.

[robocoder]

In #2869, the Yahoo metrics are no longer available and have been removed from the plugin's API and Widget.

[robocoder]

(In [5683]) refs #1459 - add SEO unit test for http://piwik.org

also:

• update Google backlinks scraper (not currently used)
• remove unused Mock in Database.test.php
• added a daily CI build in case we go a few days without a commit

[mattab]

• update Google backlinks scraper (not currently used)

Could we use google BL scraper or is the number really meaningless? maybe Bing?

[anonymous-matomo-user]

Possibly what you can do is to use something like www.opensiteexplorer.org/ or www.majesticseo.com for backlink information. If you interested I can possibly work out a deal for use of their software?

[mattab]

john, these look good but opensiteexplorer is limited to 5 queries per day. Maybe we could cache the results and call once a day (and rotate if more than 5 URLs to call).

majesticseo seems to not have daily limits to just grab the number of backlinks which is nice? is it true?

Also, what about data quality? Maybe we could pick one or call both? thoughts?

[anonymous-matomo-user]

Well I know both of these companies, can may be able to sort out a deal with them if you want me to contact them and see if we could sort out a deal?

I think they both have API options.

Just let me know.

[mattab]

Thanks for proposing!! :)

I don't know what kind of deals we could sort out though because if we get access to a special API token allowing more requests per day, this token would be visible to everyone (if bundled in Piwik source code) so could be reused by non piwik users.

it would be interesting to ask them their thoughts, for sure.

Let us know if you had another idea or if you manage to contact them! cheers

[mattab]

See cutroni.com/blog/2012/08/13/seo-customizations-for-google-analytics/

Decreasing priority as scope is too broad.

[anonymous-matomo-user]

Hi. I am one of the directors of MajesticSEO and we are long time users of Piwik on our site as well.

We would like to help to give Majestic data directly into Piwik and will try to work with Piwik to make this happen.

majesticseo seems to not have daily limits to just grab the number of backlinks which is nice? is it true?<<

That's not EXACTLY true - but near as makes no difference. What I think would be great here is two or three three ways we can give FREE information to Piwik users and then more ways in which users taht have accounts with Majestic can get a closer integration.

Free ideas:

Yes - we could easily give PIWIK users link counts for any URL. In addition, we can give Flow Metric scores (values between 0-100) showing the influence of a URL. We can also show how many links to are URL are from .EDUs, how many are nofollowed, etc etc.

In addition, Piwik could add value to this data by maintaining in particular the Flow Metrics scores and the link counts over time. OUr numbers update every day, but we obviously cannot keep complete historical data of all 4 trillion URLs in our system. We do have backlink history charts (which could also be given for free within Piwik) but if PIWIK maintained Flow Metrics history scores for the site's URLS, this would show whether a page is increasing or decreasing in influence over time.

Premium ideas:

If a user wanted, then the actual link LISTS could be brought back by users connecting their PIWIK accounts to their MajesticSEO accounts. This would obviously cost the user to have an account on Majestic, but they would only need an entry level subscription. This uses our "OpenApps" API and from there, we would be able to show users a list of links won and lost on a day-by-day basis, for example... not necessarily just for their own site, although as Piwik is predominantly an analytics package and not a competitor analysis package, starting with your own sites seems to make sense.

If this all sounds of interest, we'll try and work with PIWIK to get it done.

Dixon Jones.

[mattab]

@MajesticSEO Thanks for your message. Let's discuss if and how we could integrate the reports in the default SEO widget http://dev.piwik.org/trac/raw-attachment/ticket/3525/piwik-seo-plugin-update.png

[mattab]

Created ticket in: #3734 for majestic SEO integration

SEO Panel at: seopanel.in looks interesting as well!

[mattab]

Great list of seo reports: http://www.seobook.com/setting-actionable-seo-dashboards-new-google-analytics

in GA: http://www.blueriotlabs.com/wp-content/uploads/2012/06/google-analytics-seo-monitoring-dashboard.png

[mattab]

After brainstorming and discussion we have narrowed down the vision for Piwik 1.X SEO reporting following @john's awesome advice. See the ticket description for the specs. It will be awesome!

[mattab]

SEO: See cutroni.com/blog/2012/08/13/seo-customizations-for-google-analytics/

myseotool.com/