search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.63k stars 2.62k forks source link

Java Tracking Client V1 - PiwikTracker.java #2172

Closed anonymous-matomo-user closed 12 years ago

anonymous-matomo-user commented 13 years ago

hi folks,

I transformed PiwikTracker.php to Piwiktracker.java and wanted to share the code for further improvement/fixes. There is still stuff missing but it works for me so far.

To track a (JSP-)page I use the following code:

<% PiwikTracker pt = new PiwikTracker(1, "http://localhost/piwik", request); ResponseData rd = pt.doTrackPageView("Tracker Test"); if(rd.getCookies().size() > 0) response.addCookie(rd.getCookies().get(rd.getCookies().size()-1)); %>

I am still a newbie to Piwik so I might have gotten something wrong...

cheers Keywords: PiwikTracker Java

anonymous-matomo-user commented 13 years ago

Attachment: Cookie Handling Helper Class ResponseData.java

anonymous-matomo-user commented 13 years ago

Attachment: The PiwikTracker.php equivalent in Java PiwikTracker.java

anonymous-matomo-user commented 13 years ago

Attachment: again latest version. NPE fix + protected fields and methods. please test! piwik.2.zip

anonymous-matomo-user commented 13 years ago

Attachment: the latest and greatest - all others can be ignored/deleted piwik.jar

anonymous-matomo-user commented 13 years ago

Attachment: Mavenproject of the java implementation JavaPiwikTracker.tar.gz

anonymous-matomo-user commented 12 years ago

Attachment: New version of PIWIK JAVA plugin. piwik_java_plugin_1.0.2.zip

robocoder commented 13 years ago

Looks good (except for some formatting inconsistencies). Can we use org.piwik and redistribute under a BSD license?

mattab commented 13 years ago

Exactt, thanks for your submission. This is the first third party language we will support after PHP, good news!

Code review:

The code looks very close to the PHP version which is good as well. I noticed json_encode is not implemented? this is important since Custom Variables are sent json encoded to the tracker.

You can safely remove all XDEBUG related code.

Also, maybe you haven't used the latest 1.2 PiwikTracker to do the conversion. There are a few fixes and now the setIp and set date of visit work with token_auth, so they can be passed in getRequest() method. Implementation has slightly changed (ie. was finalized) in 1.2 around Custom Variables (eg. the parameter is called cvar not _cvar)

for readability, you can remove the PHP functions not implemented

Thanks for this, Java users from the piwik community will definitely appreciate. We can also let you know if there are further updates to PiwikTracker.php if you'd like to maintain

anonymous-matomo-user commented 13 years ago

hi guys, thx for your comments!

@vipsoft: I removed the license part because it contained too many URLs. These kept me from uploading because the file was recognized as spam. Maybe someone can change the spam recognition setting so uploads with a handfull of URLs work... Please change the licensing and package as you prefer!

@matt: I will investigate the changes and upload a new version in the next couple of days. I would love to get informed about updates and will maintain the file as far as possible.

robocoder commented 13 years ago

You can upload a .zip or .jar to get past the attachment filter.

anonymous-matomo-user commented 13 years ago

here my refined calling code:

<% String pageTitle = "Tracker Test"; PiwikTracker pt = new PiwikTracker(1, "http://localhost/piwik", request); ResponseData rd = pt.doTrackPageView(pageTitle); if(rd.getCookies() != null && rd.getCookies().size() > 0) response.addCookie(rd.getCookies().get(rd.getCookies().size()-1)); %> <title><%= pageTitle %></title>

mattab commented 13 years ago

exactt, thanks for the updates, it looks really cool.

Have you tested all the functions? Are you confident it all works?

If so, and because you are keen to maintain the lib, I propose that we commit this class to the SVN and link it from the Tracking API page (like we do for the PHP version). Also could you please post a simple forum post and then we can ask for testers on twitter and facebook, I'm sure there are many java fans using Piwik :)

Let me know your thoughts, great work anyway! Now on to Python, ruby :)

anonymous-matomo-user commented 13 years ago

I just posted in the forum: http://forum.piwik.org/read.php?2,73439

After some success reports we can take the next steps...

mattab commented 13 years ago

exactt,

We posted the link on twitter as well so hopefully Java people will find it!

Maybe it is easier if you edit the ticket description and put the README there (full sample code to track a page view, other interesting notes).

anonymous-matomo-user commented 13 years ago

Hi folks, yesterday i have taken a look on the sources, because i need easy access to the trcker api from apache wicket, and it's a good start, but IMHO not yet production ready. J'am right now on doing some changes and will post it on my github page in the next few hours.

The changes involve:

Background: I just started to know about piwik (from the heise.de newsticker) but have 10 years of java web development skills (jsp, wicket, gwt, vaadin, grails).

So far my few points. But without the start from exactt i would right now fight with the js api :) so thanks for the commit!

mattab commented 13 years ago

mheuser, this sounds great. Please post a comment here when you have done work in your github, and when you are done please post the Zip here.

Ideally, we will include the Java API in the Piwik SVN and then give you access to SVN so you can maintain it (even though there shouldn't be much work, we might add new features in the API later).

anonymous-matomo-user commented 13 years ago

It's been a while, just want to update this ticket. I had to do some more important tasks since my last post, but i will post the code till Friday.

anonymous-matomo-user commented 13 years ago

hi, nice to read this ticket.

I just started working with the provided code. I'm really interested in the changed code, cause I use maven dependency management as well and don't want to make the changes you have listed before by myself. ;)

mattab commented 13 years ago

mheuser, exactt, how is the PiwikTracker.java going? Can we submit it on SVN and link it from the product and documentation as the official Java client?

anonymous-matomo-user commented 13 years ago

Thanks for the effort! They have been a very good starting point. Unfortunately I do not find the API well documented, i.e. which parameters are used with which data type so I was very happy to find this class.

We have also used the class in a project now but had to adjust them to our needs. The PiwikTracker.java attached here requires JRE 6 for cookie handling which is not feasible for all project setups - and has not been for ours. Furthermore it uses a synchronous approach which might block further processing if the PIWIK server is unavailable.

We have gone for an asynchronous approach and a handling w/o cookies (we are identifying unique visitors differently). I would provide a diff, but we are using PiwikTracker as POJO without any logic in it except building the URL.

We have found some small issues and here are changes that need to be done to URL building in PiwikTracker#getRequest(int) regardless of the project setup:

Parameter _refts requires epoch time in seconds, a Date was given. Java furthermore counts epoch times in milliseconds.

"&_refts=" + ( forcedDatetime != null ? forcedDatetime.getTime() : System.currentTimeMillis()) / 1000

The parameter forcedDateTime needs to be encoded in a way PHP's strtotime() can handle. We are using the following formatter and had no issues with it:

private static SimpleDateFormat SDF = new SimpleDateFormat("yyyyMMdd HH:mm:ssZ");
[..]
(forcedDatetime != null ? "&cdt=" + urlencode(SDF.format(forcedDatetime)) : "")

As the #empty(int) is a wrong method (returns true if value != 0), the resolution was not transmitted correctly. We have changed the line to:

(width > 0 && height > 0 ? "&res=" + width + "x" + height : "")

The "biggest" change affected custom variables. It fails with an IndexOutOfBoundsException because the position is given although there are no elements in the List. So we are "adding" the variables to the list rather than "putting" them in. I have no clue if the order of custom variables is that important. However as we are always calling the method in the same order, the order of the variables remain the same anyway. The second change affected the json_encode which should JSONObject instead JSONArray:

private List<List<Object>> visitorCustomVar;

public void setCustomVariable(Object name, Object value)
{
    if (visitorCustomVar == null) {
        visitorCustomVar = new ArrayList<List<Object>>();
    }
    List<Object> list = new ArrayList<Object>();
    list.add(name);
    list.add(value);
    visitorCustomVar.add(list);
}

private String json_encode(List<List<Object>> visitorCustomVar) throws JSONException
{
    JSONObject json = new JSONObject();
    int pos = 1;
    for (Iterator<List<Object>> iterator = visitorCustomVar.iterator(); iterator.hasNext();)
    {
        List<Object> objects = iterator.next();
        json.put(Integer.toString(pos++), objects);
    }
    return json.toString();
}
anonymous-matomo-user commented 13 years ago

Thx for the valuable feedback. I adopted the changes except the last one.

Could one of the PHP developers shed some light on the custom variable thingy please!? Where can we find find the documentation regarding ordering etc.?

mattab commented 13 years ago

I hear your complain about lack of documentation :) I think the best way to start is to look at the PHP Tracker, the code should be fairly easy to understand: https://github.com/piwik/piwik/blob/master/libs/PiwikTracker/PiwikTracker.php

Regarding Custom Variables, here are the spec: {"1":NAME 1","VAR VALUE 1","4":NAME 4","VAR VALUE 4"}

The order is not important but the value array must be indexed by the custom variable slot.

Hope it helps!

mattab commented 13 years ago

Few more details:

anonymous-matomo-user commented 13 years ago

just uploaded another version.

@knarf: what do you think? looking forward to your feedback!

@all: please test and report back! thx!

mattab commented 13 years ago

(In [4350]) PiwikTracker

anonymous-matomo-user commented 13 years ago

I added my maven project of the implementation to the ticket. I have done some java 1.5 changes and rewrite the hole java class in a more java stylish way. Please look into the code and give me feedback.

julienmoumne commented 13 years ago

In pom.xml,

<url>http://maven.apache.org</url>

should be

<url>http://piwik.org</url>

Is there a reason why org.json classes are included as project source files and not as a maven dependency ?

mattab commented 13 years ago

(In [4378]) Fixes #2168

Refs #2172

Refs #2222 Accurate Paypal tracking (or any other third party "after the fact" Goal conversion): implementation is now done, we must write some kind of guide and test

mattab commented 13 years ago

As well as the previous comment, the Java client must be updated with the new functions:

All this code is pretty trivial hopefully you can reach an agreement soon for the Piwik Java client and we can start advertising it and commit it to SVN!

Thanks Java devs

anonymous-matomo-user commented 13 years ago

Replying to JulienM:

In pom.xml,

<url>http://maven.apache.org</url>

should be

<url>http://piwik.org</url>

Is there a reason why org.json classes are included as project source files and not as a maven dependency ?

The pom.xml change is trivial. The problem of the dependency is that I don't find a proper one.

The getVisitorId function is also trivial to add.

Is there a way to get access to svn or the get a path where I can upload my source? The attachments will get very confusing if we do the source providing in that way.

julienmoumne commented 13 years ago

have you tried from mvnrepository.com ?

<dependency>
    <groupId>org.json</groupId>
    <artifactId>json</artifactId>
    <version>20090211</version>
</dependency>

we really need a decentralized VCS!

mattab commented 13 years ago

It is no problem to get access to Piwik SVN to work on the PiwikTracker.java please contact us at hello att piwik.org with your request and we will add you!

mattab commented 13 years ago

(In [4411]) Fixes #2299, refs #2172 (in case it is easier for you, now the function will accept a timestamp as well)

anonymous-matomo-user commented 13 years ago

Replying to matt:

It is no problem to get access to Piwik SVN to work on the PiwikTracker.java please contact us at hello att piwik.org with your request and we will add you! I have send a mail to the given address but got now answer so far.

mattab commented 13 years ago

mfochler, we found your email in spam folder sorry. You should now have access, per email.

anonymous-matomo-user commented 13 years ago

uploaded all Data with r4563 Sadly I have committed it with the wrong ticket number.

mattab commented 13 years ago

Thanks Martin! To all Java developers in here, would you mind checking out Martin's code from https://github.com/piwik/piwik/blob/master/libs/PiwikTracker/java/ and give your feedback here?

Also, in order to add the Java lib in the Analytics tracking documentation, would you be able to prepare a 3-4 lines description of the tool, and add the code documentation "How to use it" in the Class header comment?

Thank you for the great work!

anonymous-matomo-user commented 13 years ago

I will try to create more documentation to the classes this week, but can not promise it ;)

anonymous-matomo-user commented 13 years ago

Hi,

I'm using your last code (from SVN), but the JSP code is not updated.

Can you add it please.

Thanks.

anonymous-matomo-user commented 13 years ago

Hello,

I tried to adapt the old JSP code to this one :

<% String pageTitle = "Tracker Test"; SimplePiwikTracker pt = new SimplePiwikTracker(1, "http://localhost/piwik", request); ResponseData rd = pt.sendRequest(pt.getPageTrackURL(pageTitle)); if(rd.getCookies() != null && rd.getCookies().size() > 0) response.addCookie(rd.getCookies().get(rd.getCookies().size()-1)); %>

But i have an exception in SimpleWikiTracker.java at this ligne :

     this.setUrlReferer(request.getHeader("Referer"));

wich is in readRequestInfos fonction. It triggers an exception because i haven't "Referer" in my request (urlReferer=null)..

So what's the point? and what's the solution ?

Thank you :)

anonymous-matomo-user commented 13 years ago

Hi,

I found one error in SimplePiwikTracker.java , you must replace "user-agent" by "User-Agent", in all instances, to make it work ;)

mattab commented 13 years ago

mfochler, thanks for your update in trunk!! Excellent class for Java developers.

In Piwik 1.5 (currently in RC Testing), We have added Tracker API support for Ecommerce analytics tracking.

The Tracking API is now stable. The new functions are:

AS well as these new functions, the Tracking API functions added in 1.4 haven't been ported in the Java client:

Finally, the following functions have been updated ( a new parameter added, for Custom Variables for scope "page" support)

The code for these functions is fairly straightforward (very small code) and don't introduce any new concept so it is mostly reusing existing code. The PHP client is at: https://github.com/piwik/piwik/blob/master/libs/PiwikTracker/PiwikTracker.php and the auto generated doc at: http://qa.piwik.org:8080/phpdocs/PiwikTracker/PiwikTracker.html

Thank you for your useful help so far! Hoping that someone can submit a patch which mfochler can then commit to trunk :-)

anonymous-matomo-user commented 13 years ago

(In [4848]) Workaround for the NullPointer in urlReferer, will set the referer to null. This has to be testet for other consequences. Changes both appearances of user-agent to the capital letter writing. refs #2172

anonymous-matomo-user commented 13 years ago

I will create a patch in near future.

In the new code of the piwikTracker the param "_ref" is used for the attributionInfo element 4. In the previous version the param "ref" was used by the same param as "urlref". I implemented it in this way. Is it right that the old param "ref" was never used? I'm really confused cause the attributionInfo element 3 is now the param "_refts" with was used by the 'forcedDatetime'. Please enlighten me so I can make the changes to be compatible with the older versions of piwik.

The other changes can be made easily.

I hope I can make the changes at the weekend.

mattab commented 13 years ago

Summary:

(!empty($this->forcedDatetime) ? '&cdt=' . urlencode($this->forcedDatetime) : '') .


There is no need to make it compatible with older version of Piwik, the version of PiwikTracker.java is only compatible with the latest version of Piwik only. Most Piwik users upgrade to the latest version all the time, so it is not a problem and would be too complicated to do (also hard to test etc.)

So, maybe just look at the PiwikTracker.php and follow the implementation in this class. we test a lot this class, so if the Java Tracker is using the same code more or less it should be good :)

Another change I didn't mention above:
* setUrlReferer is deprecated, the new function is setUrlReferrer - it is the same just fixing the typo

Thanks Martin for your help, let me know if you have more question!
robocoder commented 13 years ago

(In [5079]) refs #2172 - add header

mattab commented 13 years ago

Hello Java developers, any update on PiwikTracker.java? Thanks guys!

anonymous-matomo-user commented 12 years ago

Hi! I have a problem with sending "custom variables" via "org.piwik.SimplePiwikTracker.java".

If we try to send more than one custom variable, only one custom variable of them will be shown on Piwik application.

The problem is that SimplePiwikTracker seems not to send custom variables with startindex 0, and this is wrong.

From the document of JavaScript tracking([http://piwik.org/docs/javascript-tracking/]), custom variables must be sent with "1" as startindex.

So following format should be correct : {"1":Firefox 8.0", "2":Markup","html"}

But SimplePiwikTracker.java generates a different format : Firefox 8.0",Markup","html"]

(Full example URL of Piwik Tracking -> http://piwik.local/piwik.php?idsite=111&rec=1&apiv=1&url=http%3A%2F%2Flocalhost%3A8080%2Finm%2Fadesso%2F&_id=BD2F6502EA8ABE3D0FF7F9DC749A3114&cip=127.0.0.1&token_auth=34f6ee3769a9e5f6239573b6637b6336&pdf=true=false&res=400x700&cookie=false&_cvar=%5B%5B%22Modelname%22%2C%22BROWSER+Firefox+8.0%22%5D%2C%5B%22Prefered+Markup%22%2C%22html%22%5D%5D&rand=0.20301514584441116&action_name=http%3A%2F%2Flocalhost%3A8080%2Finm%2Fadesso%2F )

So I guess that Piwik application doesn't parse all custom variables correctly.

I tried to fix this issue.

Class : org.piwik.SimplePiwikTracker.java Method : public final String getGeneralQuery() (Line:421) Line 461 - 467

(Now)

                       final JSONArray json = new JSONArray();
            for (final Map.Entry<String, String> entry : this.customVar.entrySet()) {
                final List<String> list = new ArrayList<String>();
                list.add(entry.getKey());
                list.add(entry.getValue());
                json.put(list);
            }
            withCustomVar = this.addParameter(withCustomData, "_cvar", json.toString());

(Should be)


             final JSONObject json = new JSONObject();
            int i=1;
            for (final Map.Entry<String, String> entry : this.customVar.entrySet()) {

                 final JSONArray jsonArray = new JSONArray();
                 jsonArray.put(entry.getKey());
                 jsonArray.put(entry.getValue());

                 try 
                 {
                    json.put(Integer.toString(i), jsonArray);
                } catch (JSONException e) {
                    e.printStackTrace();
                }
                 i++;

            }
            withCustomVar = this.addParameter(withCustomData, "_cvar", json.toString());

Can you apply the fix, and release the update version for us ?

Thanks a lot for help! Rie

anonymous-matomo-user commented 12 years ago

INFO :

We are using the JAVA tracker API from : http://dev.piwik.org/trac/browser/tags/1.6/libs/PiwikTracker/java

anonymous-matomo-user commented 12 years ago

I improved the existing official PIWIK JAVA connector: