Build a demo application that tracks all possible data for manually tracking testing

[tsteur]

I'm currently working on #7056 and looking through the code trying to understand what that bug causes. After a while I noticed it is not that easy and I actually need to reproduce this issue. As this problem is related to Ecommerce and UserId (Login/Logout) it is not as easy to test as one has to call several tracking calls manually and one has to be careful to set correct data etc.

Usually best way would be trying to create an integration test that shows the problem so we also make sure for the future that we won't have it again. But sometimes it is hard to test this way correctly and it takes much more time when someone just wants to have a look whether there is a bug or not etc. Sometimes one has to write the integration test afterwards as one sometimes has no clue how the reproduce that problem... Also sometimes the problem is not in PHP but in JS, eg Content-tracking has a lot of DOM logic etc...

While we can already track Piwik itself (config enable_measure_piwik_usage_in_idsite) it would be nice to have another separate website that anyone can use to reproduce such problems and to generate visits if needed etc. It could be just HTML pages if possible or a PHP based dummy application with a fake shop, with login/logout (userId), content-tracking, etc. Kinda like a "Kitchen Sink" demo.

For example for content-tracking I adjusted the Marketplace in my local instance to track the plugins using content-tracking. Next time I have to test this again I need to do this again and again...

Maybe this application could be even used for tracker tests somehow but probably not.

[mattab]

For example for content-tracking I adjusted the Marketplace in my local instance to track the plugins using content-tracking. Next time I have to test this again I need to do this again and again...

That's a nice idea! could we push this to Production marketplace as it would give us some interesting content tracking data?

[tsteur]

We could, maybe a new issue for this?

[mattab]

(done.)

It's a good idea to create a fake app that tracks all kinds of data, events, etc. Although somehow I feel it would be nice to do this in Piwik app itselfe since it's easy enough to click 10-20 times to generate data. We could generate events, content tracking, user id, most features, and on each page view to Piwik it would track data for most features. Of course you mention "Ecommerce" which would not be easy to fake in Piwik but if we wanted we could send fake ecommerce transactions eg. when the user visits Piwik ecommerce reports. I'd prefer to work on measuring Piwik because it's so important and this issue would be a good excuse to get started on this :-)

[tsteur]

I would rather want to see a separate app since it will allow you easier to test such stuff. Eg you see an issue and you need to exactly reproduce the steps... This will be hard with the Piwik UI since you won't be able to do it and you wouldn't be transparent when Piwik tracks what...

As a temporary solution it would be maybe ok but it wouldn't actually really solve my problem (debugging, adding new features to tracker etc)

[tsteur]

Tracking some more features in Piwik itself apart from this issue sounds good though

[tsteur]

Ideally, at some point, we would track all possible features in such an application, and also use it for automated real live tests in various browsers. This would be really valuable.

This application would be good for automated tests, but also for manual tests and for learning how to integrate tracking into a website. We would have info buttons / sidebar that show how to integrate a just used feature, how to do the same with JavaScript tracker, iOS tracker, Android tracker, via HTTP Api etc. Even though it sounds small I believe there is huge value.

Ideally we would also host it at some point on a Piwik domain and people could test it and see the requests on demo.piwik.org

If someone wants to work on such feature, please ping me. It's really interesting thing to build from scratch. Could use some mobile frameworks for this like ionic 2, angularjs2, or other fancy stuff :)

Repository for this will be https://github.com/piwik/tracking-kitchen-sink

Example for another KitchenSink: http://dev.sencha.com/extjs/5.0.0/examples/kitchensink/

[hpvd]

sounds pretty interesting and valuable!

[tsteur]

@diosmosis @sgiehl @flamisz

moving the thoughts from slack into this issue now. We could have a folder say tests/demo (can easily rename any time) where we have a lose collection of useful individual HTML pages (could partially also be PHP) that are needed/useful to reproduce or test certain features.

For example:

• tests/demo/optout-with-tracker-on-same-page.html which would embed the opt out screen and have the tracking code set up to the same domain (as this would try to additionally do the opt out using iframe messages)
• tests/demo/optout-without-tracker-on-same-page.html which would embed the opt out screen but not have the tracking code integrated
• tests/demo/tracking/ecommerce.html which could have a few buttons to simulate "add to cart", "cart update", "order"
• etc.

All tracking related test pages we would put in a subfolder tests/demos/tracking. I think we would use these pages mostly for tracking related things but it might be also useful for other things (eg optout iframe, we could also have one for tracking consent and one for cookie consent and custom opt out).

I think there wouldn't be much of restrictions on what to put there or not. The idea would be whenever we need to test/reproduce something, we'd check if an example is there and if not, then we think about whether it may be useful to add one so others can reuse it in the future if we find ourselves building such a page anyway.

As idSite we would always use 1. I think it wouldn't work much for widgets since they would mostly need a token I suppose.

Regarding keeping them up to date: Since we mostly keep BC I suppose they shouldn't regress too much and if they do become outdated, that may be fine as I suppose we'd notice this when trying to use one of those examples.

I suppose while we build new features or fix bugs (mostly tracking related) we build such pages often for ourselves (I've built many of them over the years but often find myself changing the same html pages and throwing away what I had created previously) and then we would instead share these pages with the team. There would be no obligation to add such a page as part of building a feature although it would be great to do in some cases as it would be mostly needed anyway for the author of a new feature as well as the tester.

Any thoughts/ideas? Ideally we can also just start doing this and see how it goes over time and tweak things.

[tsteur]

We just discussed the idea that the support team might need this as well so we're thinking of a new public repository where we create PHP files instead of HTML files and have eg idSite, hostname and token etc configurable.

[tsteur]

FYI I have created https://github.com/matomo-org/test-examples for now. The initial structure and how everything works is just an initial start and we can change anything.

[tsteur]

FYI created https://developer.matomo.org/guides/reproducing-issues as a start for a new doc and also added https://developer.matomo.org/guides/contributing-to-piwik-core#creating-a-pull-requestsharing-test-example-pages for now.

@diosmosis @sgiehl @flamisz @mattab any other thoughts on this or can we close this issue maybe? We could add another check to the pull request template just to remind us to think about it but it wouldn't be a requirement as most of the times this would not be possible.

[mattab]

This looks great! :sunglasses:

it would be helpful if the test examples also add the basic tracking (pageviews, events, content tracking, outlink, download tracking, goal id 1, custom dimension 1) but it can be added later of course and it'd be fine for me to close this