search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.85k stars 2.64k forks source link

JS Tracker: Add possibility to turn cookies back on, enableCookies #13056

Closed WouterFlorijn closed 4 years ago

WouterFlorijn commented 6 years ago

I'm working on a website that allows visitors to toggle cookies on or off. Currently, with Matomo I can turn them off using

window._paq.push(['disableCookies'])

I searched for a way to turn them back on, and tried

window._paq.push(['enableCookies'])

but that did not work.

Is there a way to turn cookies back on? If not, it would be great if it could be added.

Note from tsteur: Also need to look at this when working on it: https://github.com/matomo-org/matomo/issues/13056#issuecomment-646278244

sgiehl commented 6 years ago

If the page is reloaded and disableCookies isn't pushed the cookies should be automatically used "again". But for singe page apps it would make sense to have a method to turn the cookies back on.

WouterFlorijn commented 6 years ago

@sgiehl That would be great. Not only for SPAs, but any website where the user can change their cookie consent settings on the fly, which is relevant with the GDPR.

bjoe2k4 commented 5 years ago

Any Update on this? Interested in this as well!

martin-braun commented 4 years ago

The biggest issue on this is not that we cannot re-enable cookies. As a bypass, we can simply reload and prevent pushing 'disableCookies'. However, if we use a custom opt-out form, it will not work after disabling cookies, because the opt-out form requires cookies to be enabled.

It would be great to be able to allow cookies by JS without reload and make it work, so the custom opt-out solution can work. In it's current design, it's impossible to prevent conventional Matomo cookies and still getting the Opt-Out cookie.

It's been 1 1/2 years, this needs to be addressed, to be honest.

ctholho commented 4 years ago

I'm also looking for this functionality. Is there any other way to reenable cookies without a reload?

I don't understand how this flies with GDPR and cookie consent regulation in the EU and other territories. I don't want to disable tracking entirely. Simply cookies on/off is enough, I would have thought.

futureweb commented 4 years ago

Would also need this Feature, we use Matomo's disableCookies Feature + Klaro! as Consent Manager ... of course if User gives Consent I want to use Cookies again (1-Pager, ...), so it would be benefical if one could re-activate Cookies by JS

utrautmann commented 4 years ago

If the page is reloaded and disableCookies isn't pushed the cookies should be automatically used "again".

That works technically. But in this case, however, a new visitor ID is generated in the cookie. I.e. I now have 2 visitors (one for fingerprinting-scenario, another one for the cookie-scenario) in Matomo, although in reality it is only 1 visitor. That is not what you want.

tsteur commented 4 years ago

@utrautmann had a quick look in the code. That's something we'll need to fix as I assume that's because we include whether the user has cookies or not in the fingerprint: https://github.com/matomo-org/matomo/blob/3.13.6/core/Tracker/Settings.php#L145

It should be an easy fix for us to remove this from the fingerprint when we work on it or fill this setting depending on navigator.cookiesEnabled setting.

tsteur commented 4 years ago

Thanks for mentioning it and finding this 👍

utrautmann commented 4 years ago

@utrautmann had a quick look in the code. That's something we'll need to fix as I assume that's because we include whether the user has cookies or not in the fingerprint: https://github.com/matomo-org/matomo/blob/3.13.6/core/Tracker/Settings.php#L145

It should be an easy fix for us to remove this from the fingerprint when we work on it or fill this setting depending on navigator.cookiesEnabled setting.

@tsteur Is there an ticket where I can follow?

tsteur commented 4 years ago

We'll fix this as part of this issue.

utrautmann commented 4 years ago

@tsteur Oh it's late. Many of my customers would like to act in Germany after the BGH judgment and change the tracking. They want to go with fingerprinting as the default and setting cookies after the consent. Is there a way to still release that in the 3.13.x product line? The alternative would be to patch the PHP file for yourself in every installation!?

tsteur commented 4 years ago

@utrautmann I was more thinking something like this instead of ignoring the cookie server side: https://github.com/matomo-org/matomo/pull/16101/files

Are you familiar with patching files on a server? If so you could try to update your piwik.js, matomo.js, and js/piwik.min.js with this content and then give it a test: https://raw.githubusercontent.com/matomo-org/matomo/8ad6c25f3d1bb651444aff2ddf492c041dcb7306/matomo.js

utrautmann commented 4 years ago

@tsteur Thank you. I patched the files on my server. Now I have tested with different browsers. It works fine with Firefox, Chrome, Chrome-Mobile, Edge, Opera. But not with IE11. Two different visitor IDs are still created with IE11.

tsteur commented 4 years ago

Thanks for confirming. IE11 is because of https://github.com/matomo-org/matomo/issues/11507

It seems we can't easily check if cookies are enabled without actually setting a cookie there or causing a warning. Unless we could test if the site is shown in a modal dialog or not but not sure how/if that's possible. If you have any thoughts let me know

tsteur commented 4 years ago

Hi there, anyone keen on giving the enableCookies method a try? https://github.com/matomo-org/matomo/pull/16113

If you are familiar with patching files on a server then you could try to update your piwik.js, matomo.js, and js/piwik.min.js with the content from the following URL and then give it a try: https://raw.githubusercontent.com/matomo-org/matomo/567ca3070a5ea62660d347b53d3704258af4f59c/matomo.js

I did some basic testing and it works for me.

Similar to disableCookies it includes a new method _paq.push(['enableCookies']);.

@utrautmann @futureweb @ctholho @martin-braun @WouterFlorijn @bjoe2k4

utrautmann commented 4 years ago

Hi there, anyone keen on giving the enableCookies method a try?

I tested it with enableCookies. It works fine in single page websites. Thanks alot.

  1. EnableCookies creates a cookie _pk_id ... and
  2. disableCookies removes the Matomo cookies.
  3. When I trigger an event after EnableCookies I get the cookie _pk_ses... .
  4. But I never got a _pk_ref... Cookie for attribution. If I call up a page (single page or normal page) with parameter pk_campaign and do not have the user's consent at the moment and then get it and call enableCookies, no cookie _pk_ref is set. Is this possible so that I can measure the attribution in goals?

To IE11: In a single page all actions of a visitor - started with disabled cookies and go on with enabled cookies - are always assigned to the same visitor as long as I don't reload the single page website. Then I have the IE11 problem from above again, that one and the same visitor counts as 2 visitors depending without and with cookies.

tsteur commented 4 years ago

The IE 11 is kind of expected and can't do really much about it except disabling the cookie as part of the fingerprint on the server itself which I now did in this PR: https://github.com/matomo-org/matomo/pull/16113/files#diff-02edccc593e3badd1e9fd8d278fd1d35R102-R147

This will then make it work.

utrautmann commented 4 years ago

@tsteur

This will then make it work.

Thank you very much. I've tested it successful with IE11.

Could you take a look to my question number 4 please?

But I never got a _pk_ref... Cookie for attribution. If I call up a page (single page or normal page) with parameter pk_campaign and do not have the user's consent at the moment and then get it and call enableCookies, no cookie _pk_ref is set. Is this possible so that I can measure the attribution in goals?

tsteur commented 4 years ago

Thanks @utrautmann I've made a minor adjustment but haven't tested it yet. I reckon it should be set now. Seems so far Matomo sets attribution cookie only when a tracking request is done so when you enable cookies and there's no tracking request it wouldn't be set.

tsteur commented 4 years ago

FYI @utrautmann did some more testing and the referrer cookie is now set.

However, there's one thing that's not quite clear...

I was going to send a tracking request to update the existing visit with the newly created visitorId when enabling cookies. Noticed that Matomo does currently not support updating the visitorId of an existing visit.

What this means is the next time this visitor visits your website, it will be counted as a new unique visitor and only then will be tracked with the new visitorId. So flow is like this

So it basically once create a new unique visitor and in total Matomo counts 2 unique visitors.

I was thinking about creating some functionality to update the visitorId when the user gives consent and we enables cookies. This way it would have counted only one unique visitor. However, I'm not sure if we can/should do this since we maybe didn't have consent to attribute all previous page views before the visitor gave consent to this unique visitor. I'm meaning the page views that were done within the visit in which the user enables the cookies and gives consent:

I don't think we can/should do this. I guess if this behaviour was wanted, someone should maybe force a new visit so the visits before giving consent and after giving consent are clearly separated.

I can see some would maybe want to create a new visit while some others maybe wouldn't want to.

Anyone any thoughts on this?

It's quite complicated so I hope everyone can follow my logic.

utrautmann commented 4 years ago

Now user gives consent and you enable cookies. Matomo generates a random visitorId in JS because it can't know which visitorId was created server side However, the current visit still has a different visitorId because we can't overwrite it Visitor leaves your website Visitor visits your website again say one hour or a day later. This visit will be recorded under the new idVisitor.

Why is that? I thought after the enabling cookies I have a cookie with a visitor ID that is valid for 13 months and is recognized again? Or will I be after enabling cookies further identified via backend fingerprinting in the visit?

I was thinking about creating some functionality to update the visitorId when the user gives consent and we enables cookies.

I understand the problem you are describing. I thought that consent is only required for tracking with cookies. However, but it's no consent for cookie-less tracking required. Or do you see a consent for cookie-less tracking as necessary?

In my opinion, it is not critical from a data protection point of view if the page views from the current visit with cookie-less tracking are attributed to the same visitor after the consent to tracking with cookies.

I can see some would maybe want to create a new visit while some others maybe wouldn't want to.

In that case, an option would be required, depending on the customer's requirements, right?

tsteur commented 4 years ago

Why is that? I thought after the enabling cookies I have a cookie with a visitor ID that is valid for 13 months and is recognized again? Or will I be after enabling cookies further identified via backend fingerprinting in the visit?

That's right the cookie will be set immediately for 13 months and is valid for that long. It's just that for the already started visit Matomo knows this visit under a different visitorId which is only known in the Matomo backend but can't be known in the Matomo JavaScript tracker. And when Matomo receives the new visitorId from the JS tracker it is currently not updating the visitorId of the already started visit. It would become only active on the next visit.

Or do you see a consent for cookie-less tracking as necessary?

As long as no personal data isn't tracked there shouldn't be any consent needed. See eg this FAQ.

I was just wondering because technically the previous page views (might be 1 or might be 10...) while in the current visit cookies were disabled maybe weren't meant to be put into the visitor's profile that is being created once consent is granted. Considering this would only affect the page views within the current visit it might be fine though.

utrautmann commented 4 years ago

So it basically once create a new unique visitor and in total Matomo counts 2 unique visitors.

I tested that again after a day. After my first visit (disable and then enable cookies) I am back on my test page. The visitor log says that my visit is a returning visitor. That would be good. But the visitor profile says there are no more visits for this visitor. It's a little strange. How does Matomo know that I returned? Is it from the one day old cookie? But Matomo cannot find the VisitorID from these cookies anywhere, because my first visit had a Backend-VisitorID? Is that correct?

I was just wondering because technically the previous page views (might be 1 or might be 10...) while in the current visit cookies were disabled maybe weren't meant to be put into the visitor's profile that is being created once consent is granted. Considering this would only affect the page views within the current visit it might be fine though.

Ok, now I understand your thoughts. Well, I don't think that's so important to most web analysts to see the pages in new visitors profile. They look at the sums of visitors and page views. But there could be things where it would be interesting, right? Just one thought from me: I have a customer who works with the Funnel plugin. The entry URLs could be wrong if the page on which the consent is given is missing in the profile. And could the page where i'm giving the consent be an exit page? Or am I wrong and that's not a problem with entry and exit pages?

tsteur commented 4 years ago

@utrautmann I've just tweaked the PR so the existing visit will be updated and it will be only one unique visitor in the end. So things should all work then.

utrautmann commented 4 years ago

@tsteur : Thank you very much. What are the schedules for this PR? I ask if my customer should wait for it or if the server should be patched.

tsteur commented 4 years ago

I would say in 2 weeks

tsteur commented 4 years ago

@utrautmann sorry to let you know but I renamed enableCookies to setCookieConsentGiven in this PR https://github.com/matomo-org/matomo/pull/16178 while documenting this new feature in https://github.com/matomo-org/developer-documentation/pull/359/files

This will be renamed to be consistent in how we ask for tracking consent. Asking for cookie consent works now in a very same way and is described in both PRs above.

There will now also be a new method rememberCookieConsentGiven which makes cookie consent really easy as all you need to do is always calling

_paq.push(['requireCookieConsent'])

instead of disableCookies and then once the user gives consent, you only need to once call _paq.push(['rememberCookieConsentGiven']) to let Matomo know to use cookies (also in the next visits). When user no longer wants to use cookies simply call _paq.push(['forgetCookieConsentGiven'])

tsteur commented 4 years ago

I'm hoping we can release a beta this week.