_paq.push cannot be used when also using Tag Manager unless it is used in the same Tag Manager container itself.

tsteur commented 3 years ago

refs https://github.com/matomo-org/wp-matomo/pull/421 and once this is fixed this PR would need to be reverted.

refs https://wordpress.org/support/topic/archive-warning-error-unserializing-wp_matomo_logtmpsegment/#post-14030042

This is especially an issue in Matomo for WordPress as it means the correct tracking of Ecommerce doesn't work when using the Tag Manager. It's not only a problem for Matomo for WordPress though but in general it means you basically can't use separately _paq.push when you also use the Tag Manager.

What happens:

The page loads the container async eg in <head>
The body contains eg a _paq.push(['setEcommerceView'...]) (or trackEcommerceOrder or trackEcommerceCartUpdate etc) is called
The container is loaded
- At the beginning of the container the Matomo tracking code is initaliazed
- It notices a _paq.push and creates a new tracker but there is no idSite and no tracking URL so it sets the ecommerce view or tracks ecommerce cart and checkouts on an unconfigured tracker
- Then the tag manager is initialised and it creates a new tracker for the given idSite and trackerURL. At this time _paq.push was already processed so it won't actually execute the original _paq.push call.

I've been looking for a fix for this for a long time trying various different approaches but they all don't work so far. I will create an issue for this in the tag manager repository where we need to see if/how this can be fixed. It might need a big refactoring of tracking code and/or tag manager.

For example I made a start like

diff --git a/js/piwik.js b/js/piwik.js
index 6c88b24835..1e1acee79d 100644
--- a/js/piwik.js
+++ b/js/piwik.js
@@ -7310,7 +7310,7 @@ if (typeof window.Matomo !== 'object') {
 (function () {
     'use strict';

-    function hasPaqConfiguration()
+    function hasPaqConfiguration(_paq)
     {
         if ('object' !== typeof _paq) {
             return false;
@@ -7320,8 +7320,22 @@ if (typeof window.Matomo !== 'object') {
         if ('undefined' === lengthType) {
             return false;
         }
+        if (!_paq.length) {
+            return false;
+        }
+
+        var hasIdSite = false;
+        var hasTrackerUrl = false;
+        var i;
+        for (i = 0; i < _paq.length; i++) {
+            var paqmethodtype = typeof _paq[i];
+            if (paqmethodtype === 'object' && _paq instanceof Array && _paq[i].length > 0 && _paq[i][0]) {
+                hasIdSite = hasIdSite || ( _paq[i][0] === 'setSiteId');
+                hasTrackerUrl = hasTrackerUrl || (_paq[i][0] === 'setTrackerUrl');
+            }
+        }

-        return !!_paq.length;
+        return hasIdSite && hasTrackerUrl;
     }

     if (window
@@ -7345,18 +7359,22 @@ if (typeof window.Matomo !== 'object') {

     if (!window.Matomo.getAsyncTrackers().length) {
         // we only create an initial tracker when no other async tracker has been created yet in matomoAsyncInit()
-        if (hasPaqConfiguration()) {
+        if (hasPaqConfiguration(window._paq)) {
             // we only create an initial tracker if there is a configuration for it via _paq. Otherwise
             // Matomo.getAsyncTrackers() would return unconfigured trackers
             window.Matomo.addTracker();
         } else {
-            _paq = {push: function (args) {
-                    // needed to write it this way for jslint
-                    var consoleType = typeof console;
-                    if (consoleType !== 'undefined' && console && console.error) {
-                        console.error('_paq.push() was used but Matomo tracker was not initialized before the matomo.js file was loaded. Make sure to configure the tracker via _paq.push before loading matomo.js. Alternatively, you can create a tracker via Matomo.addTracker() manually and then use _paq.push but it may not fully work as tracker methods may not be executed in the correct order.', args);
+            (function f() {
+                var originalPaqBackup = window._paq;
+                window._paq = {push: function (args) {
+                    originalPaqBackup.push(args);
+                    if (hasPaqConfiguration(originalPaqBackup)) {
+                        window._paq = originalPaqBackup;
+                        window.Matomo.addTracker();
                     }
                 }};
+            })();
+
         }
     }

to only create a tracking instance in the beginning when the tracking code is loaded if the _paq array also contains an idSite and a trackerUrl. I think something like this is the right approach and could work. However, there might be edge case conditions that we'd need to check for. Although that might just work and would improve overall the tracking code as there can be less mistakes made (eg if _paq is executed after the tracker was loaded). Code is not tested.

peterbo commented 3 years ago

Hi @tsteur, I stumbled upon this problem as well. The use case is to set the Cookie Lifetime using:

var _paq = window._paq = window._paq || [];
 _paq.push(['setVisitorCookieTimeout', 604800]);
 _paq.push(['setSessionCookieTimeout', 0]);

The Tag, containing the Cookie settings is set with priority 950 (the PageView Tag has 999).

After that, the "Pageview" is fired, but it ignored the manual cookie settings. This is quite critical, because quite some instances use that feature (it is documented or in the forum somewhere), but it doesn't get applied. So they basically violate their own cookie policy. Looking into the tracker configuration gives an idea, what happened:

_paq.push([ function() { console.log(this.getTrackerUrl())}]);

returns 2 items, 1) an empty string (tracker that got initialized without configuration) and 2) the "real" tracker, initialized by the TagManager Matomo Configuration.

This also applies for other Tags that are being used with _paq directly, which is necessary quite often (e.g. events/pageviews with an individual set of custom dimensions). Using another TagManager Matomo Configuration with these individual Custom Dimensions is not very practical, because a) a lot of Configuations (Variables) would be needed, and b) tracking events with another Matomo Configuration creates another pv_id in the background (https://github.com/matomo-org/tag-manager/issues/242)

peterbo commented 3 years ago

@tsteur sorry, I forgot that we already came up with a solution for that. The "Initialise tracker, don't track anything" Matomo-Tag was introduced to solve this. I'm keeping the last comment, because it is probably another use case, but the workaround (tracker init) works!

froger-me commented 2 years ago

How about triggering an event on the window when _paq becomes available to use for sure? Something like:

window._paqReadyTrigger = function(data) {
  window._paqReady = new CustomEvent('_paqReady', {'data' : data});
  window.dispatchEvent(window._paqReady);
};

// Some Core Code...

var someData = getTheData();

window._paqReadyTrigger(someData);

// Some More Core Code...

This way, whether tag manager is being used or not, the code for users could be safely called like:

window.addEventListener('_paqReady', function (e) {
  doSomething(e.data);
});

function doSomething(data) {
  window._paq && window._paq.push(['trackEvent', 'EventCategory', 'EventAction', 'EventName', eventValue]);
}

For now, as an unreliable workaround of sorts, I'm using:

var someUnreliableTime = 1000;

setTimeout(function() {
  window._paq && window._paq.push(['trackEvent', 'EventCategory', 'EventAction', 'EventName', eventValue]);
}, someUnreliableTime);

snake14 commented 1 year ago

I believe that the fix for this has been merged and will be in the next Matomo release.

tsteur commented 1 year ago

I'll reopen this one as I believe the fix was reverted in https://github.com/matomo-org/tag-manager/pull/620 ?

Feel free to close it again if we meanwhile fixed it differently?

For us, this issue has been the biggest and most frustrating problem in Tag Manager. It be great to consider fixing it at some point if it's still not fixed.

cc @snake14 @Stan-vw

cataclyst commented 1 year ago

FWIW, I also have a lot of problems with the fact that _paq.push() and MTM don't play well together in my setup. In my case, we need to track the exact same data into two Matomo websites A and B (same Matomo URL) and we also need to use MTM. We have two configuration variables, one for idSite "A" and one for idSite "B", everything else is identical.

Especially on page load, I am seeing a lot of requests that I _paq.push() get either delivered to website A or website B, but not both. At some point, to work around that, I built some queuing mechanism in our application to make sure that I do not actually _paq.push() until both "trackers" are available, but it's still a pain.

And I still have issues with a lot of the "built-in" mechanisms of Matomo not being delivered to both websites, like Content Tracking, and those problems always seemed to trace back to MTM not handling multiple trackers properly.

I understand that especially the last piece is not the exact same thing as described in the OP (events from "before loading the tracker" getting discarded), but I believe they are somewhat related and rooted in the same problem.

matomo-org / tag-manager

_paq.push cannot be used when also using Tag Manager unless it is used in the same Tag Manager container itself. #300