Understand why conversions are sometimes attributed to "Direct entry" or the payment provider, instead of the current channel

[tsteur]

from https://forum.matomo.org/t/referral-exclusions/33582/25

The resolution suggested by Matomo in this article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ does not really address the underlying issue - it merely recategorizes the referrer from “website” to “direct entry”, which in reality isn’t the true referrer for the user’s visit.

Here is an example of what is happening.

1. User clicks on an email link to visit the website.
2. User browses website and adds items to basket (at this stage referrer is “Email campaign”)
3. User completes the order but and website redirects to 3rd party payment provider (paypal/amazon etc)
4. User returns to website (but now the referrer for the visit is reset from “Email campaign” to “website” or if you have set the referrer exclusions it will be set to “direct entry”. What we need is for it to remain as “Email Campaign”.

As a result we are over attributing the channel “direct entry” greatly (40% vs 15% on GA).

I'm not sure if we actually should call this feature "Referrer exclusion" as it's more like an "ignore refrerrer" or "keep the original referrer"? We wouldn't exclude this tracking request. We would still track that referrer but re-use the original referrer from the same visit. If it's from a different visit then we would use again a direct visit?

[tsteur]

@mattab Any idea what this feature could be named and how we would configure this? Not sure if we want to have this per site and globally or whether it be enough only one way etc. It could get confusing maybe for people that don't need this feature. Maybe it could be even a separate plugin?

[heurteph-ei]

In my point of view, if a user :

1. Comes from one referrer,
2. Then browse the measured site (tracked by Matomo),
3. Then browse out the measured site (eg. payment gateway or really leave the measured site)
4. Then come back to the measured site during the 30 minutes delay (for any reason: payment or anything else)

Then the referrer should at least be the 1st one! (would it be possible to accumulate referrers?) And if the other referrer is considered as "direct entry” (what is done in case of excluded referrer) it means the only real referrer is still the 1st one...

Note also that in case of payment gateway, the returning back delay should not exceed short delay as in other ways, the payment token would probably expire... Then the scenario where the 1st referrer is a payment gateway should be very rare case...

[adsham]

Thanks @tsteur for creating the issue here on my behalf.

I largely agree with @heurteph-ei - the main reason for referral data is to monitor how traffic is being driven to the website, especially if we attribute value from sales to a referrer. I can't really see any benefit of having "direct entry" as the referrer as it is misleading. However I do see a legitmate scenario that could be obscured by always using the first referrer.

Here are 2 examples

1)

• Comes from Email referrer,
• Then browse the measured site (tracked by Matomo),
• Then browse out the measured site (eg. payment gateway or really leave the measured site)
• Then come back to the measured site during the lifetime of a visit (e.g. 30mins)

In the above scenario - Paypal (for example) should be set up as a referral exclusion and therefore the 1st referrer should be used (Email).

2)

• Comes from Email referrer,
• Then browse the measured site (tracked by Matomo),
• Then browse out the measured site (to search for a coupon code or referral link)
• Then come back to the measured site during the lifetime of a visit (e.g. 30mins)

In the above scenario we may want to know both the former or the latter referrer.

With this in mind I would suggest either...

• Making it a configuration option ("use first referrer" or "use latest referrer") at the measurable level ( in combination with the referral exclusions).

or

• Adding a secondary field(s) to the underlying data so the user can choose to report on either first or last referrer - (and also make this new data available via the api)

[mattab]

What i don't understand yet, is that the scenarios above should already work meaning: as long as the subsequent page views (after purchase) occur within 30min of the last pageview on the website, then the pageviews should be stored in the existing visit (with the correct original campaign). The fact that it doesn't work for some people is not clear.

@adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview?

(the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case)

[adsham]

Hi @mattab

Thanks for picking this up. You will see that several users have picked up on this issue here https://forum.matomo.org/t/referral-exclusions/33582/22 and possibly here too https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721/6

I have also just looked through our data and can confirm that there are no "delays" as you are suggesting. However I'm not really in a position to fully test this out at the moment (we do not have a test payment provider currently configured) perhaps @heurteph-ei or someone from the forum could do this? I will ask there too.

Could the resolution in the article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ be causing the original referrer to be overwritten with "direct entry"?

Many thanks once again, hopefully we can find a resolution to this soon!

[CryptoEU]

Hello,

for me its very simple and there are different ways, how you can create the "correct" goal*1.

*1Goal: as already written. We have to know, from where the customer is coming from. If Matomo delete now all referers (change to direct entry, using the payment gateway as referrer) thats just washing statistic with 0 sense. Nobody can track now, from where the customer really came then - ofc - he didnt came from PP, Klarna, CC etc.

I am not a technician, so I have no idea, why some 30 mins. timeframe is here on the table. Sometimes its happend, that someone needs - because of distraction - longer with PP then 30 mins (PP=PayPal). The usual session for PP is running much longer also "all" shop sessions. To me - 30 mins - are fine. Ofc some would be maybe out of this window, but thats 0.x% and ok if we dont have them tracked.

To make it lesser complicated: Why not just making a "just track the first referrer" option in a XX min. timeframe? or Just track (and show!) all referrers.*2

*2 this would have the advantage, that you can directly see a fast overview of the daily payment gatesways too, without needing your ERP.

Exlucde paymentgateways and putting all referrers as direct entry or use the gateway as referrer is for nobody really helpful.

[frimipiso]

Hey there,

it is great that this issue is picket up now. I have been experiencing this problem for a long time now.

Here is our setup:

• Shopware 5.6 Shopsystem with paypal as payment provider
• When the user goes to paypal for payment and returns to our Shopware shop, a new session is started with referrer paypal.

This completely messes up all conversion attribution, as most customers use paypal.

All customers return from paypal within the 30 minute timespan.

I would opt for an "attribute to first/last referrer" configuration in matomo.

Thanks for taking care of this and let me know if I can be of further help.

Cheers

Jens

P.S.: If I would use the ignore referrer option, the new session would have a direct access referrer which even more messes up attribution, since I cannot identify anymore that paypal messed up the referrer.

[SimonNL]

Coming from GA and having struggled with the same issue there years ago, I quickly ran into this again with Matomo. Specifically the 'payment service provider' scenario. With GA we started with exclusion list but could finally solve this for good by adding the 'ignore_referrer' parameter to the PSP return/error/pending page (e.g. /thank-you?ignore_referrer=true) so regardless of the payment method, the site in between is always ignored. This works better than referral exclusion lists because the payment urls tend to change over time and new ones are easily overlooked. A lot of e-commerce sites will already be using this parameter so would supporting this not solve the issue for many cases?

[adsham]

I'd like to provide some examples to demonstrate the overreporting of direct entries in one of our clients, I can confidently say the “direct entry” attribution is far too high using all of the Matomo attribution techniques.

Here is a monthly view

Here is a yearly view

Here is a segmented yearly view for new visitors (I wouldn’t expect new users to be showing such high direct entries)

Therefore I’m fairly certain something is not attributing correctly, whether it’s us the users (it seems there are many users as above having similar issues) who have not understood so something or whether Matomo tracking is not attributing as it should in some cases I’m not sure - any suggestions are welcome.

[ts1985]

We have the same problem. I think most have the same problem and it's important to find a solution for this. The conversions are thus virtually useless, since it distorts everything. Without configuration you have conversions to the payment providers, but you want to know where the visitor originally came from and not which payment provider he used. If you put the payment providers on the referral exclusion list, then you have the conversions on "Direct Entry" which doesn't make sense because the customer might have come from search engines, websites etc..

Hope somebody will find a solution soon.

[adsham]

mattab , tsteur

What i don't understand yet, is that the scenarios above should already work meaning: as long as the subsequent page views (after purchase) occur within 30min of the last pageview on the website, then the pageviews should be stored in the existing visit (with the correct original campaign). The fact that it doesn't work for some people is not clear.

@adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview?

(the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case)

Perhaps then this is a bug rather than an enhancement? - it seems that many of us are experiencing this issue.

[ts1985]

I can give you these details:

• No they don't use another browser to pay
• It's not just PayPal. We have several Payment Providers, all with the same problem.

[adsham]

Any update on this? It seems to be pushed back again into a later milestone.

The referral data is an important part of our service and we have recently seen a client leave to a competitor because of the inaccurate data in our reports (from Matomo).

The initial issue was raised almost 3 years ago in the forum https://forum.matomo.org/t/referral-exclusions/33582/25 and this Github issue was raised 2 months ago and pushed back into milstone 4.10 (4.8 released today).

Please could we have some committment as to when this will be addressed as it's getting harder to explain to clients that the issue is being looked at but cannot tell them when it will be looked into/fixed.

Many thanks in advance! Adham

[Demichev]

Hello. We have the same issue. I temporarily modified Matomo code (return false immediately in onExistingVisit() for Referrers Columns classes). We are testing it now. I think it will solve the problem for us but it is good to have an official option. What do you think?

[adsham]

Hello. We have the same issue. I temporarily modified Matomo code (return false immediately in onExistingVisit() for Referrers Columns classes). We are testing it now. I think it will solve the problem for us but it is good to have an official option. What do you think?

Hi @Demichev - this is interesting, we are still waiting on an update. We have lost business over this and have built some of our services around the referrer functionality so it is important that a solution is found. It would be great if you could share your experience and solution if you find one, perhaps it will the developers with an official solution?

Also @justinvelluppillai What is the "backlog" milestone? Will these be actioned? Do we have an idea of timings? Many thanks

[Demichev]

Hi @Demichev - this is interesting, we are still waiting on an update. We have lost business over this and have built some of our services around the referrer functionality so it is important that a solution is found. It would be great if you could share your experience and solution if you find one, perhaps it will the developers with an official solution?

Hello @adsham. I should clarify my approach. I did not try to exclude payment site referrers, I disabled all referrers from late actions. Only referrer from the first action of each visit is recorded if any. Payment site still can be set as referrer if visit is expired. We are experimenting with visit length up to one day. I can share my changes but they should be restored each time Matomo is updated. We have some other changes and it is real pain to update Matomo now. So I decided to try to write here some feature requests.

[adsham]

Hi @Demichev - this is interesting, we are still waiting on an update. We have lost business over this and have built some of our services around the referrer functionality so it is important that a solution is found. It would be great if you could share your experience and solution if you find one, perhaps it will the developers with an official solution?

Hello @adsham. I should clarify my approach. I did not try to exclude payment site referrers, I disabled all referrers from late actions. Only referrer from the first action of each visit is recorded if any. Payment site still can be set as referrer if visit is expired. We are experimenting with visit length up to one day. I can share my changes but they should be restored each time Matomo is updated. We have some other changes and it is real pain to update Matomo now. So I decided to try to write here some feature requests.

Thanks @Demichev - your approach would satisfy my requirements as it's the initial referrer which is most important in our case. Unfortunately we use the hosted service so I cannot make these changes to our tracking.

[adsham]

Here is another example of this issue (or related issue) - a new project we are doing a "proof of concept" - unfortunately it's recording intra-website campaigns which isn't very useful for what we are trying to demonstrate to a potential client. This may be the intended behaviour, but I would suggest that it is the first referrer to the site that is important in most cases. Perhaps a FirstReferrer & LastReferrer field would take care of this?

(Tiles & carousel campaign mediums here are links clicked within the website)

[adsham]

Hi @mattab

Could you provide an update on this, I'm receiving a lot of pressure both internally and externally and I'm still none the wiser as to when this will be investigated/resolved. I appreciate you have other priorities and issues, but we are currently paying $300-$500 a month for a service that isn't providing us with the data that it should.

This issue was raised on the other forum in 2019 and this github issue in January - we are also a small team so we appreciate that things need to be prioritized and scheduled, but I can't even tell our clients or my manager that this will be resolved by the end of the month, the end of next month, next year or never.

I would also say this doesn't seem to be an isolated issue, many users have reported the same issue.

[justinvelluppillai]

Hi @adsham I agree this issue is important and I can see how moving it around the milestones has been unhelpful in terms of providing clarity about when it will be done. I'll see if we can provide a timeframe for this being addressed soon.

[adsham]

Quick way to recreate this issue. Rather than a later referrer “overwriting” the first referrer – it is generating a new visit when a new referrer/campaign is encountered during the visit – and this subsequent visit will get associated with the conversion/goal.

1) On a Matomo tracked website (standard javascript tracking). Click a link or visit the website with a testable referrer. In this example I visit… https://XXXXX.co.uk/?utm_campaign=testA Looking in the real time tracking I can see the visit (Visit 4 – April 25th 2022 13.40:46)

2) In the same browser session, I edit the final character of the URL, so that the campaign now becomes testB https://XXXXX.co.uk/?utm_campaign=testB and press return. This now generates a new visit – in this example you can see visit 5 (April 25 2022 13.41:46).

In a real-world situation what this means is that it is the latter referrer that gets associated to the visit that goes on to make an ecommerce order or goal. When the “real” referrer in the test above was testA and should be associated with the subsequent ecommerce order/goal (unfortunately testB will get the credit in this example). My understanding from the documentation and response above is this should be tracked as the same visit as it’s in the same browser tab and within 30 mins (with no other activity between each action). I have tested my test above without the utm_campaign tag and all page views are correctly allocated to the 1 visit.

[Demichev]

@adsham There are two settings in Matomo config. I don't know if they are available for change in hosted version. I think you need create_new_visit_when_campaign_changes = 0.

; if set to 1, actions that contain different campaign information from the visitor's ongoing visit will ; be treated as the start of a new visit. This will include situations when campaign information was absent before, ; but is present now. create_new_visit_when_campaign_changes = 1

; if set to 1, actions that contain different website referrer information from the visitor's ongoing visit ; will be treated as the start of a new visit. This will include situations when website referrer information was ; absent before, but is present now. create_new_visit_when_website_referrer_changes = 0

I want the option to use referrer from the first action only. I think this option has sense.

[justinvelluppillai]

The create_new_visit_when_campaign_changes setting seems relevant here. There is also the option to use first referrer attribution for conversions with _paq.push(['setConversionAttributionFirstReferrer', true]); (as described here: https://matomo.org/faq/general/faq_106/).

[heurteph-ei]

@justinvelluppillai reading the FAQ, I think the setConversionAttributionFirstReferrer should work on several visits. Not in the case of a single visit. The problem in this thread is during a single visit (within the 30 min timeframe):

• Referrer A
• Some Pages (at this time the referrer is A)
• Outlink to the payment provider
• Thankyou page _(now the referrer is the payment provider)

The rule (for a visit) should be: if the page is the first one of the visit, then the referrer (or campaign or etc.) must be registered. In other case, the referrer shouldn't be changed...

[justinvelluppillai]

That's correct re setConversionAttributionFirstReferrer. We will look into the other problem when we review this issue next milestone - also the first comment's issue of many of them maybe being attributed to Direct Entry seems unexpected.

[adsham]

Hi @justinvelluppillai

It would also be useful to know how those of us with the hosted Matomo service (Innocraft) can change these config settings. As far as I am aware there is no option to do this, but can these settings be changed for us server-side? If so what is the process for doing this?

Thanks

[justinvelluppillai]

Hi @adsham, Matomo Cloud customers can get in touch with our support team about changing config settings that can't be changed in the UI.

[adsham]

HI @justinvelluppillai

How do Matomo Cloud users contact support? I sent an email 7 days ago with no response? It's really frustrating with clients waiting. There is no ticket number or acknowledgement.

May I suggest a support ticket system, where we can track progress? On the website it states that Matomo Cloud offers "In-depth" support, but currently we are feeling under supported, despite paying $300-$600 a month.

For instance - If we require a config change as above.

1) What is the request process? 2) How long should we anticipate before an acknowledgement? 3) Can we expect an estimated timeframe for completion? 4) Are their any SLA commitments from Matomo?

Many thanks Adham

[frimipiso]

It's too bad that there is still no solution to this problem.

We want to start using GoogleAds now, but we cannot attribute conversions to Ad campaigns if the customers pay via Paypal.

So we are forced to move back to Google Analytics which is a shame because we heavily adapted our shops for Matomo.

But we will essentially be wasting money if we cannot attribute sales to campaigns ...

Cheers

Jens

[mattab]

Sorry for the delay it's taking to address this issue, we agree it is impactful. it is now high priority and will be investigated by the team this month. Stay tuned for some updates & thank you for all your reports :+1:

[sgiehl]

Ok. First of all I will summarize the current behavior that is implemented in Matomo regarding referrer handling:

Whenever the referrer details changes within a visit Matomo currently forces a new visit in this cases:

• A website referrer was detected and config create_new_visit_when_website_referrer_changes = 1
• A campaign change was detected, the current referrer wasn't a direct entry and config create_new_visit_when_campaign_changes = 1

If the details changed (to something other than direct entry), but no new visit was forced (due to config) the referrer information will be updated in the current visit if the visit wasn't a direct entry initially.

---

So looking through all the issue comments my understanding would be, that we need some kind of exclusion list. If a user is coming from a url/host on that list, this action will be treated as Direct Entry, which will then prevent updating the referrer details or forcing a new visit.

I would add that list the way we are currently handling query parameter exclusion. So that means there is a global configuration option, but also the possibility to add urls/hosts on a per site level.

I will now start working on that. If there is any additional input or requirements that I should take into account when implementing that. Let me know.

[ts1985]

I don't think that this is the solution. To explain:

1. User visits example.com (doesn't matter if it's direct entry or a referrer. Let's say it was direct entry for this example)
2. User buys something and is forwarded to paypal.com
3. User comes back after he paid to example.com and reached the goal

Current issue: If you check now the reached goal, you see, that this visitor reached the goal from paypal.com and not from direct entry. This information is completely wrong. Your goal is now full of visits from paypal.com and other payment providers. But you want to know where the visitor was from and not what payment provider he used during his visit.

If you now integrate an exclusion list and add paypal.com on this list and "this action will be treated as Direct Entry" the problem is, that all visitors using paypal.com as payment method are as "direct entry" in the goal information. But maybe it was not a direct entry, it was a visitor from google.com?

It's really important to make sure that this issue is solved and no other issues are build with this solution. Otherwise more wrong data is in the system.

[Demichev]

Why should be referrer updated in the first place? What if I want one visit with the original (first) referrer?

[heurteph-ei]

Why not just 2 options (at site level):

• use_last_referrer_as_visit_referrer (default = 0)
  • case 0: same behavior as now (the last referrer overwrites any other referrer)
  • case 1: the first referrer stays the only referrer for the whole visit (even if it was direct entry?)
• use_last_campaign_as_visit_referrer` (default = 0)
  • same idea as the other option...

(you have to decide which ones between use_last_*_as_visit_referrer and create_new_visit_when_*_changes has the most precedence...)

[sgiehl]

@heurteph-ei I'm not sure if that would allow all required use cases. It might be possible someone wants the referrer to be updated or a new visit to be forced when the referrer changes, but still wants payment proxies (or similar) not to trigger this behavior. Having a list of urls that are kind of treated as direct entry, should give more flexibility I guess.

But for sure we could also discuss about a configuration value to disallow overwriting the referrer at all (if no new visits are forced).

[ts1985]

Why not just 2 options (at site level):

• use_last_referrer_as_visit_referrer (default = 0)

  • case 0: same behavior as now (the last referrer overwrites any other referrer)

Makes no sense. As we here already discussed, the current behavior results into wrong information. Leaving this bug as default would therefore not be helpful.

[sgiehl]

@ts1985 My explanation might not have been detailed enough. A tracked action that is handled as direct entry will never update the referrer stored in database. So when the user comes back from the payment provider, within the visit time frame, the action would then no longer trigger a new visit, nor would it update the referrer that was initially set with the visit (if the referrer is ignored, as described above).

[heurteph-ei]

@ts1985 this was proposed for 2 reasons:

1. This has been implemented like this, so I assume there was a good reason... Also as I am not in the head of everybody, maybe some business need to keep only the last referrer...
2. By default, we must not change the behavior of existing Matomo instances (in case of case 1). But for new instances, you're right, the default value should be 1 (as I think that it is the behavior most user would expect)

@sgiehl This is for the case where some "in-process provider" (like payment providers) can also be a referrer... That can't be excluded...

[mattab]

Reading the recent comments, and as suggested by you, the problem is that the ecommerce/checkout subdomain (paypal.com) appears as a "website" referrer. And because Matomo by default credits ecommerce conversion to the last non direct referrer, then all ecommerce attribution reports show the paypal.com domain (making all attribution reports useless because then it won't show you the actual acquisition channel used by the user...). then the ecommerce attribution works fine. But the "Multi Channel" reports will show the paypal.com ( https://matomo.org/faq/reports/set-up-and-analyse-multi-channel-conversion-attribution-reports/ )

So Instead if Matomo had a list of domains used by the most common ecommerce/checkout platforms (including the paypal.com), then Matomo could ignore this domain and instead make the visit a "direct entry".

We could keep the list with these domains in the global.ini.php so people could also override the list if they want (or empty the list, if for some reasons they want to see the paypal.com referrer). We could add new domains to the list on request or via pull requests. And it would make sense to have this become the default behavior by default.

Maybe that could work nicely, but even when it's completed, i'm not yet 100% sure that if fixes everyone's issues here - to be confirmed?

[adsham]

Reading the recent comments, and as suggested by you, the problem is that the ecommerce/checkout subdomain (paypal.com) appears as a "website" referrer. And because Matomo by default credits ecommerce conversion to the last non direct referrer, then all ecommerce attribution reports show the paypal.com domain (making all attribution reports useless because then it won't show you the actual acquisition channel used by the user...). BTW one solution is to use Multi Channel attribution to get this information: https://matomo.org/faq/reports/set-up-and-analyse-multi-channel-conversion-attribution-reports/

But obviously it's a big issue, and it could make a lot of sense to have a "special" handling of websites acting as checkout/ecommerce platforms (such as paypal.com).

So Instead if Matomo had a list of domains used by the most common ecommerce/checkout platforms (including the paypal.com), then Matomo could ignore this domain and instead make the visit a "direct entry".

We could keep the list with these domains in the global.ini.php so people could also override the list if they want (or empty the list, if for some reasons they want to see the paypal.com referrer). We could add new domains to the list on request or via pull requests. And it would make sense to have this become the default behavior by default.

Maybe that could work nicely, but even when it's completed, i'm not yet 100% sure that if fixes everyone's issues here - to be confirmed?

Thanks for giving this some thought. This has already been provided as a solution here https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ however this doesn't provide a real solution and still hides the first real referrer.

I'd like to provide some examples to demonstrate the overreporting of direct entries in one of our clients, I can confidently say the “direct entry” attribution is far too high using all of the Matomo attribution techniques.

Here is a monthly view

Here is a yearly view

Here is a segmented yearly view for new visitors (I wouldn’t expect new users to be showing such high direct entries)

Therefore I’m fairly certain something is not attributing correctly, whether it’s us the users (it seems there are many users as above having similar issues) who have not understood so something or whether Matomo tracking is not attributing as it should in some cases I’m not sure - any suggestions are welcome.

In my response above https://forum.matomo.org/t/referral-exclusions/33582/25 You can see that using the marketing attribution report we have an extremely high proportion of direct entries as a result.

[sgiehl]

Maybe there is another issue with calculating the numbers in MultiChannelConversionAttribution plugin. For now I would like to focus on how Matomo itself should track and attribute the referrers. Once this was sorted we can revisit the multi channel topic and check that.

So the current visit referrer attribution behavior when using some sort of payment provider is this:

• Visitor visits site from certain source X
• Visitor is forwarded to payment provider Y
• Visitor comes back from payment provider Y (within the visit time frame)

Now the behavior depends on the configuration and set up

• If the payment provider url was added as a site url --> The action will be attached to visit and the attributed visit referrer will remain X
• If config create_new_visit_when_website_referrer_changes = 0 (which is default) --> The action will be attached to the visit. the attributed referrer will only be updated to Y, if it was a direct entry before)
• If config create_new_visit_when_website_referrer_changes = 1 is set --> A new visit will be started having Y as attributed visit referrer

Note: If the visitor comes back from the payment provider after the visit time frame, a new visit will always be created. If the payment provider url was added as site url, that new visit will be attributed as direct entry. Otherwise this visit will have the payment provider as referrer.

So far the referrer attribution of visits. Conversion attribution has a special handling, that additionally depends on the javascript tracking:

When a visitor comes to the site from a certain source X (other than direct) and cookies are enabled, Matomo will create a attribution cookie that contains the referrer details. This cookie is sent with the tracking requests and will be used to attribute the conversions. If the cookie isn't present Matomo will automatically use the current visit referrer for attributing the conversion.

Note: If the url stored in the cookie can be resolved as a site url while tracking, it will be ignored and the visit referrer is used as well

If a visitor opens the site again, coming from any source other than the urls configured for the javascript tracker (e.g. with setDomains), the referrer cookie will be updated to this new referrer. And following conversions will be attributed to it.

This behavior can be changed by using the javascript tracker method setConversionAttributionFirstReferrer. In that case the cookie won't be overwritten and the first referrer will be used for attributing the conversions as long as the cookie exists. By default this cookie has a lifetime of 6 months, which can be adjusted using setReferralCookieTimeout

---

So thinking about the currently suggested workaround to add the payment provider to the site urls, this might be only the half way, as the conversion attribution cookie would still be updated (and later ignored) when coming back from payment provider. So ongoing visits will no longer be attributed to the first visits referrer. To solve that adding the payment providers url to setDomains might be required.

The tracking code we are suggesting to add, by default doesn't contain a call to setDomains, unless the option In the "Outlinks" report, hide clicks to known alias URLs of matomo.org is checked. But if a site for example is running and tracked on more than one domain, the referrer cookie might also get updated each time when switching between the urls, which might be unexpected behavior as well.

So overall if the payment provider is added as a site url and also added with setDomain in the javascript tracker, I actually think the attribution behavior should be correct. Or am I seeing something wrong?

[adsham]

If the payment provider url was added as a site url --> The action will be attached to visit and the attributed visit referrer will remain X

This behavior is what I think most of us expect and require - currently though, this will change X to Z (direct entry) and we lose the first referrer, X.

See this document https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/

[sgiehl]

@adsham Looking at the code I would disagree. It looks like the referrer should only be updated if the original referrer of the visit was a direct entry and if the new referrer is not direct entry. So it should never be updated from a website, campaign or search engine to direct entry. See https://github.com/matomo-org/matomo/blob/0f89a53f82d1d0d50cd1e22773be9321256bbe93/plugins/Referrers/Columns/ReferrerType.php#L58-L65

That's at least how it should work for attributing the visit. As mentioned above this is handled slightly different for the attribution of a conversion.

Are you sure that the visit's referrer is changed? If that is the case you would also see smaller number in the referrers reports for campaigns, websites and search engines. If "only" the campaign would be incorrectly attributed, that would most likely only change the ecommerce reports.

For now I will stop implementing any possible improvement like a referrer exclusion. It still seems not to be clear what the real problem is here. I will start writing some additional tests to cover as many test cases as possible around referrer attribution. Maybe that helps to discover some unexpected behavior.

[adsham]

From the documentation (https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/)...

"Then whenever Matomo will track visits in the future, any website referrer that matches the same hostnames as the “URLs” of your website, will appear as “Direct entry” instead of “Website” referrer channel."

And here is my initial forum post of what is happening in my case https://forum.matomo.org/t/referral-exclusions/33582/25.

For one of our clients (who have since left because of this) we were reporting 44% Direct Entry vs 14% in Google Analytics. We were measuring "new" customers so we knew direct entry should not be anything like 44% - comparing with GA was showing that a lot of campaign referrers were being assigned direct entry because of the 3rd party payment referral issue referenced in the above documentation.

A bit more of an explanation of the data I am using. We use the hosted/cloud version so cannot access the database and instead download visit data from the api (could this be picking up the last referrer?) I use the endpoint Live.getLastVisitsDetails - we use this data to combine with offline data (TV, radio, direct mail etc to provide an overall attribution report of online & offline activity). This is why this data must be accurate for us.

Here are fields from the returned api data we are using.

Thanks

[mattab]

@adsham

1. in your screenshot you use attribution model "Last interaction" but instead it would be more accurate/useful to use the "Last non direct" attribution model - would this maybe give better results already?
2. also you may want to use setConversionAttributionFirstReferrer if the first referrer was more relevant to you? https://matomo.org/faq/general/faq_106/

download visit data from the api (could this be picking up the last referrer?) I use the endpoint Live.getLastVisitsDetails - we use this data to combine with offline data (TV, radio, direct mail etc to provide an overall attribution report of online & offline activity). This is why this data must be accurate for us.

Yes, the data from API will return the "Current referrer" of the visit, and not the original referrer (which is only stored in the cookie). The "original referrer" stored in the cookie will be however copied into the conversion row (not the visit).

I had a quick look and i'm not actually sure if it's possible to get the RAW data for the channel attributed to the conversions. in the "action > goal" entries eg in this request, it seems the action rows with <type>goal</type> don't have the referrer information

@sgiehl

• Does the Live.getLastVisitsDetails Raw data API already able to return the referrer attributed to each goal conversion?

[mattab]

@sgiehl Thanks for the comment above, that's a great summary of how it works :+1:

So thinking about the currently suggested workaround to add the payment provider to the site urls, this might be only the half way, as the conversion attribution cookie would still be updated (and later ignored) when coming back from payment provider. So ongoing visits will no longer be attributed to the first visits referrer. To solve that adding the payment providers url to setDomains might be required.

Feedback:

1. instead of asking people to use setDomains (which they could of course if required) but maybe we could try to make it work for many people by default by putting the "checkout domains" like paypal.com etc. in the JS tracker code also by default? (from say a INI config setting) so it just works by default at least for paypal (and the most popular ones).

2. additionally, as @SimonNL suggested above, a great additional solution to catch even more cases by default would be to support this "ignore_referer" / ignore_referrer and when set, the cookie wouldn't be updated + visit referrer in Matomo wouldn't be set, it would solve the issue for even more ecommerce shops.

Proposed executive summary of the issue:

• The first goal conversion triggered by a user purchasing on Paypal is attributed correctly in all Goal/Ecommerce reports in Matomo (this works as expected)
• BUG Because matomo uses last click by default, then once the user comes back from Paypal the first time, then all future conversions will be attributed to Paypal (the issue we're all having here).

Expected instead:

• Paypal does not show up in Attribution reports, Website Referrers, Multi channel attribution reports
• Cookie does not get filled in with Paypal.
• this should work easily for other ecommerce platforms (paypal and maybe others people use a lot?), and whenever the URL contains ignore_refer[r]er=true|1

[ts1985]

This is a general but and just adding some providers like PayPal to a list, will not fix the bug. It's a workaround and the bug will still exist. There are hundred of payment processors and payment providers. Sometimes you are not forwarded to the payment provider, you are forwarded to a payment page like Adyen, Stripe and also here hundred of services.

And payment is just a topic we discuss here. There could be other cases. Maybe if you are forwarded to Google, Facebook whatever to login.

So the solution is to fix the bug and not to build workarounds. Or you have to maintain thousands of URLs and provide a easy solution for all Matomo admins to add own URLs to the list. And on the other side, Google Analytics for example is just working. Without this bug.

And the other problem in this discussion is that here are mentioned other bugs. Can you all please create own bug tickets and don't add them here? And yes, I can also confirm, that such bugs exists like that there are much!! more direct entries in Matomo than in Google Analytics.

[adsham]

@mattab first non-direct does not help it is not reporting nearly half of the conversions.

I agree with @ts1985 and I apologize if I have confused the matter - I was hoping showing the marketing attribution report would demonstrate the issue in a simple way but I suspect its muddied the water.

If we come back to the issue many of us are facing:

1. User clicks on an email link to visit the website. (Matomo tracks utm data e.g. utm_medium=email & utm_source=abandonedCartReminder and sets referrerType=campain )
2. User browses website and adds items to basket (at this stage referrer is “Email campaign”)
3. User completes the order and website redirects to 3rd party payment provider (paypal/amazon etc) or as @ts1985 this could be any 3rd party service
4. User returns to website (but now referrerType for the visit is reset from “campaign” to “website” or if you have set the referrer exclusions (https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/) it will be set to “direct entry”. What we need is for it to remain as “Email Campaign”.

It is possible that in point 4 that it is generating a new visit and splitting the 1 visit into 2, but in any case the ecommerce conversion gets credited to website or direct entry, when in the real world we need the option to have the above steps maintained as 1 visit (if inside 30 min) and also be able to report the first referrer of this visit (not first referrer of first visit or most recent referrer of this visit) was the email abandoned cart campaign. in this example

Additionally for us cloud users we need to be able to access this from the API, so @sgiehl perhaps you are correct that the code is working as expected (and is stored somewhere in the database but could you confirm how I can access this "initial referrer for each visit" from the API so that I can use this information in our reporting?

Ideally at the visit level we would have firstReferrerOfThisVisit & lastReferrerOfThisVisit data but if not possible firstReferrerOfThisVisit would be more appropriate (with a configuration option at the site level in case some users want last referrer).

Hope this clarifies things!

[sgiehl]

Thanks for all the feedback and updates. I am actually still not sure what exactly the expected behavior in each use case would be. I'm pretty sure there might be some unexpected behaviors, that we need to fix. But to be sure we don't do some adjustments on the one end and start breaking things on another end, I have started writing tests, trying to cover as many use cases as possible. It might take till early next week until I have finished writing them. Once finished, the test cases will kind of document the current behavior (unless I find something obvious I can directly fix). Afterwards I would kindly ask you all to look through the defined test scenarios to identify those, that would need to be changed/fixed. And maybe note possible test scenarios I might have missed so we can add them. I will give an update here, once I'm done with the tests.

[MatomoForumNotifications]

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/exclude-referrers-from-conversion-attribution/45949/2