search

matomo-org / plugin-QueuedTracking

Scale your large traffic Matomo service by queuing tracking requests (in Redis or MySQL) for better performance.
https://matomo.org
GNU General Public License v3.0
82 stars 35 forks source link

new setting to enable/disable Provider reverse dns lookup #136

Open sgoeting opened 4 years ago

sgoeting commented 4 years ago

Hi we have come a long way and finally we have reproduced the issue which we are facing since february this year.

When we use queuedtracking with a cronjob (with console queuedtracking:process, in stead of using the process using the tracking-request) the location provider will always be unknown (ie NULL, not set).

We are using Matomo in high volume envirronment. That is, we have 3 frondends for request-registration and use redis/sentinel configured queuedtracking. We have 2 separated queuedtracking-servers for the cronjob-controlled queuedtracking. Next to previous mentioned servers we also have an dedicaticate administrative machine. All Matomo-servers (6) use the same config-file. Furthermore we have a master slave DB in read/write configuration.

This configuration works smooth and now processes effortless millions of requests per day. One minor(!) issue: the location_provider-field is not set. After a long quest we have managed to reproduce above mentioned fenomena: When using cronjob-based queuedtracking the location_provider-field is not set, because of the VisitDimension-hook 'onNewVisit' of the Provider-plugin is NOT executed. When using de Frontend-capabilities of processing the request directly (toggle 'Process during tracking rquest' is set) the location_provider-field will be set because, rather curious, in that case the VisitDimension-hook is executed. From performance-perspective we can NOT use the Frontend-capabilities, we have to use the cronjob style.

We are using Matomo 3.13.5 en the appropriate versioned plugins.

I hope you can help us with this problem :-)

tsteur commented 4 years ago

Thanks for creating this issue @sgoeting

This is actually happening on purpose right now and is expected behavior as we disable the provider plugin in https://github.com/matomo-org/plugin-QueuedTracking/blob/3.3.6/Commands/Process.php#L61 as it's usually slowing down queued tracking quite a bit (or even a lot).

Feel free to disable that line and give it a test. Maybe it's not making such a big difference anymore or we could add an option to not disable it.

sgoeting commented 4 years ago

@tsteur thankt for the quick response!

We were a bit surprised by the answer but okay after disabling of the disabling of the Provider plugin it seems to work fine now. As far as we can see for now, we don not have any performance issues, but we will monitor this for a few days...

For us it would be very nice to have a toggle somewhere in the config or in GUI to en-/disable this 'feature', so maybe in the near future..... :-)

Nevertheless, thanks! It solved our issue

peterbo commented 8 months ago

+1 for removing this code from the QueuedTracking (or making it optional). If this is too slow, a user still can ultimately disable the Providers Plugin.

iammattmartin commented 8 months ago

Agree. We've commented that out over 2 years ago and there have not been any perf issues.