search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.9k stars 2.65k forks source link

actions recorded does not match expected in press test #13617

Open yabaly opened 6 years ago

yabaly commented 6 years ago

we are running a press test to test whether matomo tracking correctly under high volume visit environment. we installed Queuedtracking plugin using redis. the db is Mysql. In our test, we simply create severally session request tracking API concurrently, all the Url are same: http://192.168.1.99:10086/piwik.php?idsite=2&rec=1&e_c=Games&e_a=Duration&e_n=jojo&e_v=1000&cookie=1&token_auth=e8c636b7940beb7d2418f93d50bd92c1 after a while we stop request and waiting for 10 minutes then quest on the report api: http://192.168.1.99:10086/index.php?module=API&method=VisitsSummary.getActions&idSite=2&period=day&date=today&token_auth=e8c636b7940beb7d2418f93d50bd92c1

In case we have hundreds of concurrence request, the matomo data look perfect. When we have 4000 concurrent threads and a totally 60000 request on tracking API, while 99.9% of request return 200, the value get from report api is round 25000, significantly less than the expected value.

can you give any suggestion about testing the tracking performance in developer environment? 3.6.1-b3

tsteur commented 6 years ago

Make sure all queued requests were actually processed see ./console queuedtracking:monitor and ./console queuedtracking:process. If that's the case can you check in the log tables whether all requests were recorded and are in there? Like you would see eg 60K entries in log_link_visit_action

yabaly commented 6 years ago

Sincerely thank you for advice! A large numbers of request left in queue, I can manually execute ./console queuedtracking:process , have some requests recorded in database, then the process stale again, only after repeatedly executed in console all the request left in queue have been recorded.

Also I found key list is empty when ask redis-cli show all keys, while there is still request set left in queue

It happens when there large amount of request, eager you can give any suggestion.