queuedtracking:process randomly running slow

[bijancot]

Hi all thank you for creating this plugin, really helping us here.

But we facing some issues regarding performance when trying to run process the queue.

here is our command to run process with such this command :

php console queuedtracking:process -n --ignore-warn --no-ansi --queue-id=<some-number> --force-num-requests-process-at-once=500 > /dev/null

we divide it to 16 worker and expect for this tracking process to run parallel but it seems to run sequentially for us. Is there any other config parameter or something that we missed?

One of the case :

we run 16 worker with cronjob. and only cron that consume queue 0 that working hard and other queue randomly working and idling

any suggestion guys?

Thank you!

[snake14]

Hi @bijancot . Thank you for taking the time to create this issue. I'm sorry to hear that you are experiencing this issue. My first thought is that 500 seems a bit high for the number of requests to process at once; I think that the default is 25. The 16 workers should run in parallel when you execute the console command, if that's what you have configured for Number of queue workers in Admin Settings. Are you trying to run the cron in conjunction to having the Process during tracking request setting enabled? It's best to use one or the other. Have you tried using the queuedtracking:monitor console command or the -vvv option to output more debugging information? This FAQ has some other troubleshooting information.

[bijancot]

Hi @snake14 Thank you for replying my question,

Will try your suggestion, but one more question. Why it is not recommended to run both Process during tracking request and cronjob cli ? is it will make the queue process confused/repetitive data being inserted to db?

Because if we can enable both of them it will be a great option to ingest data faster to db right?

Best Regards!

[bijancot]

Hi @snake14 for more context for now our traffic is reaching more than 20k+ active users per seconds. That's why we have 500 for --force-num-requests-process-at-once=

Looking forward for any other suggestion

Best Regards , Panji

[snake14]

Hi @bijancot . It's typically better to use one or the other. If you've got that many requests coming in, it might be good to disable Process during tracking request and rely on the cron. I think that the cron would wait till the last request triggered process finished before processing anyway.

Any ideas/recommendations @AltamashShaikh ?

[AltamashShaikh]

@snake14 I couldn't think of anything else, reading this FAQ it does recommend to lower the no of requests to reduce blocking time.

When using multiple workers it might be worth to lower the number of "Number of requests to process" to eg 15 in "Plugin Settings". By default 25 requests are inserted in one step by using transactions. This means different workers might have to wait for each other. By lowering that number each worker will block the DB for less time.

[bijancot]

Hi @snake14 and @AltamashShaikh thank you for you guys responses, so even how big the traffic and data that being queued it's always recommended to set it to max 25 right?

and one more thing we found some strange issues with some queries (i think it's not 100% related with our queue issues) where we can discuss it?

Best regards, Panji

[AltamashShaikh]

@bijancot The 25 no is decided to keep the queue relatively small with less waiting time. If the queries are not related to this plugin, try opening the issue in the relevant github repository (if premium plugin - contact support), if its with the Matomo core, you should open the issue here.

[AltamashShaikh]

@bijancot Good to close this issue now ?

[bijancot]

okay got it, the information is sufficient enough for me to close this issues. Thank you so much! closing this issues now