QueuedTracking does not handle mysql COMMIT failures

[infomotions]

Expected Behavior

The queues would be processed without error.

Current Behavior

Intermittently the Queued tracking plugin throws errors while processing its queue. When this occurs, the processing call fails and the queues grow without being relieved. Sometimes, the call will eventually go through and things will continue to process normally, though backlogs can get very large and sometimes the error persists. Below is an example of the error.

[2021-04-29 11:09:51] piwik.ERROR: Uncaught exception: /path_to_matomo/core/Tracker/Db/Mysqli.php(432): Commit failed Commit failed #0 /path_to_matomo/plugins/QueuedTracking/Queue/Processor/Handler.php(109): Piwik\Tracker\Db\Mysqli->commit() #1 /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php(208): Piwik\Plugins\QueuedTracking\Queue\Processor\Handler->commit() #2 /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php(143): Piwik\Plugins\QueuedTracking\Queue\Processor->processRequestSets() #3 /path_to_matomo/plugins/QueuedTracking/Commands/Process.php(86): Piwik\Plugins\QueuedTracking\Queue\Processor->process() #4 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\QueuedTracking\Commands\Process->execute() #5 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run() #6 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand() #7 [internal function]: Symfony\Component\Console\Application->doRun() #8 /path_to_matomo/core/Console.php(130): call_user_func() #9 /path_to_matomo/core/Access.php(673): Piwik\Console->Piwik{closure}() #10 /path_to_matomo/core/Console.php(131): Piwik\Access::doAsSuperUser() #11 /path_to_matomo/core/Console.php(82): Piwik\Console->doRunImpl() #12 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun() #13 /path_to_matomo/console(32): Symfony\Component\Console\Application->run() #14 {main} {"exception":"[object] (Piwik\Tracker\Db\DbException(code: 0): Commit failed at /path_to_matomo/core/Tracker/Db/Mysqli.php:432)","ignoreInScreenWriter":true} {"class":"Piwik\FrontController","request_id":901677}

[2021-04-29 11:09:51] piwik.DEBUG: Loaded plugins: CoreHome, IntranetMeasurable, Actions, Referrers, UserLanguage, DevicesDetection, Goals, Events, UserCountry, GeoIp2, VisitTime, VisitorInterest, Login, UsersManager, SitesManager, PrivacyManager, Contents, BulkTracking, Resolution, Heartbeat, Marketplace, UserId, PagePerformance, CustomDimensions, AbTesting, ActivityLog, DeviceDetectorCache, FormAnalytics, MarketingCampaignsReporting, MediaAnalytics, QueuedTracking, ReferrersManager, CorePluginsAdmin, CoreAdminHome, WebsiteMeasurable, Diagnostics, CoreVisualizations, Proxy, API, Widgetize, Transitions, LanguagesManager, Dashboard, MultiSites, SEO, VisitsSummary, VisitFrequency, RssWidget, Monolog, TwoFactorAuth, Installation, CoreUpdater, CoreConsole, ScheduledReports, UserCountryMap, Live, ImageGraph, Annotations, MobileMessaging, SegmentEditor, Insights, Morpheus, Intl, CustomJsTracker, DBStats, CustomAlerts, CustomiseTranslations, CustomReports, Funnels, InvalidateReports, LogViewer, MultiChannelConversionAttribution, RollUpReporting, SearchEngineKeywordsPerformance, SecurityInfo, TasksTimetable, TreemapVisualization, UsersFlow, WhiteLabel [] {"class":"Piwik\Plugin\Manager","request_id":901677}

[Piwik\Tracker\Db\DbException]
Commit failed

Exception trace: () at /path_to_matomo/core/Tracker/Db/Mysqli.php:432 Piwik\Tracker\Db\Mysqli->commit() at /path_to_matomo/plugins/QueuedTracking/Queue/Processor/Handler.php:109 Piwik\Plugins\QueuedTracking\Queue\Processor\Handler->commit() at /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php:208 Piwik\Plugins\QueuedTracking\Queue\Processor->processRequestSets() at /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php:143 Piwik\Plugins\QueuedTracking\Queue\Processor->process() at /path_to_matomo/plugins/QueuedTracking/Commands/Process.php:86 Piwik\Plugins\QueuedTracking\Commands\Process->execute() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:257 Symfony\Component\Console\Command\Command->run() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:874 Symfony\Component\Console\Application->doRunCommand() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:195 Symfony\Component\Console\Application->doRun() at n/a:n/a call_user_func() at /path_to_matomo/core/Console.php:130 Piwik\Console->Piwik{closure}() at /path_to_matomo/core/Access.php:673 Piwik\Access::doAsSuperUser() at /path_to_matomo/core/Console.php:131 Piwik\Console->doRunImpl() at /path_to_matomo/core/Console.php:82 Piwik\Console->doRun() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:126 Symfony\Component\Console\Application->run() at /path_to_matomo/console:32

Context

This may occur at random times and requires manual intervention with the queues to recover by disabling the queues and manually clearing them.

Your Environment

Matomo 4.2.1 PHP Version: 8.0.3

[diosmosis]

Hi @infomotions, sorry you're experiencing this and thanks for reporting the issue. It does seem like QueuedTracking doesn't handle failed COMMITs, I'll mark this as a bug.

[infomotions]

Has there been any progress on this issue? It makes the plugin unusable as once it happens in a specific queue, it can grow until redis runs out of memory and deletes records. Is there no way to clear a specific "bad" item from a queue?

[tsteur]

@infomotions to understand this issue better do you maybe any more information why the commit would fail?