search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.74k stars 2.63k forks source link

[Bug] Regression: Matomo ignores --concurrent-archivers=3 option on core:archive #21985

Closed atom-box closed 1 month ago

atom-box commented 7 months ago

What happened?

You can run 4 or more crontabs of the following command, simultaneously, in parallel, Matomo doesn't try to stop you: core:archive --no-ansi --no-interaction --concurrent-archivers=3 --concurrent-requests-per-website=4 --max-archives-to-process=200 --disable-scheduled-tasks --force-idsites=11

What should happen?

When attempting to start too many archivers, here is the expected message:

INFO [20:39:22] 21625 Archiving will stop now because 14 archivers are already running and max 3 are supposed to run at once.
INFO [20:39:22] 21625 Reached maximum concurrent archivers allowed (3), aborting run.

How can this be reproduced?

For core:archive:

Here is the syntax used by the person who reported this problem: core:archive --no-ansi --no-interaction --concurrent-archivers=3 --concurrent-requests-per-website=4 --max-archives-to-process=200 --disable-scheduled-tasks --force-idsites=11

Notice the log output below. You can see that four archivers start up, each reporting that 0 of 3 are running.

(In the private Jira version of this ticket I have included more information. The user says that before upgrading to Matomo 5, this bug did not occur in Matomo 4)

Matomo version

5.0.1

Server operating system

Linux

Relevant log output

THIS IS LOG FROM RUNNING IN CRON.
NOTICE THAT WE CAN GET TOO MANY ARCHIVERS TO RUN

INFO      [20:26:01] 109953  ---------------------------
INFO      [20:26:01] 109953  INIT
INFO      [20:26:01] 109953  Running Matomo 5.0.1 as Super User
INFO      [20:26:01] 109953  ---------------------------
INFO      [20:26:01] 109953  NOTES
INFO      [20:26:01] 109953  - Async process archiving supported, using CliMulti.
INFO      [20:26:01] 109953  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:01] 109953  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:01] 109953  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:01] 109953  - Archiving was last executed without error 3 min 36s ago.
INFO      [20:26:01] 109953  - Will process 1 websites (--force-idsites)
INFO      [20:26:01] 109953  - Will process specified sites: 11
INFO      [20:26:01] 109953  ---------------------------
INFO      [20:26:01] 109953  START
INFO      [20:26:01] 109953  Starting Matomo reports archiving...
INFO      [20:26:01] 109953  0 out of 3 archivers running currently
INFO      [20:26:01] 109953  Start processing archives for site 11.
INFO      [20:26:02] 109953    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
INFO      [20:26:02] 109953    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:34] 110390  ---------------------------
INFO      [20:26:34] 110390  INIT
INFO      [20:26:34] 110390  Running Matomo 5.0.1 as Super User
INFO      [20:26:34] 110390  ---------------------------
INFO      [20:26:34] 110390  NOTES
INFO      [20:26:34] 110390  - Async process archiving supported, using CliMulti.
INFO      [20:26:34] 110390  - Reports for todaycore:archive --no-ansi --no-interaction --concurrent-archivers=3 --concurrent-requests-per-website=4 --max-archives-to- will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:34] 110390  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:34] 110390  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:34] 110390  - Archiving was last executed without error 4 min 9s ago.
INFO      [20:26:34] 110390  - Will process 1 websites (--force-idsites)
INFO      [20:26:34] 110390  - Will process specified sites: 11
INFO      [20:26:34] 110390  ---------------------------
INFO      [20:26:34] 110390  START
INFO      [20:26:34] 110390  Starting Matomo reports archiving...
INFO      [20:26:34] 110390  0 out of 3 archivers running currently
INFO      [20:26:34] 110390  Start processing archives for site 11.
INFO      [20:26:34] 110390    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
INFO      [20:26:35] 110390    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:40] 110575  ---------------------------
INFO      [20:26:40] 110575  INIT
INFO      [20:26:40] 110575  Running Matomo 5.0.1 as Super User
INFO      [20:26:40] 110575  ---------------------------
INFO      [20:26:40] 110575  NOTES
INFO      [20:26:40] 110575  - Async process archiving supported, using CliMulti.
INFO      [20:26:40] 110575  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:40] 110575  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:40] 110575  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:40] 110575  - Archiving was last executed without error 4 min 15s ago.
INFO      [20:26:40] 110575  - Will process 1 websites (--force-idsites)
INFO      [20:26:40] 110575  - Will process specified sites: 11
INFO      [20:26:40] 110575  ---------------------------
INFO      [20:26:40] 110575  START
INFO      [20:26:40] 110575  Starting Matomo reports archiving...
INFO      [20:26:40] 110575  0 out of 3 archivers running currently
INFO      [20:26:40] 110575  Start processing archives for site 11.
INFO      [20:26:40] 110575    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:41] 110575    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
INFO      [20:26:42] 111165  ---------------------------
INFO      [20:26:42] 111165  INIT
INFO      [20:26:42] 111165  Running Matomo 5.0.1 as Super User
INFO      [20:26:42] 111165  ---------------------------
INFO      [20:26:42] 111165  NOTES
INFO      [20:26:42] 111165  - Async process archiving supported, using CliMulti.
INFO      [20:26:42] 111165  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:42] 111165  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:42] 111165  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:42] 111165  - Archiving was last executed without error 4 min 17s ago.
INFO      [20:26:42] 111165  - Will process 1 websites (--force-idsites)
INFO      [20:26:42] 111165  - Will process specified sites: 11
INFO      [20:26:42] 111165  ---------------------------
INFO      [20:26:42] 111165  START
INFO      [20:26:42] 111165  Starting Matomo reports archiving...
INFO      [20:26:42] 111165  0 out of 3 archivers running currently
INFO      [20:26:42] 111165  Start processing archives for site 11.
INFO      [20:26:42] 111165    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
nohup: les entr�es sont ignor�es
INFO      [20:26:43] 111165    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).
INFO      [20:26:43] 111707  ---------------------------
INFO      [20:26:43] 111707  INIT
INFO      [20:26:43] 111707  Running Matomo 5.0.1 as Super User
INFO      [20:26:43] 111707  ---------------------------
INFO      [20:26:43] 111707  NOTES
INFO      [20:26:43] 111707  - Async process archiving supported, using CliMulti.
INFO      [20:26:43] 111707  - Reports for today will be processed at most every 14400 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [20:26:43] 111707  - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO      [20:26:43] 111707  - Maximum 200 archives will be processed (soft limit).
INFO      [20:26:43] 111707  - Archiving was last executed without error 4 min 18s ago.
INFO      [20:26:43] 111707  - Will process 1 websites (--force-idsites)
INFO      [20:26:43] 111707  - Will process specified sites: 11
INFO      [20:26:43] 111707  ---------------------------
INFO      [20:26:43] 111707  START
INFO      [20:26:43] 111707  Starting Matomo reports archiving...
INFO      [20:26:43] 111707  0 out of 3 archivers running currently
INFO      [20:26:43] 111707  Start processing archives for site 11.
INFO      [20:26:43] 111707    Will invalidate archived reports for today in site ID = 11's timezone (2024-03-01 00:00:00).
INFO      [20:26:44] 111707    Will invalidate archived reports for yesterday in site ID = 11's timezone (2024-02-29 00:00:00).

THIS IS LOG FROM RUNNING IN COMMAND LINE CLI.
NOTICE THAT IT QUICKLY SHUTS DOWN - TOO MANY ARCHIVERS TO RUN

INFO [20:39:21] 21625 ---------------------------
INFO [20:39:21] 21625 INIT
INFO [20:39:21] 21625 Running Matomo 5.0.1 as Super User
INFO [20:39:21] 21625 ---------------------------
INFO [20:39:21] 21625 NOTES
INFO [20:39:21] 21625 - Async process archiving supported, using CliMulti.
INFO [20:39:21] 21625 - Reports for today will be processed at most every 14400 seconds. You can hange this value in Matomo UI > Settings > General Settings.
INFO [20:39:21] 21625 - Reports for the current week will be processed at most every 28800 seconds. You can change this value in config/config.ini.php by editing 'time_before_week_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Reports for the current month will be processed at most every 43200 seconds. You can change this value in config/config.ini.php by editing 'time_before_month_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Reports for the current year will be processed at most every 86400 seconds. You can change this value in config/config.ini.php by editing 'time_before_year_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Reports for the current range will be processed at most every 2700 seconds. You can change this value in config/config.ini.php by editing 'time_before_range_archive_considered_outdated' in the '[General]' section.
INFO [20:39:21] 21625 - Maximum 200 archives will be processed (soft limit).
INFO [20:39:21] 21625 - Archiving was last executed without error 16 min 56s ago.
INFO [20:39:21] 21625 - Will process 1 websites (--force-idsites)
INFO [20:39:21] 21625 - Will process specified sites: 11
INFO [20:39:21] 21625 ---------------------------
INFO [20:39:21] 21625 START
INFO [20:39:21] 21625 Starting Matomo reports archiving...
INFO [20:39:22] 21625 Archiving will stop now because 14 archivers are already running and max 3 are supposed to run at once.
INFO [20:39:22] 21625 Reached maximum concurrent archivers allowed (3), aborting run.
INFO [20:39:22] 21625 ---------------------------

Validations

atom-box commented 7 months ago

The user had a second Matomo instance, on Matomo 4. But now they report this:

We've had the same behavior on the 2nd instance since the matomo 5 update on Friday night.

mneudert commented 6 months ago

As already mentioned internally we need more details to reproduce the issue.

Just using nohup or running the archiving via crontab should not output "0 out of x archivers". It should always at least find itself and display a positive number (at least 1).

Without more knowledge of the system it may be impossible to reproduce (and fix) it.

atom-box commented 3 months ago

I sent a message to the person who discovered this issue.

atom-box commented 3 months ago

The user found a workaround as follows:

We found a « solution » for this issue : no more problem since we use php-fpm (fpm-fcgi) and no Apache php module (mod_php).

The problem is still there in matomo 5.1 with mod_php

We didn't have this problem before matomo 5 and with mod_php

sgiehl commented 1 month ago

Closing as duplicate of #22508, as the other issue contains more detailed information about the actual problem.