matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.66k stars 2.62k forks source link

[Bug] Incorrect archive recreation #22024

Open Gul-liver opened 6 months ago

Gul-liver commented 6 months ago

What happened?

Hi,

With a matomo hosted on a server I administer, I have a problem recreating the statistics when I want to delete a visit (which is not significant). I've opened a forum post; [https://forum.matomo.org/t/stats-de-performance-non-recreees/55766/1](forum post)

In short, when I delete a visit with the tool present in the "administration" >> "PrivacyManager" >> "gdprTools" section

The visit is indeed deleted in "Visitors", but it's still taken into account in many statistics (hours, geographical origin, device...) And in the "performance" menu, where you'll find access times, DOM generation times, etc., the deleted visit still has an impact. The website concerned has few visits, so it's easy to see this concern.

What should happen?

When one or more visits are deleted, they should no longer have an impact on the statistics.

How can this be reproduced?

Click on "Admin" >> RGPD tools" >> Search for a visite (with its ID) >> Delete it

./console core:invalidate-report-data --dates=2024-03-10 --sites=3

./console core:archive --force-all-websites --url=https://url-du-site-matomo/

Matomo version

5.0.3

PHP version

8.1.27

Server operating system

Linux - Litespeed (no cache !)

What browsers are you seeing the problem on?

Firefox

Computer operating system

Linux

Relevant log output

./console core:invalidate-report-data --dates=2024-03-10 --sites=3

ESC[32mINFO      [2024-03-15 10:56:52] 624290 ESC[39m Invalidating day periods in 2024-03-10 [segment = ]...
ESC[32mINFO      [2024-03-15 10:56:52] 624290 ESC[39m Invalidating week periods in 2024-03-10 [segment = ]...
ESC[32mINFO      [2024-03-15 10:56:52] 624290 ESC[39m Invalidating month periods in 2024-03-10 [segment = ]...
ESC[32mINFO      [2024-03-15 10:56:52] 624290 ESC[39m Invalidating year periods in 2024-03-10 [segment = ].

./console core:archive --force-all-websites --url=https://url-du-site-matomo/ 

ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m ---------------------------
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m INIT
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m Running Matomo 5.0.3 as Super User
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m ---------------------------
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m NOTES
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m - Async process archiving supported, using CliMulti.
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m - Reports for today will be processed at most every 1
800 seconds. You can change this value in Matomo UI > Settings > General Settings.
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m - Archiving was last executed without error 1 hours 5
6 min ago.
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m - Will process all 2 websites
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m ---------------------------
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m START
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m Starting Matomo reports archiving...
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m 1 out of 3 archivers running currently
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m Start processing archives for site 1.
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m Finished archiving for site 1, 0 API requests, Time e
lapsed: 0.024s [1 / 2 done]
ESC[32mINFO      [2024-03-15 11:01:25] 628826 ESC[39m Start processing archives for site 3.
ESC[32mINFO      [2024-03-15 11:01:26] 628826 ESC[39m Archived website id 3, period = range, date = 2024-03
-09,2024-03-15, segment = '', 5 visits found. Time elapsed: 0.629s
ESC[32mINFO      [2024-03-15 11:01:27] 628826 ESC[39m Archived website id 3, period = range, date = 2024-03
-09,2024-03-15, segment = 'campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023', 0 visits found.
:
ESC[32mINFO      [2024-03-15 11:01:27] 628826 ESC[39m Archived website id 3, period = range, date = 2024-03-09,2024-03-15, segment = 'campaignName==cmp-bjh-ete-2022', 0 visits found. Time elapsed: 1.090s
ESC[32mINFO      [2024-03-15 11:01:27] 628826 ESC[39m Archived website id 3, period = range, date = 2024-03-09,2024-03-15, segment = 'campaignName==news-st-valentin-2023', 0 visits found. Time elapsed: 1.090s
ESC[32mINFO      [2024-03-15 11:01:28] 628826 ESC[39m Archived website id 3, period = range, date = 2024-03-09,2024-03-15, segment = 'referrerType==search', 4 visits found. Time elapsed: 0.820s
ESC[32mINFO      [2024-03-15 11:01:28] 628826 ESC[39m Archived website id 3, period = range, date = 2024-03-09,2024-03-15, segment = 'referrerName==Instagram', 0 visits found. Time elapsed: 0.820s
ESC[32mINFO      [2024-03-15 11:01:28] 628826 ESC[39m Archived website id 3, period = week, date = 2024-03-04, segment = '', 15 visits found. Time elapsed: 0.641s
ESC[32mINFO      [2024-03-15 11:01:30] 628826 ESC[39m Archived website id 3, period = week, date = 2024-03-04, segment = 'campaignName==news-st-valentin-2023', 0 visits found. Time elapsed: 1.158s
ESC[32mINFO      [2024-03-15 11:01:30] 628826 ESC[39m Archived website id 3, period = week, date = 2024-03-04, segment = 'referrerName==Instagram', 0 visits found. Time elapsed: 1.158s
ESC[32mINFO      [2024-03-15 11:01:30] 628826 ESC[39m Archived website id 3, period = week, date = 2024-03-04, segment = 'campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023', 0 visits found. Time elapsed: 1.158s
ESC[32mINFO      [2024-03-15 11:01:30] 628826 ESC[39m Archived website id 3, period = week, date = 2024-03-04, segment = 'referrerType==search', 9 visits found. Time elapsed: 0.772s
ESC[32mINFO      [2024-03-15 11:01:30] 628826 ESC[39m Archived website id 3, period = week, date = 2024-03-04, segment = 'campaignName==cmp-bjh-ete-2022', 0 visits found. Time elapsed: 0.772s
ESC[32mINFO      [2024-03-15 11:01:31] 628826 ESC[39m Archived website id 3, period = month, date = 2024-03-01, segment = '', 20 visits found. Time elapsed: 0.746s
:
ESC[32mINFO      [2024-03-15 11:01:32] 628826 ESC[39m Archived website id 3, period = month, date = 2024-03
-01, segment = 'referrerName==Instagram', 0 visits found. Time elapsed: 1.096s
ESC[32mINFO      [2024-03-15 11:01:32] 628826 ESC[39m Archived website id 3, period = month, date = 2024-03
-01, segment = 'campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023', 0 visits found. Time elapse
d: 1.096s
ESC[32mINFO      [2024-03-15 11:01:34] 628826 ESC[39m Archived website id 3, period = month, date = 2024-03
-01, segment = 'referrerType==search', 13 visits found. Time elapsed: 1.105s
ESC[32mINFO      [2024-03-15 11:01:34] 628826 ESC[39m Archived website id 3, period = month, date = 2024-03-01, segment = 'campaignName==cmp-bjh-ete-2022', 0 visits found. Time elapsed: 1.105s
ESC[32mINFO      [2024-03-15 11:01:34] 628826 ESC[39m Archived website id 3, period = year, date = 2024-01-01, segment = '', 111 visits found. Time elapsed: 0.747s
ESC[32mINFO      [2024-03-15 11:01:36] 628826 ESC[39m Archived website id 3, period = year, date = 2024-01-01, segment = 'campaignName==news-st-valentin-2023', 0 visits found. Time elapsed: 1.086s
ESC[32mINFO      [2024-03-15 11:01:36] 628826 ESC[39m Archived website id 3, period = year, date = 2024-01-01, segment = 'referrerName==Instagram', 0 visits found. Time elapsed: 1.086s
ESC[32mINFO      [2024-03-15 11:01:36] 628826 ESC[39m Archived website id 3, period = year, date = 2024-01-01, segment = 'campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023', 0 visits found. Time elapsed: 1.086s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Archived website id 3, period = year, date = 2024-01-01, segment = 'referrerType==search', 78 visits found. Time elapsed: 1.067s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Archived website id 3, period = year, date = 2024-01-01, segment = 'campaignName==cmp-bjh-ete-2022', 2 visits found. Time elapsed: 1.067s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Finished archiving for site 3, 24 API requests, Time:
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Done archiving!
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m ---------------------------
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m SUMMARY
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Processed 24 archives.
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Total API requests: 24
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m done: 24 req, 11876 ms, no error
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Time elapsed: 11.876s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m ---------------------------
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m SCHEDULED TASKS
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Starting Scheduled tasks...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\CustomJsTracker\Tasks.updateTracker...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.002s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\TagManager\Tasks.regenerateReleasedContainers...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.230s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.deleteLogData...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.000s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.anonymizePastData...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.001s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m done
:
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Done archiving!
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m ---------------------------
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m SUMMARY
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Processed 24 archives.
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Total API requests: 24
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m done: 24 req, 11876 ms, no error
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Time elapsed: 11.876s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m ---------------------------
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m SCHEDULED TASKS
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Starting Scheduled tasks...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\CustomJsTracker\Tasks.updateTracker...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.002s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\TagManager\Tasks.regenerateReleasedContainers...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.230s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.deleteLogData...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.000s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.anonymizePastData...
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m Scheduler: finished. Time elapsed: 0.001s
ESC[32mINFO      [2024-03-15 11:01:37] 628826 ESC[39m done
:

Validations

mneudert commented 6 months ago

Hi @Gul-liver,

I think I can reproduce this problem. It seems that, after you delete every visit for a day, the archives are properly flagged for recreation. But, as no visit for that day (or segment) exists anymore, no new archive will be created to replace the old one.

To verify this assumption, could you run run archiving with "debug" output and check if a specific line is written?

# add -vvv to your archive command and all parameters you also need
./console core:archive -vvv

This may generate a LOT of output, so you may want to limit the archiving to a narrow date range that is displaying a wrong report for your site (e.g. using --force-date-range=2024-03-09,2024-03-11).

You should receive output similar to this:

...
INFO [2024-03-21 23:17:38] 5347  START
INFO [2024-03-21 23:17:38] 5347  Starting Matomo reports archiving...
DEBUG [2024-03-21 23:17:38] 5347  Applying queued rearchiving...
INFO [2024-03-21 23:17:38] 5347  Start processing archives for site 1.
DEBUG [2024-03-21 23:17:38] 5347  Checking for queued invalidations...
...
DEBUG [2024-03-21 23:17:38] 5347  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:17:38).
DEBUG [2024-03-21 23:17:38] 5347  Found invalidated archive we can skip (no visits): [idinvalidation = 489, idsite = 1, period = day(2024-03-21 - 2024-03-21), name = done, segment = ]
DEBUG [2024-03-21 23:17:38] 5347  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:17:38).
DEBUG [2024-03-21 23:17:38] 5347  Found invalidated archive we can skip (no visits): [idinvalidation = 458, idsite = 1, period = day(2024-03-21 - 2024-03-21), name = done, segment = ]
...

If my assumption is correct, you should find a row with "DEBUG ... Found invalidated archive we can skip ..." for the days with the outdated reports.

Gul-liver commented 6 months ago

Hi mmeudert,

Thanks a lot for your answer !

So I launched:

./console core:archive -vvv --force-all-websites --url=https://url-du-site-matomo/ --force-date-range=2024-03-09,2024-03-11 >./log-archivage-verbose-pb-10-MARS.txt

And this log contains (concerned website is "site 3"):

INFO [2024-03-21 23:31:48] 3449008  --------------------------- INFO [2024-03-21 23:31:48] 3449008  INIT INFO [2024-03-21 23:31:48] 3449008  Running Matomo 5.0.3 as Super User INFO [2024-03-21 23:31:48] 3449008  --------------------------- INFO [2024-03-21 23:31:48] 3449008  NOTES INFO [2024-03-21 23:31:48] 3449008  - Async process archiving supported, using CliMulti. INFO [2024-03-21 23:31:48] 3449008  - Reports for today will be processed at most every 1800 seconds. You can change this value in Matomo UI > Settings > General Settings. INFO [2024-03-21 23:31:48] 3449008  - Archiving was last executed without error 26 min 42s ago. INFO [2024-03-21 23:31:48] 3449008  - Will process all 2 websites INFO [2024-03-21 23:31:48] 3449008  --------------------------- INFO [2024-03-21 23:31:48] 3449008  START INFO [2024-03-21 23:31:48] 3449008  Starting Matomo reports archiving... INFO [2024-03-21 23:31:48] 3449008  1 out of 3 archivers running currently DEBUG [2024-03-21 23:31:48] 3449008  Applying queued rearchiving... INFO [2024-03-21 23:31:48] 3449008  Start processing archives for site 1. DEBUG [2024-03-21 23:31:48] 3449008  Checking for queued invalidations... DEBUG [2024-03-21 23:31:48] 3449008  Today archive can be skipped due to no visits for idSite = 1, skipping invalidation... DEBUG [2024-03-21 23:31:48] 3449008  Yesterday archive can be skipped due to no visits for idSite = 1, skipping invalidation... DEBUG [2024-03-21 23:31:48] 3449008  Done invalidating DEBUG [2024-03-21 23:31:48] 3449008  No next invalidated archive. INFO [2024-03-21 23:31:48] 3449008  Finished archiving for site 1, 0 API requests, Time elapsed: 0.027s [1 / 2 done] INFO [2024-03-21 23:31:48] 3449008  Start processing archives for site 3. DEBUG [2024-03-21 23:31:48] 3449008  Checking for queued invalidations... DEBUG [2024-03-21 23:31:48] 3449008  Today archive can be skipped due to no visits for idSite = 3, skipping invalidation... INFO [2024-03-21 23:31:48] 3449008  Will invalidate archived reports for yesterday in site ID = 3's timezone (2024-03-21 00:00:00). DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = , plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==cmp-bjh-ete-2022, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerType==search, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==news-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerName==Instagram, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Invalidating custom date range (last7) for site 3 DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = , plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = campaignName==cmp-bjh-ete-2022, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = referrerType==search, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = campaignName==news-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = referrerName==Instagram, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Done invalidating DEBUG [2024-03-21 23:31:49] 3449008  No next invalidated archive. INFO [2024-03-21 23:31:49] 3449008  Finished archiving for site 3, 0 API requests, Time elapsed: 0.442s [2 / 2 done] DEBUG [2024-03-21 23:31:49] 3449008  No more sites left to archive, stopping. INFO [2024-03-21 23:31:49] 3449008  Done archiving! INFO [2024-03-21 23:31:49] 3449008  --------------------------- INFO [2024-03-21 23:31:49] 3449008  SUMMARY INFO [2024-03-21 23:31:49] 3449008  Processed 0 archives. INFO [2024-03-21 23:31:49] 3449008  Total API requests: 0 INFO [2024-03-21 23:31:49] 3449008  done: 0 req, 488 ms, no error INFO [2024-03-21 23:31:49] 3449008  Time elapsed: 0.488s INFO [2024-03-21 23:31:49] 3449008  --------------------------- INFO [2024-03-21 23:31:49] 3449008  SCHEDULED TASKS DEBUG [2024-03-21 23:31:49] 3449008  27 scheduled tasks loaded INFO [2024-03-21 23:31:49] 3449008  Starting Scheduled tasks... DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 0: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 1: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 2: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 3: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 4: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 5: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 6: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 7: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 8: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 9: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 10: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 11: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 12: INFO [2024-03-21 23:31:49] 3449008  done INFO [2024-03-21 23:31:49] 3449008  ---------------------------

I've got the same issue for 2024-03-15: just one visit, a "bad visit", that I can't nomore delete completely :+1: No visit, but country stays recorded, hour range, and unfortunately "performance" (which are very poor, so statistics are really distorted).

So I tried for this day:

./console core:archive -vvv --force-idsites=3 --url=https://url-du-site-matomo/ --force-date-range=2024-03-14,2024-03-16 >./log-archivage-verbose-pb-15-MARS.txt

And log is:

INFO [2024-03-21 23:56:16] 3465615  --------------------------- INFO [2024-03-21 23:56:16] 3465615  INIT INFO [2024-03-21 23:56:16] 3465615  Running Matomo 5.0.3 as Super User INFO [2024-03-21 23:56:16] 3465615  --------------------------- INFO [2024-03-21 23:56:16] 3465615  NOTES INFO [2024-03-21 23:56:16] 3465615  - Async process archiving supported, using CliMulti. INFO [2024-03-21 23:56:16] 3465615  - Reports for today will be processed at most every 1800 seconds. You can change this value in Matomo UI > Settings > General Settings. INFO [2024-03-21 23:56:16] 3465615  - Archiving was last executed without error 24 min 27s ago. INFO [2024-03-21 23:56:16] 3465615  - Will process 1 websites (--force-idsites) INFO [2024-03-21 23:56:16] 3465615  - Will process specified sites: 3 INFO [2024-03-21 23:56:16] 3465615  --------------------------- INFO [2024-03-21 23:56:16] 3465615  START INFO [2024-03-21 23:56:16] 3465615  Starting Matomo reports archiving... INFO [2024-03-21 23:56:16] 3465615  1 out of 3 archivers running currently DEBUG [2024-03-21 23:56:16] 3465615  Applying queued rearchiving... INFO [2024-03-21 23:56:16] 3465615  Start processing archives for site 3. DEBUG [2024-03-21 23:56:16] 3465615  Checking for queued invalidations... DEBUG [2024-03-21 23:56:16] 3465615  Today archive can be skipped due to no visits for idSite = 3, skipping invalidation... INFO [2024-03-21 23:56:16] 3465615  Will invalidate archived reports for yesterday in site ID = 3's timezone (2024-03-21 00:00:00). DEBUG [2024-03-21 23:56:16] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = , plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:16] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==cmp-bjh-ete-2022, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:16] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerType==search, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==news-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerName==Instagram, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Invalidating custom date range (last7) for site 3 DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Done invalidating DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140560, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done, segment = ]. DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140565, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done3d32f56c8acbf157c19f17e770f65aa6, segment = campaignName==cmp-bjh-ete-2022] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140564, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donec24ce145a5f671ec1bdf052b72d54a0d, segment = campaignName==news-st-valentin-2023] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140563, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = doneb97ac64c174a9fe28a80ee071b64383b, segment = campaignSource==goo%253Fmtm_campaign%253Dcmp-bjh-st-valentin-2023] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140562, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donebf2a28dff7e55ab332fdae3b175f301c, segment = referrerName==Instagram] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140561, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done9eb972afd97a94801f1c96c18d9b492b, segment = referrerType==search] DEBUG [2024-03-21 23:56:17] 3465615  No next invalidated archive. DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&trigger=archivephp' 2>&1 INFO [2024-03-21 23:56:17] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = '', 71 visits found. Time elapsed: 0.586s DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140565, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done3d32f56c8acbf157c19f17e770f65aa6, segment = campaignName==cmp-bjh-ete-2022]. DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140564, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donec24ce145a5f671ec1bdf052b72d54a0d, segment = campaignName==news-st-valentin-2023]. DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140563, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = doneb97ac64c174a9fe28a80ee071b64383b, segment = campaignSource==goo%253Fmtm_campaign%253Dcmp-bjh-st-valentin-2023]. DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dcmp-bjh-ete-2022&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dnews-st-valentin-2023&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignSource%3D%3Dgoo%25253Fmtm_campaign%25253Dcmp-bjh-st-valentin-2023&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dcmp-bjh-ete-2022&trigger=archivephp&pid=80713566a39b24491bdaa1b708071b8104cff96f00a4ee7a4ea02ad9f3060cc430c022f3e3cabc7608798d9a250dcfff6d760&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/80713566a39b24491bdaa1b708071b8104cff96f00a4ee7a4ea02ad9f3060cc430c022f3e3cabc7608798d9a250dcfff6d760.output 2>&1 & DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dnews-st-valentin-2023&trigger=archivephp&pid=e76d425330497ab20ea80ab91d1bccfd6f4609438e42ecdc1d39456c31aa7fb8a394a96fc8d07f51eee730ddd9ecbeb773821&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/e76d425330497ab20ea80ab91d1bccfd6f4609438e42ecdc1d39456c31aa7fb8a394a96fc8d07f51eee730ddd9ecbeb773821.output 2>&1 & DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignSource%3D%3Dgoo%25253Fmtm_campaign%25253Dcmp-bjh-st-valentin-2023&trigger=archivephp&pid=80bf531ceaa71f5a1e68553f499e851a8054f5cc03c7093eec87ba32adbb421cc4d91e787c508a74ccb3384fbad7967864b12&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/80bf531ceaa71f5a1e68553f499e851a8054f5cc03c7093eec87ba32adbb421cc4d91e787c508a74ccb3384fbad7967864b12.output 2>&1 & INFO [2024-03-21 23:56:18] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'campaignName==cmp-bjh-ete-2022', 0 visits found. Time elapsed: 1.098s INFO [2024-03-21 23:56:18] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'campaignName==news-st-valentin-2023', 0 visits found. Time elapsed: 1.098s INFO [2024-03-21 23:56:18] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023', 0 visits found. Time elapsed: 1.098s DEBUG [2024-03-21 23:56:18] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:18). DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Processing invalidation: [idinvalidation = 140562, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donebf2a28dff7e55ab332fdae3b175f301c, segment = referrerName==Instagram]. DEBUG [2024-03-21 23:56:18] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:18). DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Processing invalidation: [idinvalidation = 140561, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done9eb972afd97a94801f1c96c18d9b492b, segment = referrerType==search]. DEBUG [2024-03-21 23:56:18] 3465615  No next invalidated archive. DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerName%3D%3DInstagram&trigger=archivephp DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerType%3D%3Dsearch&trigger=archivephp DEBUG [2024-03-21 23:56:18] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerName%3D%3DInstagram&trigger=archivephp&pid=6b34ef285b78d851ef304cf716a43ba043a1a1628d74750a223ae07ce4e68bdb8eb7028af03babaa8663bcb45100d1601be50&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/6b34ef285b78d851ef304cf716a43ba043a1a1628d74750a223ae07ce4e68bdb8eb7028af03babaa8663bcb45100d1601be50.output 2>&1 & DEBUG [2024-03-21 23:56:18] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerType%3D%3Dsearch&trigger=archivephp&pid=896fbbd1d582e3ec07d30bb2a0339a9e49b8e4aa14a0b8135d5fcfcce029178eeaafc6b45484136f3811f2c575e182362ed21&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/896fbbd1d582e3ec07d30bb2a0339a9e49b8e4aa14a0b8135d5fcfcce029178eeaafc6b45484136f3811f2c575e182362ed21.output 2>&1 & INFO [2024-03-21 23:56:20] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'referrerName==Instagram', 0 visits found. Time elapsed: 0.777s INFO [2024-03-21 23:56:20] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'referrerType==search', 4 visits found. Time elapsed: 1.385s DEBUG [2024-03-21 23:56:20] 3465615  No next invalidated archive. INFO [2024-03-21 23:56:20] 3465615  Finished archiving for site 3, 6 API requests, Time elapsed: 3.541s [1 / 1 done] DEBUG [2024-03-21 23:56:20] 3465615  No more sites left to archive, stopping. INFO [2024-03-21 23:56:20] 3465615  Done archiving! INFO [2024-03-21 23:56:20] 3465615  --------------------------- INFO [2024-03-21 23:56:20] 3465615  SUMMARY INFO [2024-03-21 23:56:20] 3465615  Processed 6 archives. INFO [2024-03-21 23:56:20] 3465615  Total API requests: 6 INFO [2024-03-21 23:56:20] 3465615  done: 6 req, 3559 ms, no error INFO [2024-03-21 23:56:20] 3465615  Time elapsed: 3.559s INFO [2024-03-21 23:56:20] 3465615  --------------------------- INFO [2024-03-21 23:56:20] 3465615  SCHEDULED TASKS DEBUG [2024-03-21 23:56:20] 3465615  27 scheduled tasks loaded INFO [2024-03-21 23:56:20] 3465615  Starting Scheduled tasks... DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 0: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 1: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 2: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 3: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 4: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 5: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 6: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 7: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 8: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 9: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 10: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 11: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 12: INFO [2024-03-21 23:56:20] 3465615  done INFO [2024-03-21 23:56:20] 3465615  ---------------------------

Does it validate what you thought about ?

Anyway, thanks a lot !

Best regards

Gul-liver commented 6 months ago

Sorry for this bad format ... I shouldn't have formatted this as "code"..

So, log for 10 Mars:

INFO [2024-03-21 23:31:48] 3449008  --------------------------- INFO [2024-03-21 23:31:48] 3449008  INIT INFO [2024-03-21 23:31:48] 3449008  Running Matomo 5.0.3 as Super User INFO [2024-03-21 23:31:48] 3449008  --------------------------- INFO [2024-03-21 23:31:48] 3449008  NOTES INFO [2024-03-21 23:31:48] 3449008  - Async process archiving supported, using CliMulti. INFO [2024-03-21 23:31:48] 3449008  - Reports for today will be processed at most every 1800 seconds. You can change this value in Matomo UI > Settings > General Settings. INFO [2024-03-21 23:31:48] 3449008  - Archiving was last executed without error 26 min 42s ago. INFO [2024-03-21 23:31:48] 3449008  - Will process all 2 websites INFO [2024-03-21 23:31:48] 3449008  --------------------------- INFO [2024-03-21 23:31:48] 3449008  START INFO [2024-03-21 23:31:48] 3449008  Starting Matomo reports archiving... INFO [2024-03-21 23:31:48] 3449008  1 out of 3 archivers running currently DEBUG [2024-03-21 23:31:48] 3449008  Applying queued rearchiving... INFO [2024-03-21 23:31:48] 3449008  Start processing archives for site 1. DEBUG [2024-03-21 23:31:48] 3449008  Checking for queued invalidations... DEBUG [2024-03-21 23:31:48] 3449008  Today archive can be skipped due to no visits for idSite = 1, skipping invalidation... DEBUG [2024-03-21 23:31:48] 3449008  Yesterday archive can be skipped due to no visits for idSite = 1, skipping invalidation... DEBUG [2024-03-21 23:31:48] 3449008  Done invalidating DEBUG [2024-03-21 23:31:48] 3449008  No next invalidated archive. INFO [2024-03-21 23:31:48] 3449008  Finished archiving for site 1, 0 API requests, Time elapsed: 0.027s [1 / 2 done] INFO [2024-03-21 23:31:48] 3449008  Start processing archives for site 3. DEBUG [2024-03-21 23:31:48] 3449008  Checking for queued invalidations... DEBUG [2024-03-21 23:31:48] 3449008  Today archive can be skipped due to no visits for idSite = 3, skipping invalidation... INFO [2024-03-21 23:31:48] 3449008  Will invalidate archived reports for yesterday in site ID = 3's timezone (2024-03-21 00:00:00). DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = , plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==cmp-bjh-ete-2022, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerType==search, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:48] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==news-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerName==Instagram, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Invalidating custom date range (last7) for site 3 DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = , plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = campaignName==cmp-bjh-ete-2022, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = referrerType==search, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = campaignName==news-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Found usable archive for [idSite = 3, period = range 2024-03-15,2024-03-21, segment = referrerName==Instagram, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:31:49] 3449008  Done invalidating DEBUG [2024-03-21 23:31:49] 3449008  No next invalidated archive. INFO [2024-03-21 23:31:49] 3449008  Finished archiving for site 3, 0 API requests, Time elapsed: 0.442s [2 / 2 done] DEBUG [2024-03-21 23:31:49] 3449008  No more sites left to archive, stopping. INFO [2024-03-21 23:31:49] 3449008  Done archiving! INFO [2024-03-21 23:31:49] 3449008  --------------------------- INFO [2024-03-21 23:31:49] 3449008  SUMMARY INFO [2024-03-21 23:31:49] 3449008  Processed 0 archives. INFO [2024-03-21 23:31:49] 3449008  Total API requests: 0 INFO [2024-03-21 23:31:49] 3449008  done: 0 req, 488 ms, no error INFO [2024-03-21 23:31:49] 3449008  Time elapsed: 0.488s INFO [2024-03-21 23:31:49] 3449008  --------------------------- INFO [2024-03-21 23:31:49] 3449008  SCHEDULED TASKS DEBUG [2024-03-21 23:31:49] 3449008  27 scheduled tasks loaded INFO [2024-03-21 23:31:49] 3449008  Starting Scheduled tasks... DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 0: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 1: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 2: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 3: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 4: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 5: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 6: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 7: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 8: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 9: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 10: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 11: DEBUG [2024-03-21 23:31:49] 3449008  Executing tasks with priority 12: INFO [2024-03-21 23:31:49] 3449008  done INFO [2024-03-21 23:31:49] 3449008  ---------------------------

And log for 15 Mars:

INFO [2024-03-21 23:56:16] 3465615  --------------------------- INFO [2024-03-21 23:56:16] 3465615  INIT INFO [2024-03-21 23:56:16] 3465615  Running Matomo 5.0.3 as Super User INFO [2024-03-21 23:56:16] 3465615  --------------------------- INFO [2024-03-21 23:56:16] 3465615  NOTES INFO [2024-03-21 23:56:16] 3465615  - Async process archiving supported, using CliMulti. INFO [2024-03-21 23:56:16] 3465615  - Reports for today will be processed at most every 1800 seconds. You can change this value in Matomo UI > Settings > General Settings. INFO [2024-03-21 23:56:16] 3465615  - Archiving was last executed without error 24 min 27s ago. INFO [2024-03-21 23:56:16] 3465615  - Will process 1 websites (--force-idsites) INFO [2024-03-21 23:56:16] 3465615  - Will process specified sites: 3 INFO [2024-03-21 23:56:16] 3465615  --------------------------- INFO [2024-03-21 23:56:16] 3465615  START INFO [2024-03-21 23:56:16] 3465615  Starting Matomo reports archiving... INFO [2024-03-21 23:56:16] 3465615  1 out of 3 archivers running currently DEBUG [2024-03-21 23:56:16] 3465615  Applying queued rearchiving... INFO [2024-03-21 23:56:16] 3465615  Start processing archives for site 3. DEBUG [2024-03-21 23:56:16] 3465615  Checking for queued invalidations... DEBUG [2024-03-21 23:56:16] 3465615  Today archive can be skipped due to no visits for idSite = 3, skipping invalidation... INFO [2024-03-21 23:56:16] 3465615  Will invalidate archived reports for yesterday in site ID = 3's timezone (2024-03-21 00:00:00). DEBUG [2024-03-21 23:56:16] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = , plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:16] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==cmp-bjh-ete-2022, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:16] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerType==search, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = campaignName==news-st-valentin-2023, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Found usable archive for [idSite = 3, period = day 2024-03-21,2024-03-21, segment = referrerName==Instagram, plugin = , report = ], skipping invalidation. DEBUG [2024-03-21 23:56:17] 3465615  Invalidating custom date range (last7) for site 3 DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Done invalidating DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140560, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done, segment = ]. DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140565, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done3d32f56c8acbf157c19f17e770f65aa6, segment = campaignName==cmp-bjh-ete-2022] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140564, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donec24ce145a5f671ec1bdf052b72d54a0d, segment = campaignName==news-st-valentin-2023] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140563, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = doneb97ac64c174a9fe28a80ee071b64383b, segment = campaignSource==goo%253Fmtm_campaign%253Dcmp-bjh-st-valentin-2023] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140562, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donebf2a28dff7e55ab332fdae3b175f301c, segment = referrerName==Instagram] DEBUG [2024-03-21 23:56:17] 3465615  Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 140561, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done9eb972afd97a94801f1c96c18d9b492b, segment = referrerType==search] DEBUG [2024-03-21 23:56:17] 3465615  No next invalidated archive. DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&trigger=archivephp' 2>&1 INFO [2024-03-21 23:56:17] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = '', 71 visits found. Time elapsed: 0.586s DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140565, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done3d32f56c8acbf157c19f17e770f65aa6, segment = campaignName==cmp-bjh-ete-2022]. DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140564, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donec24ce145a5f671ec1bdf052b72d54a0d, segment = campaignName==news-st-valentin-2023]. DEBUG [2024-03-21 23:56:17] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:17). DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Processing invalidation: [idinvalidation = 140563, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = doneb97ac64c174a9fe28a80ee071b64383b, segment = campaignSource==goo%253Fmtm_campaign%253Dcmp-bjh-st-valentin-2023]. DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dcmp-bjh-ete-2022&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dnews-st-valentin-2023&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:17] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignSource%3D%3Dgoo%25253Fmtm_campaign%25253Dcmp-bjh-st-valentin-2023&trigger=archivephp DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dcmp-bjh-ete-2022&trigger=archivephp&pid=80713566a39b24491bdaa1b708071b8104cff96f00a4ee7a4ea02ad9f3060cc430c022f3e3cabc7608798d9a250dcfff6d760&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/80713566a39b24491bdaa1b708071b8104cff96f00a4ee7a4ea02ad9f3060cc430c022f3e3cabc7608798d9a250dcfff6d760.output 2>&1 & DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignName%3D%3Dnews-st-valentin-2023&trigger=archivephp&pid=e76d425330497ab20ea80ab91d1bccfd6f4609438e42ecdc1d39456c31aa7fb8a394a96fc8d07f51eee730ddd9ecbeb773821&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/e76d425330497ab20ea80ab91d1bccfd6f4609438e42ecdc1d39456c31aa7fb8a394a96fc8d07f51eee730ddd9ecbeb773821.output 2>&1 & DEBUG [2024-03-21 23:56:17] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=campaignSource%3D%3Dgoo%25253Fmtm_campaign%25253Dcmp-bjh-st-valentin-2023&trigger=archivephp&pid=80bf531ceaa71f5a1e68553f499e851a8054f5cc03c7093eec87ba32adbb421cc4d91e787c508a74ccb3384fbad7967864b12&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/80bf531ceaa71f5a1e68553f499e851a8054f5cc03c7093eec87ba32adbb421cc4d91e787c508a74ccb3384fbad7967864b12.output 2>&1 & INFO [2024-03-21 23:56:18] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'campaignName==cmp-bjh-ete-2022', 0 visits found. Time elapsed: 1.098s INFO [2024-03-21 23:56:18] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'campaignName==news-st-valentin-2023', 0 visits found. Time elapsed: 1.098s INFO [2024-03-21 23:56:18] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'campaignSource==goo%3Fmtm_campaign%3Dcmp-bjh-st-valentin-2023', 0 visits found. Time elapsed: 1.098s DEBUG [2024-03-21 23:56:18] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:18). DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Processing invalidation: [idinvalidation = 140562, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = donebf2a28dff7e55ab332fdae3b175f301c, segment = referrerName==Instagram]. DEBUG [2024-03-21 23:56:18] 3465615  No usable archive exists (ts_archived of existing = , now = 2024-03-21 23:56:18). DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Processing invalidation: [idinvalidation = 140561, idsite = 3, period = range(2024-03-15 - 2024-03-21), name = done9eb972afd97a94801f1c96c18d9b492b, segment = referrerType==search]. DEBUG [2024-03-21 23:56:18] 3465615  No next invalidated archive. DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerName%3D%3DInstagram&trigger=archivephp DEBUG [2024-03-21 23:56:18] 3465615  process_new_segments_from set to beginning_of_time or cannot recognize value DEBUG [2024-03-21 23:56:18] 3465615  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerType%3D%3Dsearch&trigger=archivephp DEBUG [2024-03-21 23:56:18] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerName%3D%3DInstagram&trigger=archivephp&pid=6b34ef285b78d851ef304cf716a43ba043a1a1628d74750a223ae07ce4e68bdb8eb7028af03babaa8663bcb45100d1601be50&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/6b34ef285b78d851ef304cf716a43ba043a1a1628d74750a223ae07ce4e68bdb8eb7028af03babaa8663bcb45100d1601be50.output 2>&1 & DEBUG [2024-03-21 23:56:18] 3465615  Running command: /opt/alt/php81/usr/bin/php -q /home/dreamboxorg/data.dream-box.org/console climulti:request -q --matomo-domain='data.dream-box.org' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=range&date=2024-03-15,2024-03-21&format=json&segment=referrerType%3D%3Dsearch&trigger=archivephp&pid=896fbbd1d582e3ec07d30bb2a0339a9e49b8e4aa14a0b8135d5fcfcce029178eeaafc6b45484136f3811f2c575e182362ed21&runid=3465615' > /home/dreamboxorg/data.dream-box.org/tmp/climulti/896fbbd1d582e3ec07d30bb2a0339a9e49b8e4aa14a0b8135d5fcfcce029178eeaafc6b45484136f3811f2c575e182362ed21.output 2>&1 & INFO [2024-03-21 23:56:20] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'referrerName==Instagram', 0 visits found. Time elapsed: 0.777s INFO [2024-03-21 23:56:20] 3465615  Archived website id 3, period = range, date = 2024-03-15,2024-03-21, segment = 'referrerType==search', 4 visits found. Time elapsed: 1.385s DEBUG [2024-03-21 23:56:20] 3465615  No next invalidated archive. INFO [2024-03-21 23:56:20] 3465615  Finished archiving for site 3, 6 API requests, Time elapsed: 3.541s [1 / 1 done] DEBUG [2024-03-21 23:56:20] 3465615  No more sites left to archive, stopping. INFO [2024-03-21 23:56:20] 3465615  Done archiving! INFO [2024-03-21 23:56:20] 3465615  --------------------------- INFO [2024-03-21 23:56:20] 3465615  SUMMARY INFO [2024-03-21 23:56:20] 3465615  Processed 6 archives. INFO [2024-03-21 23:56:20] 3465615  Total API requests: 6 INFO [2024-03-21 23:56:20] 3465615  done: 6 req, 3559 ms, no error INFO [2024-03-21 23:56:20] 3465615  Time elapsed: 3.559s INFO [2024-03-21 23:56:20] 3465615  --------------------------- INFO [2024-03-21 23:56:20] 3465615  SCHEDULED TASKS DEBUG [2024-03-21 23:56:20] 3465615  27 scheduled tasks loaded INFO [2024-03-21 23:56:20] 3465615  Starting Scheduled tasks... DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 0: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 1: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 2: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 3: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 4: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 5: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 6: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 7: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 8: DEBUG [2024-03-21 23:56:20] 3465615  EI shouldn't have formatted this as "code".xecuting tasks with priority 9: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 10: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 11: DEBUG [2024-03-21 23:56:20] 3465615  Executing tasks with priority 12: INFO [2024-03-21 23:56:20] 3465615  done INFO [2024-03-21 23:56:20] 3465615  ---------------------------

mneudert commented 6 months ago

I've got the same issue for 2024-03-15: just one visit, a "bad visit", that I can't nomore delete completely 👍 No visit, but country stays recorded, hour range, and unfortunately "performance" (which are very poor, so statistics are really distorted).

That should confirm my assumption.

Your logs don't show the line I was looking for, but I think you haven't run core:invalidate-report-data for these days beforehand. And I also did not mention is specifically, but we should have enough info to prioritize this for a fix.

Gul-liver commented 6 months ago

Hi mmeudert,

Looking at the bash history, I can tell that I ran "core:invalidate-report-data" for this date, before: ./console core:invalidate-report-data --dates=2024-03-15 --sites=3 > ./log-invalidate-15-MARS-2024.log

But this log is not very verbose:

INFO [2024-03-21 21:41:50] 3370473  Invalidating day periods in 2024-03-15 [segment = ]... INFO [2024-03-21 21:41:50] 3370473  Invalidating week periods in 2024-03-15 [segment = ]... INFO [2024-03-21 21:41:50] 3370473  Invalidating month periods in 2024-03-15 [segment = ]... INFO [2024-03-21 21:41:50] 3370473  Invalidating year periods in 2024-03-15 [segment = ]...

As I consult this history, I realize that I may have made a mistake ... I ran this, after "core:invalidate-report-data":

./console core:delete-logs-data --dates=2024-03-15 --idsite=3

I probably shouldn't have done it... I thought that deleting this data would delete these reports in the GUI. Now I have no idea where Matomo is getting this data from.

Below are a few captures showing significant concern for March 15.

Capture-matomo-5 Capture-matomo-4 Capture-matomo-3 Capture-matomo-2 Capture-matomo-1

Would other logs or database extracts be useful to better understand the problem ?

Thanks ! Best regards

Gul-liver commented 5 months ago

In fact, there's no problem with this procedure when there are still archived visits after deleting one (or more) for a day.