search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.93k stars 2.66k forks source link

The core:invalidate-report-data command is giving an invalid date error when given multiple dates #17146

Closed nciske closed 3 years ago

nciske commented 3 years ago

I'm trying to invalidate the last week of archives so they can be re-run after some bulk updates.

I can get a single date to run just fine, but specifying multiple dates (per this FAQ) barfs out an invalid date error every time.

Example from FAQ: ./console core:invalidate-report-data --dates=2012-01-01,2011-10-15 --sites=1,3,5

Works fine: ./console core:invalidate-report-data --sites=1,2 --dates=2021-01-25

Does not work fine: ./console core:invalidate-report-data --sites=1,2 --dates=2021-01-25,2021-01-24

Results:

Invalidating day periods in 2021-01-25,2021-01-24 [segment = ]...
Invalidating week periods in 2021-01-25,2021-01-24 [segment = ]...
Invalidating month periods in 2021-01-25,2021-01-24 [segment = ]...
Invalidating year periods in 2021-01-25,2021-01-24 [segment = ]...

ERROR [2021-01-25 21:11:36] 157296  Uncaught exception: /{matomo path}/public/core/Period/Range.php(155): Specified date range is invalid.

Specified date range is invalid.
  [Exception]
  Specified date range is invalid.
nciske commented 3 years ago

I've worked around this by putting each date in it's own line of a bash script, but I assume that is less performant than passing multiple dates in one go? Or not?

tsteur commented 3 years ago

does it work to flip the date @nciske ?

./console core:invalidate-report-data --sites=1,2 --dates=2021-01-24,2021-01-25
nciske commented 3 years ago

It does work. Fascinating!

tsteur commented 3 years ago

👍 great, thanks for confirming. I'll adjust the example in the FAQ as it was wrong there. Thanks @nciske

jorgeuos commented 2 years ago

Hi,

I noticed that the --help flag shows that:

--dates                      List of dates or date ranges to invalidate report data for, eg, 2015-01-03 or 2015-01-05,2015-02-12. (multiple values allowed)

But I get this error:

./console core:invalidate-report-data --dates=2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31 --sites=1 --periods=day --plugin=Funnels --dry-run
INFO [2022-01-22 21:07:53] 63288  Invalidating day periods in 2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31 [segment = ]...
ERROR [2022-01-22 21:07:53] 63288  Uncaught exception: /var/www/html/plugins/CoreAdminHome/Commands/InvalidateReportData.php(215): Invalid date or date range specifier '2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31' [Query: , CLI mode: 1]

  [InvalidArgumentException]
  Invalid date or date range specifier '2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2
  021-12-27,2021-12-28,2021-12-29,2021-12-31'

  [Exception]
  Date format must be: YYYY-MM-DD, or 'today' or 'yesterday' or any keyword supported by the strtotime function (see http://php.net/strtotime for more information): 2021-11-
  29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31

core:invalidate-report-data [--dates="..."] [--sites="..."] [--periods="..."] [--segment="..."] [--cascade] [--dry-run] [--plugin="..."] [--ignore-log-deletion-limit]

I've tried without --dry-run too, but get the same error.
I guess single dates per command or a date-range, but not list of dates.
Or am I formatting the command wrong?

Full output of help:

./console core:invalidate-report-data --help
Usage:
 core:invalidate-report-data [--dates="..."] [--sites="..."] [--periods="..."] [--segment="..."] [--cascade] [--dry-run] [--plugin="..."] [--ignore-log-deletion-limit]

Options:
 --dates                      List of dates or date ranges to invalidate report data for, eg, 2015-01-03 or 2015-01-05,2015-02-12. (multiple values allowed)
 --sites                      List of site IDs to invalidate report data for, eg, "1,2,3,4" or "all" for all sites. (default: "all")
 --periods                    List of period types to invalidate report data for. Can be one or more of the following values: day, week, month, year or "all" for all of them. (default: "all")
 --segment                    List of segments to invalidate report data for. This can be the segment string itself, the segment name from the UI or the ID of the segment. If specifying the segment definition, make sure it is encoded properly (it should be the same as the segment parameter in the URL. (multiple values allowed)
 --cascade                    If supplied, invalidation will cascade, invalidating child period types even if they aren't specified in --periods. For example, if --periods=week, --cascade will cause the days within those weeks to be invalidated as well. If --periods=month, then weeks and days will be invalidated. Note: if a period falls partly outside of a date range, then --cascade will also invalidate data for child periods outside the date range. For example, if --dates=2015-09-14,2015-09-15 & --periods=week, --cascade will also invalidate all days within 2015-09-13,2015-09-19, even those outside the date range.
 --dry-run                    For tests. Runs the command w/o actually invalidating anything.
 --plugin                     To invalidate data for a specific plugin only.
 --ignore-log-deletion-limit  Ignore the log purging limit when invalidating archives. If a date is older than the log purging threshold (which means there should be no log data for it), we normally skip invalidating it in order to prevent losing any report data. In some cases, however it is useful, if, for example, your site was imported from Google, and there is never any log data.
 --help (-h)                  Display this help message
 --quiet (-q)                 Do not output any message
 --verbose (-v|vv|vvv)        Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
 --version (-V)               Display this application version
 --ansi                       Force ANSI output
 --no-ansi                    Disable ANSI output
 --no-interaction (-n)        Do not ask any interactive question
 --matomo-domain              Matomo URL (protocol and domain) eg. "http://matomo.example.org"
 --xhprof                     Enable profiling with XHProf

Help:
 Invalidate archived report data by date range, site and period. Invalidated archive data will be re-archived during the next core:archive run. If your log data has changed for some reason, this command can be used to make sure reports are generated using the new, changed log data.
lauxjpn commented 1 year ago

The main misconception here seems to be, that users think that --dates=date1,date2 is a list of individual dates, when in fact, it is interpreted as a date range from date1 to date2 (inclusive), as can be verified by using the --verbose option:

php ./console core:invalidate-report-data --dates=2023-03-30,2023-04-02 --sites=1,2 --verbose

Invalidating day periods in 2023-03-30,2023-04-05 [segment = ]...
Success. The following dates were invalidated successfully: 2023-03-30, 2023-03-31, 2023-04-01, 2023-04-02

For multiple individual dates, use --dates=date1 --dates=date2 --dates=date3:

php ./console core:invalidate-report-data --dates=2023-03-30 --dates=2023-04-02 --sites=1,2 --verbose

Invalidating day periods in 2023-03-30 [segment = ]...
Success. The following dates were invalidated successfully: 2023-03-30
Invalidating day periods in 2023-04-02 [segment = ]...
Success. The following dates were invalidated successfully: 2023-04-02

The docs at How do I tell Matomo to invalidate the past historical reports so they can be re-processed from the logs? don't explain what the syntax in the sample code does, so it is not surprising that it is being misinterpreted by users.

It should be clarified by mentioning both, the --dates=date1,date2 and the --dates=date1 --dates=date2 --dates=date3 syntax.