Warn users about possible slow processing when viewing a report that has a lot of rows (for example Pages reports can be slow to render when showing >1K rows)

[peteruniversum]

E.g. https://anyofour3bigmatomoinstances.universum.com**/index.php?module=CoreHome&action=index&date=yesterday&period=day&idSite=2#?period=month&date=2022-12-01&idSite=2&category=General_Actions&subcategory=General_Pages** -> Super slow.

Our customers want yearly reporting, which does absolutely not work in those reports (Pages/Page Titles). Application delivers 73Mbs of data to be rendered in the browser, which of course will not work.

-> Is this to be expected or might there be a config issue on our end?

System Check

`

Click to view System Check

Mandatory checks

PHP version >= 7.2.5:

✔ 8.1.13

PDO extension:

✔

PDO\MYSQL extension:

✔

MYSQLI extension:

✔

Other required extensions:

✔ zlib ✔ json ✔ filter ✔ hash ✔ session

Required functions:

✔ debug_backtrace ✔ eval ✔ hash ✔ gzcompress ✔ gzuncompress ✔ pack

Required PHP configuration (php.ini):

✔ session.auto_start = 0 ✔ max_execution_time = 0 OR = -1 OR >= 30

Directories with write access:

✔ $DOC_ROOT/tmp ✔ $DOC_ROOT/tmp/assets ✔ $DOC_ROOT/tmp/cache ✔ $DOC_ROOT/tmp/climulti ✔ $DOC_ROOT/tmp/latest ✔ $DOC_ROOT/tmp/logs ✔ $DOC_ROOT/tmp/sessions ✔ $DOC_ROOT/tmp/tcpdf ✔ $DOC_ROOT/tmp/templates_c

Directories with write access for Tag Manager:

✔ $DOC_ROOT/js

Optional checks

Required Private Directories:

✔ All private directories are inaccessible from the internet.

Recommended Private Directories:

✔ All private directories are inaccessible from the internet.

File integrity:

⚠ Warning: File integrity check failed and reported some errors. You should fix this issue and then refresh this page until it shows no error.

Files were found in your Matomo, but we didn't expect them.
--> Please delete these files to prevent errors.

64-bit PHP Binary:

✔

Tracker status:

✔

Memory limit:

✔ 2048M

Time zone:

✔

Open URL:

✔ curl

PageSpeed is turned off:

✔

GD > 2.x + FreeType (graphics):

✔

Other extensions:

✔ json ✔ libxml ✔ dom ✔ SimpleXML ✔ openssl

Other functions:

✔ shell_exec ✔ set_time_limit ✔ mail ✔ parse_ini_file ✔ glob ✔ gzopen ✔ md5_file

Filesystem:

✔

Set up Cron - Managing processes via CLI:

✔ Ok

Last Successful Archiving Completion:

✔ The archiving process completed successfully 12:06:37 ago.

Database abilities:

✔ UTF8mb4 charset ✔ LOAD DATA INFILE ✔ CREATE TEMPORARY TABLES ✔ Changing transaction isolation level

Max Packet Size:

✔

Forced SSL Connection:

✔

Geolocation:

⚠ Warning: The default location provider determines the country visitors connect from based on their selected language. This is not very accurate, so install and use a geolocation database.

Update over HTTPS:

✔

Writable JavaScript Tracker ("/matomo.js" & "/piwik.js"):

✔

Informational results

Matomo Version:

4.13.0

Matomo Update History:

4.4.1,4.2.1,4.1.1,3.14.1,

Matomo Install Version:

Unknown - pre 3.8.

Latest Available Version:

4.13.0

Is Git Deployment:

0

PHP_OS:

Linux

PHP_BINARY:

/usr/sbin/php-fpm8.1

PHP SAPI:

fpm-fcgi

Timezone Version:

0.system

PHP Timezone:

UTC

PHP Time:

1672996225

PHP Datetime:

2023-01-06 09:10:25

PHP INI max_execution_time:

300s

PHP INI post_max_size:

10G

PHP INI max_input_vars:

1000

PHP INI zlib.output_compression:

Curl Version:

7.74.0, OpenSSL/1.1.1n

Suhosin Installed:

0

DB Prefix:

matomo_

DB Charset:

utf8mb4

DB Adapter:

PDO\MYSQL

MySQL Version:

10.5.18-MariaDB-0+deb11u1

Num Tables:

393

Browser Segment Archiving Enabled:

0

Development Mode Enabled:

0

Internet Enabled:

1

Multi Server Environment:

0

Auto Update Enabled:

1

Custom User Path:

0

Custom Include Path:

0

Release Channel:

latest_stable

Plugins Activated:

API, Actions, Annotations, BulkTracking, Contents, CoreAdminHome, CoreConsole, CoreHome, CorePluginsAdmin, CoreUpdater, CoreVisualizations, CoreVue, CustomDimensions, CustomJsTracker, CustomVariables 4.1.1, Dashboard, DevicePlugins, DevicesDetection, Diagnostics, Ecommerce, Events, FacebookPageWidgetByAmperage 1.0.7, Feedback, GeoIp2, Goals, Heartbeat, ImageGraph, Insights, Installation, Intl, IntranetMeasurable, LanguagesManager, Live, Login, MarketingCampaignsReporting 4.1.3, Marketplace, MobileMessaging, Monolog, Morpheus, MultiSites, Overlay, PagePerformance, PrivacyManager, ProfessionalServices, Proxy, Referrers, Resolution, RssWidget, SEO, ScheduledReports, SegmentEditor, SitesManager, TagManager, Tour, Transitions, TwoFactorAuth, UserCountry, UserCountryMap, UserId, UserLanguage, UsersManager, VisitFrequency, VisitTime, VisitorInterest, VisitsSummary, WebsiteMeasurable, Widgetize

Plugins Deactivated:

ArchiveSite 0.1.1, DBStats, MobileAppMeasurable, Provider 4.0.5

Plugins Invalid:

Server Info:

Apache

Had visits in last 1 day:

1

Had visits in last 3 days:

1

Had visits in last 5 days:

1

Archive Time Last Started:

2023-01-06 09:00:01

Archive Time Last Finished:

2023-01-05 21:03:48

User Agent:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36

Browser Language:

de,en,nl,it

Total Invalidation Count:

366

In Progress Invalidation Count:

0

Scheduled Invalidation Count:

366

Earliest invalidation ts_started:

Latest invalidation ts_started:

Earliest invalidation ts_invalidated:

2023-01-05 01:00:01

Latest invalidation ts_invalidated:

2023-01-06 04:02:43

Number of segment invalidations:

311

Number of plugin invalidations:

0

List of plugins being invalidated:

Anonymize Referrer:

Do Not Track enabled:

1

`

[bx80]

Hi @peteruniversum,

It should definitely be possible to view yearly page reports. If the browser is being sent report data then it sounds like the archiving has completed successfully and the report data has been generated. However 73MiB is obviously far too much data to be sent to the browser, it could be that there is a config setting cause this.

When Matomo generates reports it used a 'maximum rows' value to show the top x rows, after which all other rows are grouped together into a single row. For sites with a lot of pages this prevents reports being generated with millions of rows.

Could you check the values you have in config.ini.php for the following settings?

By default no report will return more than 50,000 rows:

archiving_ranking_query_row_limit = 50000

Actions (pages) reports will have a maximum of 500 rows:

datatable_archiving_maximum_rows_actions = 500

Each page in the report will have a maximum of 100 sub-pages shown (500 pages x 100 sub-pages = 50,000)

datatable_archiving_maximum_rows_subtable_actions = 100

[ansgarbecker]

I'm not sure if it's the same cause, but we are also experiencing slow report rendering, even with a browser crash. In E-Commerce > Products. Some computers seem to handle it without issue, some just show a loading indicator for a minute before the browser stops working. Chrome e.g. says "out of memory":

Edit: the 3 above mentioned config options are exactly on their defaults: archiving_ranking_query_row_limit = 50000, datatable_archiving_maximum_rows_actions = 500, datatable_archiving_maximum_rows_subtable_actions = 100

[bx80]

Thanks for the extra info @ansgarbecker, Normally reports are paginated to show 5 - 500 rows at a time. Is this issue just occurring when showing all rows?

[ansgarbecker]

That's a good point. I was just trying out the "all" item, which now leads to high memory consumption in my Firefox. I had never touched the pager setting before, so previsouly it was "10", and I had no issue with that. My colleague probably had a higher setting here, so that could be the cause. I already asked her what she did but she could not remember. I'm attempting to find out more and will report back here.

[ansgarbecker]

I just got a reply from my colleague, seems she indeed used the "all" pager. You can likely close the issue. Icing on the cake would be to add a warning about probably slow processing when user wants to use more than 1000 (?) rows per page. For me personally I would have known I clicked there so I would be aware there is probably an issue with loading data. But there seem to be users clicking wildly somewhere and afterwards forgetting what they did...

[mattab]

Thanks @ansgarbecker for creating the issue and your great suggestion to "add a warning about probably slow processing when user wants to use more than 1000 (?) rows per page."

We'll consider doing this indeed :+1:

[ansgarbecker]

Another solution would be to remove the stored filter_limit setting from the server side user preferences in the option table, into a session value. So the user would just have to remove cookies to solve this issue.

Or just not store the -1 (= "all") setting at all, but that would probably annoy the user.