Live.getLastVisitsDetails TSV export can have thousands of columns

[mattab]

Live.getLastVisitsDetails TSV export can have thousands of columns (for example when some users have 500 actions in the visit, and each action has 10 custom dimensions there can quickly be 5000+ columns.) which is problematic because Libreoffice calc or Excel will only show so many columns (there is a limit around 2000 or so).

This limitation wouldn't be a problem so much, but the issue is that in the data export, the "Visitor columns" such as

Reproduce

• URL in TSV (and equivalent in XML)

• What we get: most of the visitor columns (full list below) are hidden, because actionDetails list of actions appears early in the list of visit attributes, and it can have hundreds/thousands of columns.

• Expected instead: the visitor details to be all visible, and the actionDetails to be truncated by spreadsheet when the export has too many columns

Click to see Dimensions cut off and not visible are these ones after `actionDetails`

``` 0 USD $ 2019-04-25 5 1556169886 2019-04-25 05:24:46 Divezone 1556169886 1556169886 07:24:46 Thursday, April 25, 2019 Thursday, April 25, 2019 07:24:46 returning plugins/Live/images/returningVisitor.png 0 2 none 463 0 0 0s 0 1 1 search Search Engines Google Keyword not defined 2 https://www.google.de/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0ahUKEwiUwMGbj-LbAhXP2qQKHRTpCfYQFgg-MAE&url=https://divezone.net/diving/philippines&usg=AOvVaw15iOAo7a0OvKQIY-1YBM-p http://google.com plugins/Morpheus/icons/dist/searchEngines/google.com.png Unknown Desktop plugins/Morpheus/icons/dist/devices/desktop.png Unknown Generic Desktop Windows 7 Windows plugins/Morpheus/icons/dist/os/WIN.png WIN 7 Trident Trident (IE) Internet Explorer 11.0 Internet Explorer plugins/Morpheus/icons/dist/browsers/IE.png IE 11.0 0.00 0 0 0.00 0 0 0 Europe eur Denmark dk plugins/Morpheus/icons/dist/flags/dk.png Syddanmark 83 Sønderborg Sønderborg, Syddanmark, Denmark 54.909000 9.807000 13:18:38 13 309 type diving location Philippines diving-rating 3.7 1280x1024 cookie, flash, java plugins/Morpheus/icons/dist/plugins/cookie.png cookie plugins/Morpheus/icons/dist/plugins/flash.png flash plugins/Morpheus/icons/dist/plugins/java.png java 9 AffiliateLinkEffectiveness 0 Original 10 IDivedThereLinkMove 0 Original 13 DivingPageTitles 27 Plain guest 0 ```

To solve this problem maybe we could move actionDetails to the last position in each visit in the API output?

[mattab]

This could also be worked around by manually adding the hideColumns parameter for example to the following value:

&hideColumns=dimension1,dimension2,dimension3,dimension4,dimension5,dimension6,dimension7,dimension8,dimension9,dimension10,dimension11,dimension12,dimension13,pageIdAction,idpageview,serverTimePretty,pageId,eventCategory,eventAction,timeSpent,interactionPosition,timestamp,icon

But currently hideColumns is not applied recursively (will be done in Matomo 4 in https://github.com/matomo-org/matomo/pull/11115)

[fdellwing]

which is problematic because Libreoffice calc or Excel will only show so many columns (there is a limit around 2000 or so)

The limit for Excel is 1,048,576 rows by 16,384 columns. For LO it is currently 1,048,576 rows by 1,024 (!!) columns.

---

LO is actively working on increasing this limit. With LO 6.3.0 the column count will be dynamic and if everything works fine, the limit will be increased afterwards.

Keep an eye on https://bugs.documentfoundation.org/show_bug.cgi?id=50916 for more information in the future.

[tsteur]

adding hideColumns=actionDetails works for me. Closing this issue as it seems it'll be fixed in LO eventually and by the time this becomes a priority it will be fixed in LO very likely.