Generic Calc performance ticket

[mmeeks]

Editing just now on staging with a larger group we got a number of problems with calc performance that should be improved.

Oddly we were getting large memory consumption for a smallish spreadsheet too - perhaps related to clipboads:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32675 cool 20 0 20.003t 2.904g 343088 R 103.0 9.263 6:31.37 kitbroker_012

which is really unexpected.

[mmeeks]

The trace has lots of comedic waste in it: calc-trace-2024-02-08.txt.gz I'll file some tickets for that.

[caolanm]

second half of session

[caolanm]

todays ad-hoc call

[mmeeks]

ScXMLExport::collectAutoStyles looks like something that does far too much work - fouled up by the UNO interface there too I guess; 11% of the profile and ~half of saving time; @grandinj did you look at that call-site ever? =)

[caolanm]

various profiles of todays calc session

[mmeeks]

Filed https://github.com/CollaboraOnline/online/issues/8275 - for the expensive theme color notification thrash, and https://github.com/CollaboraOnline/online/issues/8276 for yet more note export waste.

[caolanm]

flamegraphs from todays ad-hoc calc test, strongly dominated by chart related activity

[caolanm]

todays calc profile

[caolanm]

todays ad-hoc calc session

[mmeeks]

Hmm; the previous profile is back to that huge allocation for ScTableInfo::ScTableInfo - which is odd; I thought we clobbered that, and ScBoundsProvider::Compute seems very hungry. This last one - is still plagued by the ScColumn::HasCellNotes expense @kohei-us wondered if you've seen any of these ? =)

Seems checking our note-storage for emptiness with: if (maCellNotes.block_size() == 1 && maCellNotes.begin()->type == sc::element_type_empty) return false; // all elements are empty is rather more expensive than expected around the .begin()->type - there is a perhaps faster empty() method there but - I guess the multi_type_vector always has one element which is 'empty' is that a good idea (?) =) Thanks!

[caolanm]

What we ended up with for ScTableInfo was to make it a hint anywhere we "traditionally" defaulted to 1024 rows before https://gerrit.libreoffice.org/c/core/+/162277 so that after https://gerrit.libreoffice.org/c/core/+/162875 we maxed at 1024 if 1024 was what we used before.

But for cases like https://gerrit.libreoffice.org/c/core/+/162277/2/sc/source/ui/view/gridwin4.cxx#b1667 where we had an explicit size passed in then we continue to use an explicit requested size (i.e. hint is false). So I think this case is the above line where we "always" used an explicit large range (and this is drawing a range with all rows hidden from some point downwards I think). Not sure if there is an obvious calculation for what the max number of visible rows can be drawn there is, or if its safe to stick some reasonable by largish max there too for that explicit rows requested case.

[kohei-us]

Hmm; the previous profile is back to that huge allocation for ScTableInfo::ScTableInfo - which is odd; I thought we clobbered that, and ScBoundsProvider::Compute seems very hungry. This last one - is still plagued by the ScColumn::HasCellNotes expense @kohei-us wondered if you've seen any of these ? =)

I haven't profiled Calc in years! But maybe this will help: https://gerrit.libreoffice.org/c/core/+/163996

[caolanm]

todays calc profile (dominated by chart drawing)

[grandinj]

On 2/29/2024 2:09 PM, Caolán McNamara wrote:

todays calc profile (dominated by chart drawing)

There is a still a chunk of memset in RenderTiles::doRender that looks unnecessary, but last time I tried to get rid of it, unit tests broke in weird places.

[grandinj]

What we ended up with for ScTableInfo was to make it a hint anywhere we "traditionally" defaulted to 1024 rows before https://gerrit.libreoffice.org/c/core/+/162277 so that after https://gerrit.libreoffice.org/c/core/+/162875 we maxed at 1024 if 1024 was what we used before.

Possibly we should ScTableInfo use a std::vector instead of a fixed array length, and adjust size as needed. Then we can pass in a hint like I did initially (and the hint will be wrong if there are hidden rows), but it will just adjust size as needed. That will be slower for some cases (with hidden rows), but faster for all other cases.

[caolanm]

ad-hoc calc session with lots of tab-moving

[caolanm]

This weeks calc profile

[caolanm]

todays profile, I imagine the calendar swapping is not typical workload

[caolanm]

I'll put calc 24.04 flamegraphs at: https://github.com/CollaboraOnline/online/issues/8570

[caolanm]

Todays 23.05 flamegraph. There looks to be a lot of chart updating, which fits with what we saw visually wrt auto-updating chart from random numbers. And there is a lot of calendar switching, (Gregorian to Hebrew?) which I imagine is not normal workload.

[caolanm]

A follow up session on a document without constant chart updates or unusual number format

[mmeeks]

20% of the time in ScTable::FindMaxRotCol - is not a great score ;-) ( aggregated across many different callers manually - 4.8%+5.2%+5.5%+1.3%+3.3% etc.)

[caolanm]

calc 23.05 profile ah hoc testing

[caolanm]

2nd half of calc 23.05 session

[caolanm]

and follow up calc 23.05 test without copy and paste which was more performant