Slow printing of layout if filtered legend contains a layer that has a symbology classified using a virtual attribute

stafoo commented 4 months ago

What is the bug or the crash?

Filtered legends in layouts slow down printing process in QGIS 3.34.5 and 3.36.1. It happens when one layer has a symbology classified using a virtual attribute (such as a symbol showing only elements that are selected via _isselected() expression, as described here: https://github.com/qgis/QGIS/issues/57230#issuecomment-2082696967) This behavios was not observed in QGIS 3.28.10 with the same project.

In the video example below, activating the filter option in legend significantly slows down the print process:

with filtered legend: 30 seconds
without filtered legend: 2 seconds

https://github.com/qgis/QGIS/assets/11771066/5a7d7ced-bf37-4b00-8da5-c1c8f26e37c2

This becomes even more critical with multipage layouts and with projects containing several layers. This problem does not seems to be related to atlas.

Steps to reproduce the issue

create a layer with a virtual attribute (in my case I used is_selected() expression; 1 means the element is selected, 0 not selected)
classify the layer symbology using the virtual attribute
create a layout (with map + legend)
activate the "filtered legend" option, then print the layout to pdf (very slow)
de-activate the "filtered legend" option, print the layout to pdf again (this time you will notice how it is much faster)

Versions

Supported QGIS version

[X] I'm running a supported QGIS version according to the roadmap.

New profile

[X] I tried with a new QGIS profile

stafoo commented 4 months ago

This problem is still present in QGIS 3.34.6-1

stafoo commented 4 months ago

I investigated the error a bit. It seems that the slowness happens only if the filtered legend contains a layer that has a symbology classified using a virtual attribute.

immagine

Making the attribute Selezionato a "real" attribute (and not a virtual one), the printing process is fast even if the legend is filtered.

agiudiceandrea commented 4 months ago

@stafoo thanks for reporting. It would be useful if you provided a sample project and layet(s) with which the issue occurs.

stafoo commented 4 months ago

TestBug57230_project_gpkg.zip

Sample project attached. Steps to reproduce:

open the project, select one element (or more) in layer (the virtual attribute "selected" is now updated with 'true' in the selected elements)
open the print layout and export a pdf (slow print)
uncheck the filter "Only show items inside linked maps" in the legend option, then print (fast print) This test project is made with QGIS 3.34.6

agiudiceandrea commented 4 months ago

I can confirm the issue on Windows using either QGIS 3.36.2 or QGIS 3.34.6. The issue didn't occur using QGIS 3.28.15.

I also see a glitch in the legend's dimensions which incorrectly change when the "Only show items inside linked maps" checkbox is checked /unchecked Video_2024-04-30_123124

stafoo commented 4 months ago

Update on the bug. The slow printing problem is present even if the Legend in the layout does not contain the "layer" object. If you remove "layer" from the objects shown in the legend and you print the layout, it will still be slow (with filtered legend). In addition, Layer must to be visible in the layout map to notice the slow printing.

removing the virtual attribute filled by the expression _isselected(), the printing process is always fast also with the "Only show items inside linked maps" option checked
with the virtual attribute filled by the expression _isselected(), the printing process is slow ONLY if the elements of the Layer are shown in the layout map; the "Only show items inside linked maps" option must be checked
with the virtual attribute filled by the expression _isselected(), the printing process is fast if the elements of the layer are NOT shown in the layout map, even if the "Only show items inside linked maps" option is checked

elpaso commented 3 months ago

I've spent quite some time trying to find a solution but I couldn't.

The issue comes from the hit test task QgsMapHitTestTask which is run in a separate thread by the task manager and gets locked on a semaphore here: https://github.com/qgis/QGIS/blob/master/src/core/expression/qgsexpressionutils.cpp#L179

this happens because the is_selected function like may others is invoked in the layer's thread which happens to be the main thread which is waiting for the hit test to finish.

The 30 seconds come from the if ( mNotFinishedMutex.tryLock( timeout ) ) so this is basically deadlock which eventually times out and by chance continues without errors.

@nyalldawson do you have any recommendation about how to handle this? I thought about cloning the layers into the hit test thread but I'm not sure it is a viable solution.

nyalldawson commented 3 months ago

@elpaso

Hmm, very tricky

The only safe approach I can think would be to have some QgsPreparedExpressionData" which we generate in the main thread when creating that task. And then a bunch of plumbing to get that object passed to the expression at evaluation time (maybe attaching it to the expression context?). And then if the prepared expression data is available, we use it instead of calling the map layer function direct. And then we prepopulate the expression data with values we need for functions like this, storing them in a variant map for thread safety.

stafoo commented 2 months ago

@elpaso I have tested again this bug with this build: https://github.com/qgis/QGIS/pull/57649#issuecomment-2146708384

As noted by @m-kuhn (https://github.com/qgis/QGIS/pull/57649#issuecomment-2147822345) the expression _isselected still slows down the printing process.

My 2 cents

stafoo commented 1 month ago

Bug confirmed with QGIS 3.34.9

qgis / QGIS