Read legend JSON file from file system

[lucadelu]

Is your feature request related to a problem? Please describe

The problem is that the getLegend request to get the icons for the legend is really time consuming (depending on layers number and style complexity it can arrive to take more than 30 seconds) and it is blocking the entire website

Describe the solution you'd like

Create the JSON file within the plugin and save the file with the .qgs and .cfg file and read from it

Describe alternatives you've considered

Put a "loader" that not permit to use the client and set a sentence like "we are loading layer and styles for a better experience" until the getlegend request

[mind84]

hi @lucadelu,

one thing you can do to speed up loading is to store the PHP session data cache in a database (e.g. Redis or other). By default PHP stores the session in a file and each request can access this file one at a time, making the application "synchronous". This means that each request is not processed in parallel, but serially since each of them must access the file where the session data is stored. Storing the session cache in a DB allows to process the request in parallel.

Of course this will NOT reduce the time consumption of each request but the application loads asynchronously and became "usable". In particular, getLegendGraphic is unfortunately directly proportional to the number of layers and, at the time of writing, there is nothing you can do about it.

[lucadelu]

hi @lucadelu,

hi @mind84,

one thing you can do to speed up loading is to store the PHP session data cache in a database (e.g. Redis or other). By default PHP stores the session in a file and each request can access this file one at a time, making the application "synchronous". This means that each request is not processed in parallel, but serially since each of them must access the file where the session data is stored. Storing the session cache in a DB allows to process the request in parallel.

ok, how can I set it, just changing the php.ini ?

Of course this will NOT reduce the time consumption of each request but the application loads asynchronously and became "usable". In particular, getLegendGraphic is unfortunately directly proportional to the number of layers and, at the time of writing, there is nothing you can do about it.

ok, so why not read it from a file? It is a small file less than 1MB for hundred layer but it takes time

[gioman]

ok, so why not read it from a file? It is a small file less than 1MB for hundred layer but it takes time

This seems a reasonable request/solution for a problem that affects the platform since it moved to use the legend in JSON format rather than png as it did before, if I understand well the problem. I guess that the really issue is the longer time that QGIS Server takes to generate the JSON, still the suggestion seems a good solution to "solve" the problem from the LMWC side of things. Is there any room for discussion about how LMWC could help making it happen?

[gianvav]

From version 3.6 to 3.7 lizmap, the application's ability to satisfy the needs of public administrations that need to overlap a lot of structured technical information has disappeared. The request for getLegendGraphic is directly proportional to the number of layers but also to their style, increasing in case of categorized styles rather than single symbol style. It is therefore not only a problem linked to the quantity of layers, but also of communication. For a project intended for medium-sized technical users (professionals, engineers, architects), the response time of getLegendGraphic varies from 40 seconds up to a minute, causing frequent timeouts and an unpleasant experience.

The time could be reset if the legend was created together with the project and the conf file.

[Gustry]

The legend depends on different things : user rights(group visibility for a layer) for the current logged user, scale visibility for rule based renderer etc I'm not sure, but also the state of collapsed legend a well will change the response time (it says in the plugin that it decrease performance if legend image is displayed at startup, I can check that)

It would be interesting to first compare the GetLegendGraphic in image format, in JSON format (both without Lizmap server plugin), image and JSON format again but with the plugin

A server cache would make more sense than generating static files IMHO, so it's automatic,

[gioman]

A server cache would make more sense than generating static files IMHO, so it's automatic,

@Gustry Yeah, I understand that would be the ideal solution, or may be even better would be to make QGIS Server quicker. Still I think here there is really an issue for many. Maybe the plugin solution could help mitigate it on the short term; as the plugin now talks with the server, maybe it could be made smart enough to generate all the necessary legends depending on rights for groups, etc. (just rumbling here). Either way... I'm available to support any type of effort that would make possible to move on from 3.6.

[lucadelu]

The legend depends on different things : user rights(group visibility for a layer) for the current logged user, scale visibility for rule based renderer etc I'm not sure, but also the state of collapsed legend a well will change the response time (it says in the plugin that it decrease performance if legend image is displayed at startup, I can check that)

yes this is true, however you could add all layer to the json file and filter it later on the client

It would be interesting to first compare the GetLegendGraphic in image format, in JSON format (both without Lizmap server plugin), image and JSON format again but with the plugin

I tested the single request without JSON format is really faster, the mail problem I think is the base64 conversion, it is cool but take a lot of time to be done

A server cache would make more sense than generating static files IMHO, so it's automatic,

Yeah this could be a solution too (however I think that json file is the better one because the json could change within the project) I tried to cache the request in nginx but I was not able to do that (I'm not an nginx expert), if you could help it would be really appreciated. We are using your docker-compose solution, so a improvement there could be the better way.

[gioman]

@Gustry seen you have added the sponsor needed label, which is great. What would be the plan? Patch QGIS Server? Cache the legends? Create static legends with the LM plugin?

[nboisteault]

I think we could at least use etags for the GetLegendGraphic request but issues will happen if legends change with data changes. Most of the time legends are static so we might need an option to always cache by default and allow to not cache. cc @rldhont

[gianvav]

Legends may change when you update the project, in style, in order, and when new layers are added.

It would be great if the getLegendGraphic request to be cached is generated when the cfg file is saved together with the project, considering a warning when saving the project having modified the layer tree.

[nboisteault]

@lucadelu Do you have a project to reproduce the issue? We don't have this slowness problem with clients hosted on 3Liz servers.

[gianvav]

here https://provinciadilatina.gishosting.eu/index.php/view/map?repository=webgis&project=SIT

[Gustry]

Can you have some QGIS logs (with verbosity) for the GetLegendGraphic ? There should be a bottleneck somewhere

[nboisteault]

here https://provinciadilatina.gishosting.eu/index.php/view/map?repository=webgis&project=SIT

27s / 167 nodes = 160ms per node

With another test where I just take 10 nodes : 5s / 10 nodes = 500 ms per node

There is something wrong server side but you'll need to monitor/benchmark as we don't have those values on our servers.

[Gustry]

@Gustry seen you have added the sponsor needed label, which is great

Any help : either technically on the source code (some debug would be nice, to first check where is the bottleneck, adding some verbosity in logs, doing a "profiling" to see where the time is spent in the process) or financially if it's really an issue.

We have more customers on 3.7 than 3.6 now, we didn't experience this issue so far, with such big delay.

[gianvav]

waiting @lucadelu for the log file, I checked the getLegendGraphic request for each layer and put the results into a CSV response_times.csv getLegendGraphic_layers_response.xlsx

As you can see, the categorized and rule-categorized layers are the most time-consuming.

I also noticed that when the user load is high the average request time increases

[lucadelu]

Can you have some QGIS logs (with verbosity) for the GetLegendGraphic ? There should be a bottleneck somewhere

Sorry for delay, here you are a log file when people access to the project (it is repeated twice)

Responding also to @nboisteault our system is based on this compose-file, the server is quite powerfull more than 100GB ram 12 cpu and the server load is really low, I never seen it more the 5. I also customized postgresql, nginx and pyqgis-server adding more capabilities but this only mitigated a little bit but it not solved the problem.

Thanks a lot

log_pyqgis.txt

[lucadelu]

@Gustry seen you have added the sponsor needed label, which is great

Any help : either technically on the source code (some debug would be nice, to first check where is the bottleneck, adding some verbosity in logs, doing a "profiling" to see where the time is spent in the process) or financially if it's really an issue.

We have more customers on 3.7 than 3.6 now, we didn't experience this issue so far, with such big delay.

@Gustry can you help me to provide any good profile?

However if you think the creation of json file in the plugin could be a solution, I can work on the plugin side if someone will work on the lizmap client side...