QGIS 3.34 takes >10 minute to load up with WMS/WFS datasources

frida-161 commented 4 weeks ago

What is the bug or the crash?

With the current LTR (3.34.7) it takes more then 10 minutes to start QGIS when I have a large number of predefined WMS/WFS datasources in my profile. I tested old LTR (3.28) and current (3.36) version with the same profile and the startup takes a only a few seconds.

Apparently QGIS does some changes to the QGIS3.ini file in the profile during the startup and shutdown processes . I tried making a copy of the QGIS3.ini while qgis was running. After the shutdown process which takes a few minutes and overwrites the ini again, I replaced it with my copy. The next startup was almost instant. But during the shutdown the ini gates rewritten again so the next startup again takes more then 10 minutes.

I could reproduce the bug on Arch Linux (qgis-ltr build from the AUR) as well as Windows 11 (Installer from the qgis.org website).

I build qgis with Debugging enabled but I dont find much about whats going on in the log except for the second line which seems to be the part that takes > 10 minutes.

I found this issue about rewriting of connections, maybe it is related #56713

.cpp:514 : (main) [0ms] Starting qgis main
messagelog.cpp:34 : (logMessage) [673134ms] 2024-06-05T08:40:49 [0] Successfully loaded globalsettingsfile path: /usr/share/qgis/resources/qgis_global_settings.ini
viders/qgsproviderregistry.cpp:367 : (init) [1245ms] Loaded 30 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
messagelog.cpp:34 : (logMessage) [6086ms] 2024-06-05T08:40:56 Plugins[0] Loaded MetaSearch Catalog Client (package: MetaSearch)
messagelog.cpp:34 : (logMessage) [86ms] 2024-06-05T08:40:57 Plugins[0] Loaded DB Manager (package: db_manager)
messagelog.cpp:39 : (logMessage) [1754ms] 2024-06-05T08:40:58 Processing[2] Problem with GRASS installation: GRASS was not found or is not correctly installed
messagelog.cpp:34 : (logMessage) [7ms] 2024-06-05T08:40:58 Plugins[0] Loaded GRASS GIS provider (package: grassprovider)
messagelog.cpp:34 : (logMessage) [1256ms] 2024-06-05T08:41:00 Plugins[0] Loaded Processing (package: processing)
messagelog.cpp:34 : (logMessage) [598ms] 2024-06-05T08:41:00 [0] Application state:
QGIS_PREFIX_PATH env var:       
Prefix:     /usr
Plugin Path:        /usr/lib/qgis/plugins
Package Data Path:  /usr/share/qgis
Active Theme Name:  default
Active Theme Path:  /usr/share/qgis/resources/themes/default/icons/
Default Theme Path: :/images/themes/default/
SVG Search Paths:   /usr/share/qgis/svg/
        /home/frida/.local/share/QGIS/QGIS3/profiles/rvr_wms_laden/svg/
User DB Path:   /usr/share/qgis/resources/qgis.db
Auth DB Path:   /home/frida/.local/share/QGIS/QGIS3/profiles/rvr_wms_laden/qgis-auth.db

messagelog.cpp:39 : (logMessage) [358230ms] 2024-06-05T08:46:58 Processing[2] Problem with GRASS installation: GRASS was not found or is not correctly installed
True

Steps to reproduce the issue

Create a fresh QGIS profile
Go to Datasource manager
Go to WMS and click Load and load the wms.xml file contained in the zip file below
Go to WFS and click Load and load the wfs.xml file contained in the zip file below
close QGIS
start QGIS again and wait..

connections.zip

Versions

QGIS version	3.34.7-Prizren	QGIS code branch	Release 3.34
Qt version	5.15.14
Python version	3.12.3
GDAL/OGR version	3.8.5
PROJ version	9.4.0
EPSG Registry database version	v11.004 (2024-02-24)
GEOS version	3.12.1-CAPI-1.18.1
SQLite version	3.46.0
PDAL version	2.7.1
PostgreSQL client version	unknown
SpatiaLite version	5.1.0
QWT version	6.2.0
QScintilla2 version	2.14.1
OS version	Arch Linux

QGIS version 3.34.7-Prizren QGIS code branch Release 3.34 Qt version 5.15.14 Python version 3.12.3 GDAL/OGR version 3.8.5 PROJ version 9.4.0 EPSG Registry database version v11.004 (2024-02-24) GEOS version 3.12.1-CAPI-1.18.1 SQLite version 3.46.0 PDAL version 2.7.1 PostgreSQL client version unknown SpatiaLite version 5.1.0 QWT version 6.2.0 QScintilla2 version 2.14.1 OS version Arch Linux

This copy of QGIS writes debugging output.

Active Python plugins MetaSearch 0.3.6 db_manager 0.1.20 grassprovider 2.12.99 processing 2.12.99

Supported QGIS version

[X] I'm running a supported QGIS version according to the roadmap.

New profile

[X] I tried with a new QGIS profile

Additional context

No response

3nids commented 3 weeks ago

@nyalldawson since you have more or less followed the story, what do you think about dropping the backward compatibility and make the forward migration a one time operation (on demand, on first start, you would be asked if you want to migrate former settings, and once it's done, you don't go over it once more)?

nyalldawson commented 3 weeks ago

@3nids

dropping the backward compatibility and make the forward migration a one time operation

+1

(on demand, on first start, you would be asked if you want to migrate former settings, and once it's done, you don't go over it once more)?

Instead of on-demand , could we just change the key and then add a new setting which tracks which ones were migrated so we never do that more than once. (And store the migrated list outside of somewhere were a user action such as deleting a connection will remove it?)

frida-161 commented 3 weeks ago

@3nids It seems like I'm not up to date on the story :)

Are there any Issues/CRs I can read to get in the loop?

We have a customer who wants to upgrade to the current LTR but this loading times are a dealbreaker for them. I'm willing to do my part in getting this resolved by I'll probably need some pointers.

Frida

3nids commented 3 weeks ago

@nyalldawson what about this:

On first load QGIS imports the settings with the current forward import code. Whenever done, it saves in a settings that this was done. That import is never run again. We bring this functionnality to current release and 3.34.

We drop the backward compatibility code. In the forward compatibility code, we obviously drop the deletion of old settings part (which were recreated in backward compatibility).

This means that the settings won't be synchronized anymore with 3.32- which I found OK now that 3.34 is LTR.

3nids commented 3 weeks ago

@frida-161 there is a new system for storage of settings which has some backward/forward compatibility code to ensure integrity across old versions of QGIS. This compatibility code is the reason of slowness and has brought some painful issues.

frida-161 commented 1 day ago

@3nids thanks for the answer.

I understand that the compatibility code runs on 3.34 LTR and 3.36. What I don't understand is why this process takes minutes on the 3.34 and only seconds on 3.36. It seems to be that there are some changes to the compatibility code. So maybe backporting these changes to LTR would be another option to fix this? Can you point me to the relevant parts in the code?

Frida

qgis / QGIS