search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.6k stars 3.01k forks source link

Sort by expression freezes qgis #40272

Open dingausmwald opened 3 years ago

dingausmwald commented 3 years ago

3.16 windows 10 x64 geopackage local/network drive/shapefile local

Point layer in different formats (shp, geopackage), from different sources (local, networkdrive), with about 47000 objects. When trying to sort for duplicates ( count("FieldWithDuplicates","FieldWithDuplicates") ) qgis freezes, prozessor is working. No result after 20 min.

was working on 3.4

gioman commented 3 years ago

@dingausmwald please attach a dataset we can use for local tests.

dingausmwald commented 3 years ago

Same goes for select by expression (COUNT(1, "attribute")>1)

Adressen.zip

gioman commented 3 years ago

Ok I see the freeze, but only when doing the sort within the table as from processing seems ok.

agiudiceandrea commented 3 years ago

Hi @dingausmwald, you could try with a subset of your layer of e.g. 1000 records and you will see it works in a reasonable amount of time.

I think this approach is inefficient 'cause the same calculations (group 47000 records and count the occurrences of all the groups) will be re-evaluated for each row, i.e. 47000 times.

Could be better to relies on the RDBMS functionalities of i.e. a Virtual Layer to do this job.

For example you could create a Virtual Layer using the SQL statements: select Adressen.*, B.count from Adressen left join (select adresse, count(*) as count from Adressen group by adresse) B on Adressen.adresse = B.adresse that adds a column with the number of the occurrences, and then you can sort by this column.

gioman commented 3 years ago

I think this approach is inefficient 'cause the same calculations (group 47000 records and count the occurrences of all the groups) will be re-evaluated for each row, i.e. 47000 times.

then shouldn't we warn the users when using some type of operators? By the way... it is really just a matter to wait a looong time, is not a complete freeze.

agiudiceandrea commented 3 years ago

@gioman, I've also tested the provided layer file using the Select by Expression (COUNT(1, "adresse")>1) tool with QGIS 2.18.23 and 3.4.9 and it seems they behave the same way of QGIS 3.10 and 3.16.

roya0045 commented 3 years ago

So not a bug, just inefficient operation.

Pedro-Murteira commented 2 years ago

Still valid on QGIS 3.22.3.

m-kuhn commented 2 years ago

Are we able to abort the operation or is qgis just frozen?

Pedro-Murteira commented 2 years ago

From what I've tested, QGIS simply freezes. It was able to complete this selection after waiting for 10 minutes tho.