Various filter errors on a layer with multiple numeric types

[orcaomar]

@agiudiceandrea, @orcaomar: Finally, I remembered this and found the time to look closer into the matter. My findings so far:

1. A numeric field cannot have ALL their values equal to NULL (as @agiudiceandrea states in his/her comment).
2. It seems like the maximum number of type numeric and integer fields is four. If I have more then four numeric fields I get the error message.
3. The layer has 23 features. This gets narrowed down to 16 when adding the "Data Range Filter". Why?
4. The slider "floors" the maximum value in the field? A polygon in my data set has the maximum value = 74797,9247. The slider's max. value (on the right side) shows 74797 resulting in the polygon not being visible (included).
5. The layer has four numeric fields. However in the "Filter" I see only: "fid" > 4.000000 AND "fid" < 34.000000 AND "artype" > 12.000000 AND "artype" < 60.000000 AND "areal" > 372.704900 AND "areal" < 74797.924700 Why is not delta_m2 included in the filter? It is included in the legend in the layers panel.
6. Removing the legend widget in a) Layers properties is not enough. One has to also b) clear the Filter. Shouldn't doing a) do the trick alone?

Anyway, I find this legend widget very useful and would really like to se it included in QGIS. Any chances for that?

Four numeric fields:

23 features:

16 features filtered:

_Originally posted by @9ls1 in https://github.com/orcaomar/qgis_range_filters/issues/1#issuecomment-1234081912_

[orcaomar]

@9ls1 I created a new issue with your other comments. For some of these, the fix is clear. Eg the floor issue. But others, I'll see what I can do to reproduce. Any chance you can create a subset of this layer and point me to it so I can see if I can reproduce the errors?

[9ls1]

@orcaomar Thanks for looking into this.

I have been playing further around with the widget. I see now that all my four numeric fields area included in the filter. So point 5 regarding filter in my list above seems to be ok.

As a quick test I changed from > and < to >= and <= respectively in line 88 in the data_layer_range_filter_widget.py. (I know some python and just starting looking into making QGIS-plugins, so for now I'm not able to change too much to try to improve your code). In this way I got all my end values included and the number of features is 23. However, when I change (reduce) the maximum value on the slider, some polygons disappear accordingly. BUT: When I try to change back again (increase the max value back to the original maximum value), the original maximum is unavailable, i.e. it has changed to a new smaller value?!? Resulting in not all polygons showing up. Originally, all polygons included (the max value set to 54): Reducing the maximum of delta_m2, some polygons disappear correctly (Inotice the change from 54 to 53): Trying to include all again, but the max. value is now 49 and not 53 or 54 (should be 54 since delta_m2 max = 54,523?):

I thought perhaps the occurrences of NULL values in delta_m2 might cause the problems, but also after including values for these, the max. value never reaches back to it's originally value. And the filter says: ..."areal" <= 74797.924700 AND "delta_m2" >= 0.380000 AND "delta_m2" <= 49.650130

The gpkg-file for testing (without NULL values for the delta_m2): ar5_vega_widgettest_u33.zip

[orcaomar]

Thanks a lot for this info, was very helpful at debugging the various issues. It's been fixed in this commit: b8e6f01

Hope it's helpful!

[9ls1]

@orcaomar Thank you!!! (Glad I could be of some help even though limited PyQGIS knowledge.) It's getting very close to perfect now. Some small remarks, though:

Point 6 in my originally post: "Removing the legend widget in a) Layers properties is not enough. One has to also b) clear the Filter. Shouldn't doing a) do the trick alone?" If I have reduced my max. value for one field (resulting in e.g. two polygons disappearing accordingly) and then removes the "Data Range Filter" legend widget, the layer remains filtered, not showing all features. Perhaps no obvious logic in resetting (clearing) the filter when removing the widget, but I think people may forget (and overlook the filter symbol to the right of the layer name) and might be puzzled later on when polygons are missing? I would have appreciated and expected the filter to be cleared whenever I remove the widget.

The legend widget is applied:

One field is changed and some features disappears accordingly:

I have removed the legend widget and the filter remains (I have to manually clear the filter to get all features):

[9ls1]

Another thing. I set two values to NULL in the field delta_m2. The widget works. Everything turns up as expected.

They remain if I change the "areal"-slider.

All polygons return when changing areal back to original maximum value.

But once I change the delta_m2 slider, the two polygons with "empty" delta_m2 (= NULL), disappear.

If I change back to originally min. and/or max. values, the two polygons don't return. That may be logical (having NULL), but since they appear in the very beginning, it may seem a little bit odd when they don't show when the slider-values are back where they started?

[9ls1]

I tested yet another thing... I added a new integer field antall. My layer has now five numeric fields. This works fine in the sense that a slider for all five numeric fields turns up. So no longer limited to four numeric fields it seems. However, I made a "twist", I set all values for antall to 43.

1. The sliders min. and max. value is set to 43.01
2. When I change the slider, all polygons disappear.

Expected behavior:

1. Min. and max. value = 43
2. The slider for field antall should a) be disabled (not possible to change) or b) not appear at all or c) go from 0 to 43 only allowing 0 or 43 (no values in between) - like a boolean - but that makes no sense as all polygons will disappear if 0 since they all have antall = 43...

Perhaps alternative a) is best - the user sees that the field is "included" but it is not possible to change since min. value = max. value. Alternative b) may make the user wonder if the field is forgotten?

[orcaomar]

Thanks for all your comments. Code submitted to fix most of them.

Are you a QA person in your downtime ha, your work on edge cases is impressive, takes me back to my software engineering days.

I fixed the minimum and maximum issue, but left the slider there enabled. I could do what you're asking but it would be more work, didn't have time.

I've cleared the filter string when the filters are removed

In terms of NULLs in the dataset, this is an interesting challenge, not sure what the right approach is, honestly. If you use normal filter strings and put in a range, say data > 3 AND data < 20, and you also have data = NULL for some of your items, then those will be filtered out, because they are not strictly in the range. This behavior is probably a user preference... I think I'll leave the result at is for now.

Thank you!

[9ls1]

@orcaomar , not particularly a QA-person, just very curious ;-) Again, thank you so much for your time and effort.

The removing of the legend widget works fine as long as I have not changed any of the sliders. Whenever any of the sliders has been changed before removing the widget, the filter symbol ("funnel" ) appear to the right of the layer name in the TOC after removing the widget (even though the Filter is empty):

And yes, there may not be any obvious solution when it comes to the slider and null values. As you say it's most likely a question of user preference. I'm fine with the behavior as it is now.

Finally, three remarks regarding the minimum and maximum values: a) I really think all values in the sliders should be presented as integers (regardless the field definition). Here I have added another field of type real with a constant value of 23,15. It should be 43 and 23 in my opinion.

b) If you find the time, I do think a solution where the slider is present for numeric fields with a constant value, but where you can't move the slider neither to increase nor decrease the min. or max. value, would make this great widget even more great. :) c) If possible, a tiny gap between the square and the value, would make the slider even more readable?

[orcaomar]

Again thanks for your comments. All are done except I cannot figure out how to remove the filter symbol (I tried even before your comment!) when the filters are cleared. It's odd.. there are no active filters but I'm not sure what triggers that symbol and it's a bit too much work at this point to figure this out.

Thanks for all your feedback, I'm glad you're finding this useful. If there are additional changes you want or features you think would be useful don't hesitate to post an issue and assign it to me. It's nice to code a bit here and there, takes my mind off other things. Hopefully something like this feature gets more integrated into QGIS. My next task is to get this plugin actually into the plugin library, I just think there's a bunch of small things I need to do before I get there and I don't really want to do that.

On the off chance that you want to help out, you can figure out how to get this plugin into the gallery, like what changes we'd need to make or just what things we need to do to get it listed.

[9ls1]

Wonderful - many, many thanks for your great work, @orcaomar! It works like a charm and looks very nice. For time being I just ignore the filter symbol.

Although it is easy to include the widget in QGIS (just download, unzip and store the folder in plugins), I believe the widget will gain much more fame and users if it's present in the QGIS plugin repository and perhaps even as a default legend widget in QGIS core.

I will look into the procedures of getting a plugin into QGIS library (I hope to develop and publish my own in near future and then I may be to further help 🤞). I assume you have already read https://plugins.qgis.org/publish/ and https://g-sherman.github.io/Qgis-Plugin-Builder/.

In any case, I will spread the word about this useful plugin that can be downloaded from here in anticipation of being included in plugins/QGIS.

[orcaomar]

Hey @9ls1 so taking your advice, I did the work of getting this plugin in the qgis repository: https://plugins.qgis.org/plugins/legend_data_filter/

So hopefully more people will use it and file bugs and feature requests and I'll slow keep at it.

Tell all your QGIS friends!

[9ls1]

@orcaomar 👏👍Yes, I will, I'm already on to it.

[agiudiceandrea]

I did the work of getting this plugin in the qgis repository:

The plugin is listed in the "Plugin Update September 2022" blog post on https://blog.qgis.org.