Support combination of advanced filters and Quick filter for feature grid

This issue is caused by the structure of filters that is modelled on the panel data structure (and it shouldn't be). Because we need also to implement some of functionalities from #6625 I propose a refactor of current filter system to allow an easier of future or custom tools

RFC

@mbarto , @allyoucanmap , @tdipisa , @MV88

Here some notes about the current filtering functionalities, to understand the complexity of the current system and the things to change in order to have a system that can adapt to future requirements.

Filter structure

The internal fliter object structure is made on the model of the query panel and not well separated from the WFS:

{
    featureTypeName: 'gs:us_states', // ← Used for select a different layer, see #6430
    filterType: 'OGC',
    ogcVersion: '1.1.0',
    pagination: {
      startIndex: 0,
      maxFeatures: 100000 // ← this is used to deactivate virtual scrolling when multiple selection is active
    },
    spatialField: [ // this is the original spatial field of the query panel (1 element) extended to array because now in feature grid we need multiple geometric filters
      {
        attribute: 'the_geom', 
geometry: {center: [ // includes circle geometry info, to repopulate the query panel in a second time
            -12997963.78583765,
            4693845.032936105
          ],
          coordinates:  [...]
          extent: [
            -13017531.665078655,
            4674277.1536951,
            -12978395.906596646,
            4713412.912177109
          ],
          projection: 'EPSG:900913',
          radius: 19567.879241004586,
          type: 'Polygon'
        },
        method: 'Circle',
        operation: 'INTERSECTS'
      }, {...}    ],
    groupFields: [  // ← Groups, now used also for quick filters
      {
        id: 1615198153790,
        index: 0,
        logic: 'AND'
      },
      {
        groupId: 1615198153790,
        logic: 'AND',
        id: '1_1',
        index: 1
      }
    ],
    filterFields: [ // ← The attribute fields, linked to groups by groupId
      {
        attribute: 'STATE_FIPS',
        rowId: 1615198153790,
        type: 'string',
        groupId: '1_1',
        operator: 'ilike',
        value: '32'
      }
    ],
    spatialFieldOperator: '' //← Operator to apply multiple spatial filter
  }
}

Moreover inside the code there is support for “simpleFilter”: coming for a previous version of the query panel, or custom development, maybe it is not used anymore. The filters from this field are all in and with the others.

Now as you can see this structure has many limitations:

contains accessory information that should be separated (featureId, pagination, drawing info)
The structure considers attribute filters nested in groups, with a group id, with a flat tree structure. This is hard to maintain
It consider spatial filters separated from attribute filters
do not include filtering by ID.
Spatial filters structure is flat (only SP1 SP2 SP3 filter are allowed)
Not well documented

This structure has some utilities in FilterUtils that are confusing

toOGCFilter → creates a WFS Query
Mix versions of protocols to reuse code
not well documented

They produce only filter (CQL/OGC) based on the structure above, without any kind of support

In order to provide support for development, we created some utilities in utils/ogc

Filter/FilterBuilder: provides a series of utility functions to create filters like:
`and(property(“P”).isEqual(2) , property(“P2”).intersects(geometry))

Filter/CQL/parser parse CQL returning a filter with this shape (more rational and simple, realized as a nested tree):

{
  type: "and",
    filters: [{
       type: "=",
       property: "property1",
       value: "value1"
    },{
       type: "=",
       property: "property1",
       value: "value2"
    }]
  }

Filter/fromObject that converts the output of CQL/parser into OGC filter, using filterBuilder (missing support for unary and geometry operators)

layerFilter property inside a layer has the filter object structure to:

create the CQL filter for WMS
load the layer in the Attribute filter

These filters are saved in maps, so we need to maintain retro-compatibility.

Quick filters:

they have this shape

[{
attribute: "STATE_ABBR"
operator: "ilike"
rawValue: ""
type: "string"
}, {
attribute: "SUB_REGION"
operator: "ilike"
rawValue: "W N Cen"
type: "string"
value: "W N Cen"
}]

Also these filters are saved in dashboards. Anyway this is an internal structure, flat by nature, and can be converted in an and filter (that is implicit for the purpose of this tool). This different structure doesn't look to be problematic at all.

Conclusions

The whole MapStore filtering functionalities are confusing and makes difficult to develop.

Adding ID filtering to the current system add even more complexity to a system that need to be simplified and uniformed.

Proposal

A possible solutions to keep retro-compatibility is to add a "version" entry to determine if the filter has old or new style.

The new style proposed, lets say version: 2 is the nested structure present in CQL filter management, that is more rational and processable, because of it's tree structure.

In order to match the current query panel with the structure of version 2. We will have an AND filter as root, that includes the actual spatialFilter, layerFilter and attributeFilter.
If possible then whe should move all the logic that handles these 3 fields only inside a conversion tool, to keep retro-compatibility, and inside the query panel. Outside we should use only the new structure.

Some utility functions (like FilterBuilder and other utils) will allow to compose filters without needing to know and manipulate the internal structure.

geosolutions-it / MapStore2