filter - reject/require

[ImreSamu]

Proposal ....

• I have refactored the syntax according to ( https://github.com/omniscale/imposm3/pull/133 )

current syntax :

    filter:
        reject:
          key1: [ val1, val2, ... valn ]
          key2: [ val1, val2, ... valn ]
          key3: [ __any__ ]
          .... 
        require:
          key1: [ val1, val2, ... valn ]
          key2: [ val1, val2, ... valn ]
          key3: [ __any__ ]
          ....
        reject_regexp:
          key1: 'regexp'
          key2: 'regexp'
          ....
        require_regexp:
          key1: 'regexp'
          key2: 'regexp'
          ....

golang struct

type Filters struct {
    ExcludeTags   *[][]string    `yaml:"exclude_tags"`
    Reject        KeyValues      `yaml:"reject"`
    Require       KeyValues      `yaml:"require"`
    RejectRegexp  KeyRegexpValue `yaml:"reject_regexp"`
    RequireRegexp KeyRegexpValue `yaml:"require_regexp"`
}
type KeyRegexpValue map[Key]string

• Added a regexp implementation ( if not accepted, I will remove, but sometimes it is very useful )

      ## housenumber cleaning ... 
      require_regexp:
        'addr:housenumber': '^\d+[a-zA-Z,]*$'

      ## filtering extreme tagging
      require_regexp:
        building: '^[a-z_]+$'

• Added some warnings
• Added minimal test cases

example mapping:

  config_test_t3_highway_with_name:
    fields:
    - name: id
      type: id
    - key: highway
      name: highway
      type: string
    - key: name
      name: name
      type: string
    filters:
      require:
        name: ["__any__"]
      reject:
        highway: ["no","none"]
    mapping:
      highway:
      - __any__
    type: linestring

// Allowed to modify, refactor, optimalize as you like

=== Edited:

• Important Regexp Note: be careful because yaml interprets escape sequences inside double quoted strings, so use single quoted string !

[ImreSamu]

@olt : I'd appreciate any feedback you can give me on the new syntax.

If accepted :

• we would like to replace the old exclude_tags filtering in our mappings
• and start real life testing ...

[olt]

Sorry, the first notification slipped through... Yes, the syntax looks good. Haven't reviewed the code though.

I would deprecate the old exclude_tags syntax: Print a warning (see pseudoarea) and convert it to a reject filter.

Thanks!

[ImreSamu]

Yes, the syntax looks good.

Thank you !

• In the next few weeks I will start using and deeply testing the new syntax
• I will try re-check the warnings ( as you suggested )

if I will finished, I will ping you ...

[ImreSamu]

@olt: ready for review !

I would deprecate the old exclude_tags syntax: Print a warning (see pseudoarea) and convert it to a reject filter.

I have refactored ( According to my best knowledge )

some experience:

• ./example-mapping.json and ./example-mapping.yml not exact same ,
  • for example , the landusages mapping order is different
• I have updated the mapping/test_mapping.yml and maybe the mapping/test_mapping.json can be removed ....
• Important Regexp Note: yaml interprets escape sequences inside double quoted strings, so use single quoted string !

// Allowed to modify, refactor, optimalize as you like

[ImreSamu]

ping @olt

[olt]

I've updated the docs, moved the test mapping inside the tests and merged your changes.

Thanks a lot for working on the patch and your patience! 👍