OSM-POI: Include brand property

[mattigrthr]

OSM objects may include a brand or operator tag, which you can use to derive the brand of a POI.

The issue that exists is that the values of those tags can be spelled differently across several entities (e.g., "McDonalds", "Mc Donald's", or "McDonald's").

There exists a repo that tries to unify the spelling across OSM: https://github.com/osmlab/name-suggestion-index/.

Otherwise, it is an option to find a clean list of worldwide brand names and use string distance measures to connect a POI to a brand.

[mattigrthr]

@IritaSee, we already parse the operator and brand tags from the OSM objects and include them in the Parquet files.

This is the file where we process the Parquet files after running the osm-parquetizer pipeline which transforms the pbf files to Parquet files: https://github.com/kuwala-io/kuwala/blob/master/kuwala/pipelines/osm-poi/src/Processor.py

The idea would be to find the best match of a brand or operator in the name-suggestion-index.

Those are the necessary steps:

• [x] Compile a list of all brand names and operator names from the name-suggestion-index as a CSV with the columns id, display_name, and wiki_data and store it in the tmp folder with the other OSM-files. Deadline: 04.01.2022
• [ ] Create a PySpark UDF similar to the ones in the osm-poi/src/Processor.py that takes a string and returns the best match against the name-index-suggestions. I recommend using fuzzy-wuzzy Deadline: 06.01.2022
• [ ] Apply that function for the brand and operator columns which creates two new columns brand_matched and operator_matched. The best place to apply the function is after the data frames for nodes, ways, and relations have been merged into one at line 353 in the Processor.py atm. Deadline: 07.01.2022

[mattigrthr]

More details about how Spark UDFs are used are in the PR discussion: https://github.com/kuwala-io/kuwala/pull/69#issuecomment-1009823381

[mattigrthr]

After doing some initial tests with the brand and operator name matching, it turns out that including the matching in the OSM-POI pipeline directly would increase the runtime significantly. Therefore, we have decided to store the consolidated list of brand and operator names in a separate table in Postgres, which can then be used later in transformation blocks (e.g., on a filtered set of POIs and thus drastically reduce the runtime).

Since the canvas development currently has a higher priority for the core team, this issue is up for grabs again.