Fix/Add POI categories

[hhornburg]

Important Notice:

This is for developers only. As a user use the bug report or Everything that's not a bug template.

The parent ticket

add ticketnumber behind the #

QMS-#294

What do you plan to do

Improve and add SQL queries and icons to show more POIs from the *.poi files.

Please report missing POIs fbelow and I will add them to the TODO list after checking the issue. Please be so kind and check/provide the following:

• Is the POI in the *.poi file? If not I'm afraid we can't do anything at this point. You could try find a reasonably quickly machine-readable source of POIs where it is included and then open an Issue requesting the support of that file type. Ideally implement it yourself :wink:
• What are the osm tags?
• In what category do you want to place it? If it does not exist consider donating an icon.

The result will be

TODO List: Status	OSM Tags	Suggested Category	Icon	Notes
:red_circle:	leisure=park	leisure_park	missing
:yellow_circle:	leisure=water_park	sport_swimming_outdoor	exists
:red_circle:	sport=climbing	sport_climbing	missing	there are a lot of rock climbing sites (not hill climbing)
:red_circle:	sport=scuba_diving	sport_scuba_diving	missing	different to cliff diving
:red_circle:	tourism=wilderness_hut	tourism_wilderness_hut	missing	see description in OSM, there are a lot.
:yellow_circle:	:arrow_up:	accommodation_shelter	exists
:red_circle:	waterway=waterfall	poi_waterfall	missing	in the 'poi' family like poi_cave
:red_circle:	natural=spring AND NOT amenity=drinking_water	poi_spring	missing	in the 'poi' family like poi_cave

[hhornburg]

@mitxel-m Just wanted to notify you that I opened an issue here

[mitxel-m]

@pingurus Sorry for the delay in answering.. The attached file contains the missing icons up to now, in svg and png 32x32 version. I have also added a modified version of the icons for alpinehut and shelter, as the existing version doesn't look very good on the map when QMS reduces it to 22x22 px. Since they are POIs that we want to view easy on the map I have also differentiated the colour from the rest of the accommodation family. Red is the colour that contrasts the most with the usual background of the maps. I think this makes total sense.

I've had doubts about how to proceed with the icons, I haven't fully learned the PR process yet and I've been a bit confused. In order to not delay more time and move forward I include the icons in a zip file

more_icons_1.zip

I also have some work done on improving the DB queries to get more refined results, I think I will be able to write it during this weekend.

[hhornburg]

No worries, we can move on with the rest of the code and then do this PR as a last one so we don't have to open this topic a second time.

I could help you with questions regarding the PR process (and maybe with compiling, if you use Ubuntu), if you want to implement and test this stuff yourself.

[hhornburg]

Another thing I noticed: Cities, Towns and so on are in the pois category not in the places category, which I found rather counter-intuitive.

[wthaem]

There is an easy answer to the topic of missing categories: Mapsforge has an XML file that defines all used categories (about 400):

https://github.com/mapsforge/mapsforge/blob/master/mapsforge-poi-writer/src/main/config/poi-mapping.xml

There, the categories are arranged in a tree. Only the "amenities" category hasn't only child but grandchildren categories. The XML file defines the OSM tags belonging to each category.

But their category structure is different to the one now used in QMS and can be seen when using the poi_categories and poi_category_map tables in a POI file.

[hhornburg]

Thanks, I had been looking for that file but somehow missed it. The reason why I switched to using a custom map was that I found no public icon set for the categories defined by mapsforge (again, I might have missed something :wink: ) If we switch back to using the predefined categories the search performance also should increase.

[wthaem]

the search performance also should increase:

I think so, too. You can then replace the more or less full-text search in the data field by a query in the categories tables. But you have the icon problem - no idea if there is a Mapsforge set.

[hhornburg]

There is a set at mapsforge, but that set is the same as we have right now (at least at the first glance). @wthaem or @mitxel-m would you be willing to systematically check for which of the predefined categories icons would be needed (that are not in the set we already have)?

[wthaem]

Maybe, the attached pretty long txt file (891 lines) helps to identify the differences between the complete Mapsforge list defined in the above mentioned XML file and the list used in QMS. The file was created with a Python script.

The file shows:

• Some statistics
• QMS only tags/categories (not in Mapsforge)
• Tags/categories used in QMS and Mapsforge
• Mapsforge onla tags/categories (not in QMS)
• The complete Mapsforge category tree derived from the XML file

If you let me know the relations between categories/tags in the cpp file and icons I might be in a position to do more.

categories.txt

[hhornburg]

Nice, thanks! You might want to remove the symbol '%' from the QMS tags, as I inserted it into at least some of the shop tags for shops that sell multiple stuff and/or sport terrains that are used for several sports. (I think those are just a dozen tags or so, but it still cleans up the list a bit more)

[wthaem]

Done. Attached is updated file.

categories_rev.txt

[hhornburg]

I just checked and the mapsforge POI xml doesn't seem to be the same as used at openandromaps.. For example the tag waterway=weir does not appear in the list you linked to, but appears in the POI file as a separate category. Could you also create a tree of the categories present in the openandromaps POI files?

[wthaem]

Please give me a link to such a POI file or something similar to the Mapsforge XML. With a quick search I didn't found ready-made openandromaps POI files, sorry.

[hhornburg]

I take the POI files from here: https://www.openandromaps.org/downloads/deutschland From my side this feature request is based on the .poi file included in any download from openandromaps. I have found no tagmapping for their POI files, only for their maps in general (https://www.openandromaps.org/map-basics-2/tag-mapping).

[wthaem]

Attached is the category tree for the file baden-wuerttemberg_oam.osm.poi. I can't compare it with the Mapsforge tree because the OSM tags needed for that purpose in my script are missing. I'll try to compare the POI names somehow.

categories_BW.txt

[wthaem]

The quick attempt to compare a Mapsforge category tree with an openandromaps one failed. The category names are similar but nevertheless different. The tagmapping-urban/min.xml files give some OSM tags for some POIs. But again the relation to a openandroidmaps POI file doesn't give an easy 1-1-match. The missing piece is a file with a mapping between the oam categories and their OSM search tags. Both types of POI files have the same database structure, but different categories.

This situation makes the business a bit difficult: QMS is going to offer POI support. Both types of POI files are very popular. When trying to support both types, then certainly there should be some mapping between both category types. If there were the OSM tag definition for the oam categories, then this mapping should be possible with the help of some automatically created table. Otherwsie, there seems to be only the boring manual way or the decision to support only one type of POI files. In the last case, the Mapsforge variant seems to be easier to handle than the oam one.

[hhornburg]

I'd say that if there is no easy mapping between those two types of poi files QMS should stick with choosing it's own categories by using the OSM tags. This of course has some drawbacks regarding performance, but makes it easy to add other sources for POIs later as long as those are based on OSM. @kiozen what do you think?

[wthaem]

Quote from

https://github.com/mapsforge/mapsforge/blob/master/docs/POI.md:

Custom Category Configurations

If the provided category configuration does not fulfill your needs, you can easily create your own.

The default internal poi mapping is defined in https://github.com/mapsforge/mapsforge/blob/master/mapsforge-poi-writer/src/main/config/poi-mapping.xml. So if you want to define your own custom mapping, use the internal mapping as a reference.

Please consult the XML-Schema documentation of https://github.com/mapsforge/mapsforge/blob/master/resources/poi-mapping.xsd to learn about how to implement your custom mapping. The XML-Schema is very easy to understand. We recommend to use a proper editor for XML to allow auto-completion and validation if you provide an XML-Schema. End of quote.

Why not merging the existing QMS categories into the Mapsforge XML file as described in the quote? This can be done more or less automatically with a script and this approach would open the possibility to add more user-defined (QMS-defined) categories later (one manual step could be the removal of some duplicates).

Keep this extended XML in the QMS sources and parse it as a preliminary step to POI handling. Then it gives something similar to the table used in CPoiPOI_TagMap.cpp. As far as I can see, using the oam POI files needs anyway an appropriate category table in the mentioned cpp file.

[mitxel-m]

The .poi DB file contains these tables:

poi_categories: defines the categories and parent categories, each one has an unique id. Note that there are some replicated categories with different parent values.

poi_data: lists each POI with an unique id, and its data (OSM tags)

poi_category_map: each row relates a poi_data.id with a poi_categories.category. There are more rows than existing POIs as long as a POI can be related to one or more category (eg. a POI with more than one OSM tag matching several categories).

I assume that the main idea is to to define some convenient QMS categories. Each QMS category has an icon and a query defined in CPoiPOI_TagMap.cpp. For now the query searches for a text string in the poi_data table, which can become slow.

I think it would be better and faster if we query for the category (an integer) in the poi_category_map table.

Here is an example for alpine huts:

For now it is defined in CPoiPOI_TagMap.cpp like:

map["accommodation_alpinehut"] = SQLQUERY("tourism=alpine_hut"); which performs a query like this SQL statment:

Select id, data from poi_data where data like '%tourism=alpine_hut%'

That returns a list with the id and data of the matching POIs. Then ,with the id , the coordinates are obtained to paint the POI on the map, ...right?

Well, we can obtain the same pair id and data with this query :

Select d.id, d.data from poi_data d join poi_category_map c on d.id=c.id where c.category = 16

• (16 is the id for the alpine_hut category)

@pingurus could you adapt the code to perform this query. If yes we can use the predefined categories in the poi file schema.

Then in CPoiPOI_TagMap.cpp it would be enough to pair each "QMS category" with an "OAM category" id ( an integer)

Here is the id list for all the defined categories in the poi file:

category_list.txt ( csv renamed as .txt for uploading)

If all this can be done you can preserve your work to show and group categories in the way you have done at the POI panel, (with a few exceptions). And I think there is no need to define queries for the parent categories.

This would also work for those POIs matching more than one category.

Example: An alpine hut named 'Utopia hut' has three different OSM tags, matching three (categories):

• alpine_hut (16)
• drinking_water (1)
• shelter (2)

so in the poi_category_map table there are three rows for that POI, all with the same id ( it is not a key) and each one with a different category value. Thus you can find the same poi searching for any of the categories.

This said, the order in which the icons are painted on the map matters, because if you have selected all the categories the shelter icon could hide the alpine_hut icon.

This also happens with the actual string search.

So a kind of draw priority should be set. May be reordering the list in CPoiPOI_TagMap.cpp would be enough.

Remarks on predefined categories of the poi file:

Some of the categories that @pingurus has organized separately are merged in a single category in the poi file schema. Example: Hotel, Guest house, and Pension are all in the same category. The same for temples. So you have to read the description to know a bit more. For me this is not a problem.

Take in mind that to expand those categories the query should search the tag text string like in the actual approach (which is you want to avoid)

In the other hand I find the sport list extremely exhaustive. I would take the most relevant ones for outdoor activity, and group the others in a single category "other sports"

• There are 273 categories ( in the poi file for Spain). May be some of them are irrelevant or with no practical use and can be omitted.
• There are other 29 parent categories, but no real need to use them.

buff!!, a bit long but I hope it can be useful.

[hhornburg]

What I've been thinking about is that maybe we should split icons from categories. The categories would be those defined in poi_categories in the *.poi file, whilst the icons would be selected based on the OSM tags of the search results.

This has some advantages:

• The user can select which categories he has by choosing the source of his POI file
• The search performance should increase, as we can use the predefined categories
• QMS can support any mapsforge POI file
• The icons can be very granular without cluttering the categories to choose from

Of course the problem mentioned by @mitxel-m remains (or maybe it is even aggravated a bit) that there may be multiple OSM tags (and thus icons) matching a POI. Two solutions come to my mind which are independent of the source of the *.poi file:

• We display all the icons, which are then 'de-cluttered' by some algorithm as already suggested by @kiozen in #327
• We introduce some ranking of the icons and only the one that is ranked the best (the most meaningful) is displayed. This requires a lot of manual work and may cause some headache for the example in the case of 'utopia hut' mentioned by @mitxel-m.

[wthaem]

@pingurus: there is some hope that I can provide a list of OAM categories with their most likely OSM selection tags.

[wthaem]

Attached is the result of parsing baden-wuerttemberg_oam.osm.poi. This list gives the OSM search tags most likely used for defining an OAM category.

Example line to explain how to read the file:

Veterinary Clinics: [('amenity=veterinary', 940), ('entrance=yes', 5)]
                    ------------------------5-------------------------
---------1---------    --2---- ----3-----   -4-

• 1: English part of poi_categories.name in POI file
• 5: the 2 most common tag/value pairs found for POIs with this category in poi_data.data
• 2=3: OSM tag=value pair
• 4: frequency of tag/value pair in poi_data.data field

Some obviously not interesting tags (like name=, wheelchair=) have been suppressed.

In most cases, the second tag/value pair can be dropped because of the low frequency.

tags4cats_BW.txt

[wthaem]

The results of a comparing OAM and Mapsforge categories using a script can be seen in the attached file.

Compare_OAM_Mapsforge.txt

[hhornburg]

Great job, thanks! I think there are some mistakes by your script though, as those combinations don't really make sense (or are not complete):

OAM category not found in Mapsforge: 'Electrician', tag: 'shop=craft''
OAM category not found in Mapsforge: 'Emergency Access point', tag: 'hiking=yes''
OAM category not found in Mapsforge: 'Emergency Phone', tag: 'check_date=2018-05-31''
OAM category not found in Mapsforge: 'Emergency', tag: 'hiking=yes''

I think we can use your table and the one from mapsforge to fix the problem of not knowing what osm tag is relevant for each category, if we decide to split categories and icons, as I proposed. QMS can then look up in these tables what the relevant tag is and then display the corresponding icon. We could also implement something that allows the user to add/override lookups to support also other sources of *.poi files.

[wthaem]

I'll have a look! First job is: cope with plenty of snow.

[wthaem]

(The main job is still the non-ending snow.)

Nice bug or better configuration error. Thanks for finding!

There is a lot of "ele=999" tags that I wanted to exclude. And by this, the electricians were hit.

The "Emergencies" had a similar difficulty (access, phone were dropped).

There might be other similar problems in the file. That is, why I used the formulation "it is likely that ...".

Attached is a new version of the file.

BTW: the strange '' is just a formatting error.

Compare_OAM_Mapsforge_0802.txt

[hhornburg]

I think this is done.