Read Mapsforge POI files

[hhornburg]

Important Notice:

Do not delete any of the sections. Read what is requested. Replace(!) the instructions by your answer(!). Feature requests reports that are not properly filed might be closed without further notice.

Choose a good title

What is a good title? The title will be listed in the changelog when the issue is fixed. Do you think someone else would understand what's fixed or done from your title? Does it really name the real topic of the issue?

• [x] Did you choose a good title?

Is your feature request related to a problem? Please describe.

When planning a trip it is convenient to be able to see POIs that are relevant to you on the map. Depending on the zoom level and style of the background map this is not always possible.

Describe the solution you'd like

Loading the poi files provided with the mapsforge maps as a project similar to the geosearch would enable users to display the pois that are relevant to them.

Describe alternatives you've considered

As an alternative to the .poi file one could load the .db file generated for locus maps. The .poi file seems easier to handle to me (both are SQLite files).

Give a reason why someone else should spend time on that request

I think POIs are relevant to most of the users and currently there is no fully integrated way to view POIs directly on the map. Having POIs loaded as a project would enable a user to directly copy those that are relevant to them into their respective project without having to manually fill in everything.

Additional context

Add any other context or screenshots about the feature request here.

[kiozen]

"Loading the poi files provided with the mapsforge maps as a project similar to the geosearch would enable users to display the pois that are relevant to them."

Can you provide more details on that? Maybe sketch some example dialogs? I would expect that the files could be loaded just like any other project. Therefore it would be an extension of the supported GIS data formats. But this sentence reads like you want to do something else.

[hhornburg]

Yes, I want that the files are loaded just like any other project. However, the poi file contains nearly 1 million pois only for Baden-Württemberg. Loading all of those would a) clutter the map b) hang QMS.

My solution idea was that there is a settings dialog for the project (just like there is one for the geosearch), where you can select the poi types that you want to view with tickmarks in a treeview. QMS then only reads pois of the selected types from the database and only creates gisItems for those.

As that still may be too much to handle, I am thinking whether it is possible to only load data that is spatially relevant to the user. An idea would be that only those are loaded, that are in the current view (which would have to be updated on every map move) and those that are in the bounding box of the other project. I am not sure though if that is feasible and a good idea.

[kiozen]

On outer zoom levels it's probably too much. Keep in mind that there are users planning long trips and to see those trips they have to zoom out quite a lot. Usually they are exactly those users that want POIs along the track and see them all. Ideally by focus follows mind :wink:

You probably would need some thinning algorithm like on the Garmin maps.

[hhornburg]

Let's take the discussion about this issue back here, as I'll close #296, since we will take a larger approach.

@kiozen wrote at the pull request #296:

Ok, here is my first impression:

It takes ages to load the POIs. I started to load 4000 POIs and QMS is loading now for several minutes.
While loading the cursor blinks between wait and arrow. It should stay as wait cursor. And a progress dialog must be shown with the chance to cancel
The default icon is ugly as hell for a reason: To make everyone to pick an appropriate icon asap. As there is no way to save a change and QMS is quite inefficient changing icons for several thousand points appropriate icons should be provided.
The POI format is missing in the list of supported formats (file dialog when opening a POI file)
The project is not restored in the workspace.
The number of POIs available only show up when expanding the items (users will complain)

I have a bit of a problem with such kind of "he-who-dies-with-the-most-wins" collections (aka Archies camp site list, aka Tonies Hüttenliste). They do exist, there are users that insist on using them, they are a huge pain in the butt because they stress the system. And it will be those users using these collections complaining that QMS is crappy as it does not perform very well. The experience from above simply confirms that experience from the past another time.

As already mentioned in the ticket it needs a bit more intelligence for such kind of databases. They can't be handled like any other GIS project file. Imho they are more like another feature next to maps, dem, routing etc. There need to be some algorithm to load and show only those POIs that fit into the viewport. And for outer zoom levels none or only selected POIs are shown. Additionally users want to search these databases and they want to spice their tracks with POIs along tracks. These are all requirements that are not really covered by the workspace. And for most of them the workspace as a place for user data to work with is not the right scope for the task.

I think working with POI collections is an interesting topic and is useful while planning. The more I think about it, it's an entirely new feature and should go into it's own docker widget. Just like maps or DEM data. POI collections can be activated or deactivated per view. Within that docker we are much more flexible to do stuff the right way as we do not have to stick to the rules of the workspace.

This will be quite a large change and I do not want to review it as one complete blob. This will need a feature branch next to dev. This allows us to create tickets that can be processed as a PR without providing a complete feature ready for the users. I have created such a branch in the main repo. It's name is feature/pois. Please use it instead of dev. I will rebase it after each PR on dev. Your PRs will target feature/pois in the main repo. You can create the tickets step by step in the normal tracker. Afaik there is no way to define tickets as epics with sub-tasks. Maybe it's good practice to reference the initial ticket with each new ticket.

Anyway the idea is to plan this step by step and to finish one step before you proceed to the next one. By that we can test and verify the implementation in smaller atomic steps. You do not have to plan all steps in advance. Stay agile and define ticket(s) that can be done in a reasonable time frame. They do not have to result into something that is ready for the user. By that it should be quite easy to tackle the beast.

[hhornburg]

I am thinking about how much of a detour we should take to make this run smoothly.

Can/Should we still make use of the IGisItem Classes? I am not completely sure (yet) how the triggers are set up. Using the GisItems would spare us of rewriting the code to render the POIs and make copying over of POIs to the workspace easier. However defining POIs in a more lightweight manner (no history etc) might also be good for performance.. (Especially as the majority of the POIs will never be copied to a project)

[mitxel-m]

A search for POIs will be very useful.

Here are some (crazy?) ideas:

If we need POIs around a track, why don't we look just X meters around from that track? This is by creating a buffer on the track and looking for elements of the DB that intersect with that buffer.

This is a task that I have to do from time to time in my work, where I use QGIS, and maybe a similar process can be done for QMS since it shares the same sqlite support and gdal base. So here is an example made in QGIS for the only purpose of illustrating the process:

I have loaded in the workspace mapsforge POI database specifically the one from the island of Mallorca , because it has only 12 Mb. I have added two tracks in the area, one is a circular route (typical of hiking), and the other one a very long and linear route(typical of cycling).

Then I can create a new virtual layer with this query:

SELECT * FROM myPOI WHERE st_intersects (geometry, (SELECT st_buffer(geometry,0.006,2) as geom FROM mytrack))

• myPOI is the DB.table with the POIs
• mytrack is the DB.table with the track
• 0.006 is the buffer radius in decimal degrees and equals approximately 450-600 m, (it varies with latitude but we do not need an exact value).

And here is what we get:

• The pink dots are all the POIs in the database.
• The orange lines are the tracks.
• The blue polygon is the buffer around the track ( I have put it in the image to illustrate but in the process it is virtual and not shown)
• The green dots are what the query returns, being a new table/layer in the workspace.

Of course that's the spatial filter part, and an additional filter based on an attribute could be added. (eg. searching for drinking water)

@pingurus, @kiozen, Do you find possible to perform a query like this in QMS with a SQL statement?

If the query can be done using the buffer it would be great.

Workflow proposal (Assuming that such a query can be made)

The search could be featured in the context menu as "Search items around", and the workflow for users could be as follows:

1. Right click on a track - ' Search items around'

2. A dialogue appears and prompts for:

   • Range (input buffer radius in meters, must be converted to decimal degrees)
   • Where to search ... - Browse for the DB
   • What to search - select categories, or/and type of items (wp, tracks, all) (may be a combo box)
   • Cancel Search

3. Once the search is done:

   • If there is no result: a notification appears "Nothing found"
   • If items are found: The same dialogue that alredy exists for new items appears, so you can choose where to put them.

   

   If there is an active project the items are copied in the active project without any dialogue. (just the same when new tracks, wp, are created...)

Remarks

This exceeds the initial approach of @pingurus but I think it is interesting with the following advantages:

• It would allow to search around any element of the work area: track, wp, area...

• You get at once the items around the track. In the initial idea based on a bounding box or the extension of the canvas, you will get for sure more items and probably need more time and effort to select the relevant ones.

• The same procedure could be used to search in a radius from a point clicked on the canvas.

• Maybe it could be extended to search your own DB in QMS format,. Then, and in combination with the previous one: It would serve to bring directly to the workspace only the tracks that pass by a point on the map, (without the need to load the DB in the work space - select and copy items - close the DB)

About the query in the above example:

• The query can be modified to simplify the polygon of the buffer, which I think could be useful for a faster search on tracks with many km. I will try to check it with a very long track.

• The query of the example above is intended to make the buffer on a single element (eg a track). It can be discussed if it is more convenient to do it for an individual element, or to do it for the project (all tracks at once), or to consider both cases. In any case, the query for a table with several elements is also possible.

select * from myPOI where st_intersects (geometry, (select st_union (st_generalize( st_buffer(geometry,0.008,2) ,0.0012)) as geom2 from mytrack))

• Another alternative would be to perform that query using the ogr2ogr command with the -sql option.I have tried to do it by myself in a terminal to put here an example, but without luck. Probably I am making a mistake in the syntax.

[kiozen]

@pingurus There is not much sense in having treewidgets with literally thousands of items to iterate. They are just an overhead with no benefit for a user. No one will find anything useful in such a list. The way to go is the same as with maps.

You do not list the pois you list the poi-collections and their groups. A bit like in your setup dialog. When the viewport is rendered you do a query to select the pois with the viewport matching the selected categories.

Converting a poi into a waypoint can be done similar to how it is done for Garmin maps. And for selecting several pois we need something more intelligent like pois-along-track.

I could provide you with a framework (docker widget etc., draw items etc.) if you want. Guess I can do that pretty fast.

I also will add a new template to github as the current ones are not really appropriate for that kind of tickets.

[hhornburg]

@kiozen

I could provide you with a framework (docker widget etc., draw items etc.) if you want. Guess I can do that pretty fast.

That would be great, thanks!

@mitxel-m I like your Idea and I had something like that in mind as a next step.

However, QGIS uses PostGIS and PostgreSQL to be able to do such SQL queries. While Qt can read PostregSQL, I don't think that we can easiliy include PostGIS. QGIS uses some python scripts to connect to PostGIS, so I guess we might have issues with bundling QMS with PostGIS.

In that case I think we need to write the code to find the POIs near tracks on our own.

[kiozen]

@pingurus In the meanwhile you could identify the POI categories and groups we need icons for. @mitxel-m would you volunteer to draw icons?

[hhornburg]

As I think the most exhaustive source of POIs is OSM, I think it may be best to have the same categories as defined in that project. This would also have the advantage that we could use and "auto-assign" the symbols found here (any other map style based on osm that provides the symbols in the same systematic way would also work).

The Mapsforge POI files contain the OSM tags that each POI has. If we opt for using the categories defined by these tags we can ignore the categories that the POI file defines. It would still be better to use the .poi files instead of raw osm data, as unnecessary stuff is thrown out and we can read the .poi files via sqlite r-table to have a better performance.

EDIT: Another source of icons. Maybe it's the origin of those mentioned above. https://github.com/twain47/Open-SVG-Map-Icons

[kiozen]

Yes I think we should take OSM as source.

I started a PR. It's not complete yet but you can start on it's base. Once it's merged you have to rebase to feature/pois

[kiozen]

Ok, done. I will give it a final review and a bit more testing and then I will merge it.

[kiozen]

@pingurus it's merged. You can start with CPoiPOI.

[mitxel-m]

@pingurus In the meanwhile you could identify the POI categories and groups we need icons for. @mitxel-m would you volunteer to draw icons?

Yes, I can help with icons. The second icon set that @pingurus pointed is good. POI categories and groups are different in orux .POI file and locust .db file, in orux POI file are about 300, and some of them have no icon in the icon set.

[mitxel-m]

As I think the most exhaustive source of POIs is OSM, I think it may be best to have the same categories as defined in that project. This would also have the advantage that we could use and "auto-assign" the symbols found here (any other map style based on osm that provides the symbols in the same systematic way would also work).

The Mapsforge POI files contain the OSM tags that each POI has. If we opt for using the categories defined by these tags we can ignore the categories that the POI file defines. It would still be better to use the .poi files instead of raw osm data, as unnecessary stuff is thrown out and we can read the .poi files via sqlite r-table to have a better performance.

EDIT: Another source of icons. Maybe it's the origin of those mentioned above. https://github.com/twain47/Open-SVG-Map-Icons

Regarding icons. @pingurus : good work colouring and converting to png

I have prepared two sets of waypoints that collect all POI types, so you can load them into QMS and see how they look on the map

You can download from here and take a look

https://mega.nz/file/1NVEEZ4T#Yl79b8CGDYes9dV7wXK1S5wQlqClWEz_exa7a5U7TSI

To test

• In QMS configure your custom icon folder pointing to th folder containnig those icons: qmapshack/src/icons/poi/png
• download the waypoint sets and open them in QMS, and take a look

There are 2 sets:

PIR : located at the Pyrenees GER : located in Germany

both them have two versions

• Regular : a grid in alphabetical order, easy to find each category.
• Random: As icons usual appear on a map

Most of them look nice, but there are others that I find difficult to identify. I think it worth to touch some of them to get a better view. I will edit and propose some changes specially for the landuse ones (color), and some others. like the poi_place ones (symbol). Color changes will be easy editing the XML with a search/replace.

Let me know if you find someone you want to change.

Of course this is a graphic question, and does not interfere with the development you are doing in QMS-#301

[hhornburg]

good work colouring and converting to png

thanks, but it wasn't me :laughing: I found a source where they were colored.

Most of them look nice, but there are others that I find difficult to identify. I think it worth to touch some of them to get a better view. I will edit and propose some changes specially for the landuse ones (color), and some others. like the poi_place ones (symbol). Color changes will be easy editing the XML with a search/replace.

Yes, that would be great. But before you make yourself to much work, double check if the icon you want to create/edit also has the tags included in the poi file (the landuse ones fore example only appear as by-catch, I think). I think I'll finish tomorrow with the tagMap (and thus I think I'll convert #302 to a regular PR) for the current icon set and afterwards I think anyone with interest in the topic can propose changes to that list since it is neither complete nor perfect. I tried my best, but if you have to choose tags for ~250 icons, you skip those where nothing comes to your mind quickly on what tags to assign. Also be aware that for some things included in the poi file there are no icons (e.g. leisure=park and some sports) and for some it might be debatable if it is useful to have them listed (e.g. fixme=yes).

[mitxel-m]

@pingurus thanks for the hints I took a look at CPoiPOI_TagMap.cpp and I think I got it. Just to verify, so I can continue on the right way:

1. Each line with Map [ ...] corresponds to a unique POI category in QMS, and needs its own icon.
2. The name of the icon file should match the exact name of the category in CPoiPOI_TagMap.cpp. Example: accommodation_alpinehut needs accommodation_alpinehut.png
3. For each QMS category there is a defined query , which will search in the data field of the poi_data table.
4. Lines commented with // are categories for which there is an icon, however no occurrence in the DB is found, and for now they will not be included.
5. We have found relevant categories in the DB that do not have an icon. Therefore we need both a new icon and add a new query for them. Examples: sport=climbing ; sport=scuba_diving ; tourism=wilderness_hut

Is this all right?

First I will do icons for:

category	(...) where data	comment
sport_climbing	like %sport=climbing%	there are a lot of rock climbing sites (not hill climbing)
sport_scuba_diving	like %sport=scuba_diving%	different to cliff diving
tourism_wilderness_hut	like %tourism=wilderness_hut%	see description in OSM, there are a lot
poi_waterfall	like '%waterway=waterfall%'	in the 'poi' family like poi_cave
poi_spring	like %natural=spring%' and data not like '%amenity=drinking_water%	in the 'poi' family like poi_cave

Any more? Let me know if someone else is needed.

I also have some comments about some queries in tagmap. Should I use the function to comment on lines, or do you prefer a separate text like this in the conversation panel? For me it is easier the second way

[hhornburg]

Lines commented with // are categories for which there is an icon, however no occurrence in the DB is found, and for now they will not be included.

Yes, those are categories for which an icon exist, but either there is no occurence in the DB (as you said), or I've found no good way to differenciate it from other categories with the OSM Tags. (See the parking ones for example)

Any more? Let me know if someone else is needed.

I think leisure=park was a frequent tag in the DB I opened and I haven't found an icon to which I could assign it.

I also have some comments about some queries in tagmap. Should I use the function to comment on lines, or do you prefer a separate text like this in the conversation panel? For me it is easier the second way

I think the way to go would be to open a 'sub-issue' for adding new categories and then you could comment in the pull request for that also on the existing lines. But as long as #302 is open, you can comment there as well (In text or at the lines, I don't mind).