Include information from "Data items"

[Tigerfell]

There is a Wikibase instance running at wiki.osm.org. It includes structured data of tags and keys. I suggest add this information to your system as it might be more convenient than parsing wiki pages. There is a documentation at https://wiki.openstreetmap.org/wiki/Data_items.

CC @nyurik

[shathor]

Hi there, thanks for the hint. This would certainly be more convenient to fetch data from instead of parsing wiki pages. Though TagFinder gets the tag informations from TagInfo (https://taginfo.openstreetmap.org/taginfo/apidoc) and then builds a semantic web. Or do you mean that TagFinder items should include a link to this Wikibase? If so, does TagInfo expose this information aswell? Also does Wikibase contain relations among tags? That would be interesting to enhance the semantic web. I'm not familiar with Wikibase.

Kind regards, shathor

[Tigerfell]

Okay, I see. Data items contains some information which is not available from Taginfo. Wiki users can assign one group to each key/tag and document aliases ("this feature is also referred to as ...") which might be interesting for your search function. Just to give an example railway:etcs=*: http://tagfinder.herokuapp.com/search?query=railway%3Aetcs&lang=en https://taginfo.openstreetmap.org/keys/railway:etcs https://wiki.openstreetmap.org/wiki/Item:Q1729 (the entry for railway:etcs=*) I must admit, I recently updated this data item.

The concept of Wikibase is basically that you have an item (in this case a key) and then the wiki editors can add statements like "status" -> "in use". Data consumers can then use the API at https://wiki.openstreetmap.org/w/api.php to query the information. It is already used by the Wiki pages to display consistent information across translations and by the iD editor to provide a description of a tag.

MediaWiki Wikibase API doc: https://www.mediawiki.org/wiki/Wikibase/API

[sfkeller]

@Tigerfell I see the benefit of the OSM Wikibase, although the main job of TagFinder is to find tags, then pointing to other existing webpages.

@shathor Can you explain why http://tagfinder.herokuapp.com/search?query=railway%3Aetcs&lang=en isn't showing Wikipages?

P.S. @Tigerfell

• Is JOSM also using OSM Wikibase?
• And doesn't Taginfo also already store status of "in use" (https://taginfo.openstreetmap.org/projects ) ? Is there some sync'ing between of OSM Wikibase with Taginfo?

[Tigerfell]

There is a meta ticket in JOSM for Data items integration https://josm.openstreetmap.de/ticket/17842. Taginfo has numerous tickets regarding Data item integration

• taginfo/taginfo#248 (Use Wikibase data instead of parsing wiki text)
• taginfo/taginfo#263 (Add link to Wikibase / Data item)
• taginfo/taginfo#277 (Obtain formatter URL for OSM keys from Wikidata and OSM Wikibase) However, there is no direct synchronisation between Wikibase and Taginfo, both only take information from wiki pages, but Wikibase is also editable by itself and displays the information in the wiki, if there is no textual information on the wiki, which is then ignored by Taginfo, because it relies on a textual in-wiki information (#248). If I recall the discussion correctly, Taginfo maintainers are not interested in using in using non-statistic information other than the description (no groups, no aliases, no value validation regexes, ...), so there will always be information only available from one of the sources.

[shathor]

I will rerun the data update in due time and see if this particular key (railway:etcs=*) gets updated aswell. Last update was 04.11.19. I have to manually kick the updater and push the results to the cloud instance.

Useful to TagFinder would be the description if it isn't already in TagInfo (TagFinder also uses that description) and especially - as @Tigerfell mentioned - the "this feature is also referred to as ...". This information could be added as skos:altLabel if it is more like a label or maybe as an alternative description somehow if it is a longer text. The "Group" could maybe be added as skos:broader, so very interesting too. E.g. railway:etcs=* points to https://wiki.openstreetmap.org/wiki/Item:Q4675 which has more "referred as" (de: Bahn, Schienenverkehr, Eisenbahn, en: railway)

[sfkeller]

I have to manually kick the updater and push the results to the cloud instance.

Would it be possible to configure a Github Action and schedule an automated "updater workflow"?

[nyurik]

I have done some of the code to export data items into taginfo (as a sqlite db), but it is not fully finished yet. Hopefully soon, at which point data items will be the central structured metadata store editable by users.

[shathor]

I've updated the tagfinder data. For instance railway:etcs=* has more information now. Still no depiction as taginfo also does not provide one.

@nyurik Good work. Does it mean, that the dataitems will be available through taginfo API when done?

@sfkeller Potentially could work. However there's actually a batch job implemented, which would automatically periodically update the RDF and the index while the web app keeps running. It just doesn't work at the moment, because it runs on a free Heroku instance that does no allow writing files.

[nyurik]

@shathor yes, but my suggestion -- go to the source directly if possible. I need to look closer at what you do already, and will try to offer some thoughts on the easiest way to get the needed data.

Generally you can:

• If you want to get info about a few keys/tags/relations/roles (can do more than one at once), you can use OSM wiki API. This call gets two entries - Key:highway and Tag:higway=motorway -- api.php?action=wbgetentities&sites=wiki&titles=Key:highway|Tag:highway=motorway. Make sure you use it as described in my PR for the iD editor with all the extra params. The actual code that performs lookup is here.
• If you know the key/tag/relation/role Q-ID, can get it directly. So for Key:highway, which is Q335 you can get all of its data via https://wiki.openstreetmap.org/wiki/Special:EntityData/Q335.json
• Lastly, you can query for anything specific you need, e.g. crazy things like "give me the keys that have more than 100 values defined for them in the wiki", using the sophox service. See docs.

[shathor]

@nyurik @Tigerfell Would someone like to do this and have time to implement (some) of it? I could do the analysis, create sub-issues and guide through the code.