Bring Your Own Data - Githubissues

Bonkles commented 1 year ago

Description

Over the past couple of years we've fielded requests to have folks add their own GPX trace data, geojson data, or vector tile data to the map. This issue will likely be wide-ranging and I encourage folks in our user community to comment with specific use cases that you are interested in seeing come to fruition.

Currently, dragging and dropping Geojson onto the map will display the geojson in a non-editable layer. You can see it, but you can't do anything with it. If the user brought some .geojson onto the map, we might make a wizard-style importer that says: 1) Are you interested in bringing this data into the map? 2) What kind of data is it? 3) What OSM tags would you like to attach to the data?

We may want to include some sort of assertion that the data being brought is licensed correctly and/or owned by the uploader.

devdattaT commented 1 year ago

You could also have templates for the data, such that tags get carried over. Say the input geojson has properties with keys of highway and access. Then the way which gets created, automatically gets those tags, and values.

devdattaT commented 1 year ago

Our Use case: I work for an organisation that has a large fleet of vehicles which send GPS pings every 5 seconds. We internally use OSM for routing, and we have a process which identifies missing roads.

We want to share this back to OSM, but can't share our pings, or these missing roads directly. What we will like to do, is to have a team of mapmakers, who use these internal sources, and edit the main OSM API, using a feature such as the one described in the OP.

jakewarren commented 1 year ago

we might make a wizard-style importer

Just my two cents but I would prefer that importing operates more or less identical to the existing workflow for other RapiD sources such as Facebook Roads or Microsoft Buildings. Or in other words, RapiD would provide assistance for the user to quickly manually import specific entities since they would likely also be manually conflating the data. For large scale automated imports I feel that a tool like JOSM is more appropriate but that is just my opinion. This type of feature would be incredibly helpful for small scale manual imports.

I have created a crude mock-up to help illustrate better:

In this mockup the user has selected an object from a GeoJSON file. If the user clicks the 'Use this feature' button, a node/way is then created using the tags from the GeoJSON file. The changeset is then reviewed and submitted via the normal workflow.

tordans commented 1 year ago

In the spirit of sharing UseCases and Workflows, here are a few UseCases that I don‘t have an easy answer to … which RapID could help with and make a lot easier.

Building Footprints/Address

Berlin and Brandenburg both have OpenData about Buildings (including building levels) and Addresses. They are not suited for an import, since they only represent what the Cities know about the buildings officially. But they are a great starting point. It‘s the same with addresses, they need reviewing to be placed correctly (or not at all in cases where there are no houses and no construction on the ground).

AFAIK this is basically what the ESRI cooperation handles; but I never figured out how to get this data into this pipeline to use the core RapID features to handle them.

A „bring your own data“-RapID workflow could looks like this: We pre-process the OpenData in QGIS to filter it down to only missing data. We host a GeoJSON with the current version of that processing. We have a RapID URL with that hosted GeoJSON to start mapping. RapID would ideally allow some process to easily copy the external Data in OSM. A more fancy approach would be to prepare the data with https://github.com/ngageoint/hootenanny (at least I think this is a tool for this use case) to improve shapes and position of existing buildings., but I never looked into this.

General UseCase: Increase Trust in OSM Data

To share a motivation that is driving a lot of work with external data sources:

We work with administrations in Berlin and other Regions in Germany to help them use processed OSM data for their decision making. An important topic is, to build trust in OSM data in terms of „up to date ness“ and „completeness“. One proxy for those is to take different OpenData sets and compare them with OSM data to create tasks to check the Data. We use this for a number of UseCases. But the Workflow is complex and disconnected.

Zebra Crossings: We took two different open datasets, created a delta to OSM with buffer in QGIS, hosted the GeoJSON and created a MapRoulette Challenge. Each task was a zebra crossing that the city claims exists. Results where: Some where missing and added in OSM. Some where mistagged and corrected in OSM. Others where very badly positioned in official data, so OSM based on areal images was more precise. Other could not be found on areal images or Mapillary at all.

Detailed links (MapRoulette, Data, Processing) via https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-Zebrastreifen and https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-Zebrastreifen-Strassenbefahrung

Parklets: There is OpenData on Parklets in Berlin. We use the same process to add them. Since there is no Preset for Parklets, yet, we have a task list in MapRoulette that Mappers should follow when editing in ID/JOSM which is not a great way to explain what and how to map.

Detailed links (MapRoulette, Data, Processing) via https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-Parklets

Driveways: For our parking processing we process driveways (which translate to no parking for a few meters). Berlin has old OpenData on those. We use the same process described above.

We did not document this one, yet. Maproulette is at https://maproulette.org/browse/challenges/28466

Road side parking: Same as above but for Places where parking is only allowed in small areas next to the road. Again, the broader topic is to use the open data available and check it on the ground or with Mapillary to only import what we can tell is still present and still correct. In the end we can claim to have validated OSM data against the best official data available and checked that data in the process.

Detailed links (MapRoulette, Data, Processing) via https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Parkraum/Mapping_Kampagne_Xhain/QA-parkbuchten

Drinking water: Same story but different Group and Data: The goal is to make user all public drinking water places are in OSM and are well tagged.

https://atiptap.org/projekte/wasserwende/openstreetmap/, https://maproulette.org/browse/challenges/28303

Help with data preparation

A big issue with all those topics is preparing the external Data to create tasks for OSM. For most of the simpler datasets this can be done with QGIS with a few steps:

take data > take the center point > make a buffer > create OSM data > take the center point > make a buffer > find intersections of both > filter those without intersection > recreate point data of the buffer > polish the data properties (I like adding a custom task property with dynamic Mapillary Lookup that I use in MapRoulette as a Task Description).

There is no tooling out there, that makes this easier AFAIK. There is also a lack of direct tutorials on this stuff. So my nudge is, that improving external data handling in RapID might also be about improving the pre processing or creating some educational content (screenshare-sessions) on how to prepare data well.

General Idea: MapRoulette integration

Most of the Examples above use MapRoulette to store the status of Tasks and an overall Status of the Project. I like the fact that MR allows user to use the editor they like but still work on the same Tasklist. And that I get an overall status of the progress. So I wonder if a feature to handle external data in ID could be focused on integrating MR. At some point, I hope more editor – especially on the ground editors, see https://github.com/osmlab/maproulette3/issues/1737 – will pick up this theme and provide a data validation / adding experience that is even closer to the task in question (like: I am standing next to it…).

General Idea: Merge external and OSM Data

A low hanging fruit would be to make it easier to copy a set of tags that are provided within the external data to copy into an OSM object. That could be a „osm_tags“ property of a geoJSON (multilingual or array). Or just making the RapId UI a DIV instead of the table of readonly input fields.

More fancy would be to provide a UI to do some custom mapping, but that becomes too complex very fast… Even more fancy would be a way to select OSM + External Objects and have a merge UI/Action. (UIs that merge Address-Data could come to mind for this.)

General Idea: List next task nearby

One thing I missing when working with external Data as an overlay is something that will guide me to the next piece of data that needs adding. MapRoulette has a „next task nearby“ feature, but that external. External Data in ID can only be navigated visually (right?), so whenever the data is not dense (but spread around the city) it is hard to find the next „task“.

Hope that provides some inspiration.

Gustavo22Soaresh commented 1 year ago

One thing I missing when working with external Data as an overlay is something that will guide me to the next piece of data that needs adding. MapRoulette has a „next task nearby“ feature, but that external. External Data in ID can only be navigated visually (right?), so whenever the data is not dense (but spread around the city) it is hard to find the next „task“.

I believe that for this case it would be better if you didn't have to reload the ID or RapiD completely with each new task and also it would be nice to support Maprolette Cooperative Challenges

skunkman56 commented 1 year ago

This would be a very helpful option, to specify another Open Data API to pull from and assist with adding authoritative data

tsmock commented 1 year ago

The JOSM MapWithAI plugin has the ability to get data and add data from additional sources.

We should probably set down and put together a format for that -- right now, the plugin reads https://github.com/JOSM/MapWithAI/blob/pages/json/sources.json for all sources, and I don't know if the json format makes sense for Rapid, and it currently isn't set up for data sources that haven't set up mappings for their keys to OSM keys.

cbeddow commented 9 months ago

One very simple piece of this I want to suggest focus on is what @tordans mentions, an "osm_tags" property.

For example, if the user opens a local geojson in Rapid, it could have any format, and the geometry is displayed. however, if the Rapid editor senses it has "osm_tags", it should then enable the use/ignore buttons and be suggesting any tags in that. Could maybe do a tag lint check, but mostly just trust the input as-is. So format would look like:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "amenity": "restaurant",
        "name": "Casa d'Oro",
        "cuisine": "Italian",
        "addr:city": "Modena"
      },
      "geometry": {
        "coordinates": [
          10.45683300592367,
          46.06594839555103
        ],
        "type": "Point"
      }
    }
  ]
}

A PMTile, a vector tile endpoint, CSV or other filetypes could have the same expectation. Any key/value in the properties will be converted to OSM tags. If it does not exist in the presets... that's another issue. We could argue that at least 1 of the key/value pairs should be matching the presets or it throws an error, or could throw an error with an override option.

If a tile layer is used then the changeset should reference that tile layer as a source, and could otherwise reference the geojson filename as a source (which may be not useful anyway, but helps establish some possible trail of source evidence in case reverts or license checks are needed).

baditaflorin commented 2 months ago

I would love to suggest adding this so we can make the IDEditor as a simple way where other websites can point to this as an official way to add informations into OpenStreetMap in a simple way https://github.com/openstreetmap/iD/issues/10247

facebook / Rapid

Bring Your Own Data #585