search

pelias / polylines

Pelias import pipeline for polyline (road network) data.
MIT License
17 stars 24 forks source link

feat(parser): Filter out URLs before sending to pelias/model #225

Closed orangejulius closed 5 years ago

orangejulius commented 5 years ago

We have had numerous reports from Pelias users about concerning error message during builds regarding the URL regex filter from https://github.com/pelias/model/pull/115.

While this filter is good, the resulting error message is alarming. Looking today at the output of a planet build, it appears that many of these errors come from the polylines file created by Valhalla out of the OSM street network.

Looking at the contents of the polyline file and corresponding record on OSM, it seems that Valhalla puts the contents of the ref tag in the polyline file as an alternate name. The ref tag will often contain a URL.

This means that not only will the error happen frequently, but many records that are actaully valid will be filtered out.

An example of this is the Iowa Women of Achievement bridge which is completely valid in terms of name, geometry, and tagging but contains a URL in the ref field. Screenshot_2019-07-03_13-03-36

However the resulting line in the polylines file contains a URL as one name:

ejqinArl~pqDkA|T?bQ\pH\nIxB`QdE~SIowa Women of Achievement Bridgehttps://www.principal.com/riverwalk/iowa-woman-achievement-bridge.htm

The polylines importer currently selects a single name value from the list of names in the polylines file by choosing the longest, which will often be a URL.

This PR adds an additional filter that first removes any URL-like values from consideration, and should completely eliminate any of the otherwise concerning errors while ensuring all valid records make it into Elasticsearch.

Fixes https://github.com/pelias/whosonfirst/issues/456 Fixes https://github.com/pelias/polylines/issues/216 Fixes https://github.com/pelias/docker/issues/89 Connects https://github.com/pelias/model/issues/116

orangejulius commented 5 years ago

I added another small change to this PR that ensures only records with a valid name will be sent to later stages of the import pipeline and therefore have Pelias model documents created.

This was needed because it's now possible for a valid polyline row (already defined in the code as a row with a valid polyline encoded string and at least one name) to have zero valid names after URLs are filtered out. This should also cut down on possible errors like those in https://github.com/pelias/polylines/issues/3