GeoCoder/GeoSearch tool (Apache SOLR based)

[novalis]

ETL from GTFS/Streets/OSM/etc... data over to SOLR for searching.

1. Extract -- launched by either mvn or ant (either as part of a build, or a stand alone running thing...)
   • POIs from GTFS (eg: stops, routes, etc...)
   • POIs from OSM
   • Street Intersections from .shp or OSM
   • etc...
2. Transform -- same time as Extract step above
   • Create XML format for SOLR.
3. Load -- look for a running instance of SOLR, and load
4. Check in standard SOLR config/ files and also solr.war (or do we use mvn to get solr.war -- would need to add solr.war to a repository somewhere).

Picture that solr.war and associated config / ETL tool would be part of api-webapp?

Frank: +1

[novalis]

+1

--novalis

[novalis]

I'm moving this to a milestone created just for geocoding: 0.7.

--novalis

[novalis]

+1. Having this would make OTP much more useful.

Also, Solr is in Apache's maven repo, so I don't think there's a need to check it in or host a copy on our own Maven repo.

--nicholasbs

[novalis]

Adding to the agenda for today so we can discuss how this should fit in with the rest of the project (i.e., new module, part of api-webapp, etc).

--nicholasbs

[novalis]

All agreed this makes sense as a new module.

Also agreed that for new modules we should get agreement via a vote prior to creating them.

--nicholasbs

[novalis]

Great to see this module on the way :) It would be great if it would offer an option to search not only by precise address (house number), but also by nearest intersection (street X, intersection with Y)

That would be an answer in cities where detailed house-level geodata are not available (city governments and private companies keep them proprietary) which is probably most common case. It would also respond to the fact that people in some countries use intersection information as the most common way of orientation (rather than door numbers). It is the case here in south cone (Uruguay, Argentina, Chile) and I imagine elsewhere.

See http://mapa.buenosaires.gov.ar/sig/index.phtml for an example of a system allowing to search by both direction as well as intersection (The 'official' map by Buenos Aires' city government): 'Altura o Intersección' lets you to enter house number OR intersecting street.

--karel

[novalis]

Might be worth taking a look at http://wiki.openstreetmap.org/wiki/Nominatim which already does reverse geocoding based on OpenStreetMap data.

It can calculate the lat/long of house/street numbers only if it's defined in the data, otherwise it can provide the street and its polygon.

--halex

[novalis]

Thanks, halax, I hadn't looked at that one closely (though I just realized someone has added it to the GeoCoders page).

I just tried its demo and the results I got were pretty spotty. Have you had good experience s with it?

--nicholasbs

[novalis]

Replying to [comment:8 nicholasbs]:

Thanks, halex, I hadn't looked at that one closely (though I just realized someone has added it to the GeoCoders page).

Woops, didn't see the [GeoCoders] page, my bad! Probably should've put it there instead of commenting here.

I just tried its demo and the results I got were pretty spotty. Have you had good experience s with it?

Well, works better than OSM Gazetteer IMHO, which can't even find my street: http://gazetteer.openstreetmap.org/namefinder/?find=Undercliff+St%2C+Neutral+Bay vs http://nominatim.openstreetmap.org/search?q=Undercliff+St%2C+Neutral+Bay&polygon=1

Also, I imagine the frontend application would make a call to the SOLR app directly?

--halex

[novalis]

Also, I imagine the frontend application would make a call to the SOLR app directly?

My thought was to implement something similar to the trimet search tool, where 3-characters -- try numbers -- starts a google suggest like search (this is pure SOLR, loaded with a master address file / landmarks / stops / routes / etc... data): http://maps.trimet.org/?tool=search

The upside to SOLR is that it's fast & easy to use...the downside is that it's not a true geocoder, and thus no street number interpolation. So if you're in an area with new construction or an incomplete address file, then users may get frustrated not being able to find their address (plus, the way the search algorithm in SOLR works, its not looking to match based on the nearby street numbers; that said, SOLR's search is pluggable, so address range searches could theoretically be implemented in SOLR).

--fpurcell

[novalis]

The post below is an attempt to consolidate the discussion around OTP Geocoder design, which has taken place both here as well as on the Dev list.

== High-level requirements ==

• A high-quality way to go from an address to a latitude/longitude.
• Support plot-level data and points of interest (not just to interpolate address ranges)
• Multiple matches for addresses, especially for cases where the correct match has low certainty
• Flexible geocoding module, so use of external geocoding services with varying standards can be easily configured
• Support for intersection-based geocoding (where precise address data is not available and/or orientation by intersections is common)
• OTP client code to be flexible enough to easily switch to using a local instance of GeoSearch in the future – FRANK: “and by easily switch, I'm thinking 10 lines of javascript code or less to override the default geocoder service / format / interaction model”

'''Geocoder specifications proposed initially by Frank''' (Solr-based geocoder for handling stops, intersections, and landmarks similar to what TriMet uses for their location search):

ETL from GTFS/Streets/OSM/etc... data over to SOLR for searching.

1. Extract -- launched by either mvn or ant (either as part of a build, or a stand alone running thing...)
   • POIs from GTFS (eg: stops, routes, etc...)
   • POIs from OSM
   • Street Intersections from .shp or OSM
   • etc...
2. Transform -- same time as Extract step above
   • Create XML format for SOLR.
3. Load -- look for a running instance of SOLR, and load
4. Check in standard SOLR config/ files and also solr.war (might not be necessary as SOLR is in Apache's maven repo).
5. There is a general agreement that geocoder should be developed separately as an individual module

'''Upsides and downsides of using SOLR:'''

• Up: SOLR is that it's fast & easy to use
• Down: Not a true geocoder, and thus no street number interpolation. So if you're in an area with new construction or an incomplete address file, then users may get frustrated not being able to find their address (plus, the way the search algorithm in SOLR works, its not looking to match based on the nearby street numbers; that said, SOLR's search is pluggable, so address range searches could theoretically be implemented in SOLR

== Other specifications that emerged along the way ==

• In the cases where the geocoder is a separate hosted service, it would just proxy over to the "real" geocoding service, and normalize the results. This should allow for the javascript to be pretty standard, because the server is adapting to the different services. It should also allow for an easy transition to use a custom self hosted geocoder.
• The javascript should be pluggable, in the case where the service offers a javascript api that can be used directly instead of going through an extra hop.

== Other lose ends – initiated development related to OTP geocoding ==

• Frank planning to get OTP's current map UI to interact with the OpenAddressing geocoder services (http://code.google.com/p/openaddresses/wiki/RESTService ). ''Any advance on that?''
• Frank started a project called GeoSearch http://code.google.com/p/geosearch in response to http://opentripplanner.org/ticket/154 , as a separate project because it made more sense then making it part of OTP. ''Any advance on that?''

== Existing issues regarding flexibility of geocoding configuration ==

• The geocoder should be pluggable on the back end to get around the cross site issues. AFAICT, not all the services have a javascript api, so we'd need to proxy through something to use them.
• ???

'''Possible solutions'''

• ?????

== Remaining questions ==

1. how should the javascript be configured? Options are:
   • as a pointer to a function. It would get called with whatever should get geocoded and would respond back with the results in a normalized format
   • ???

--karel

[mattwigway]

This is now external to OTP, correct?

[abyrd]

As I understand it, this is now handled by a separate OTP module which proxies requests to an external geocoder to respect the same-origin policy. I know some people have had a hard time finding an acceptable geocoder to use in their regions.

It is conceivable to index all the stop names from GTFS and street names from OSM in the graph, then use something like elasticsearch to provide a built-in geocoder in OTP (maybe even using street number information). That's a complicated project, so moving this ticket to "future".

[abyrd]

Closing, this has not been updated for 6 years and all current deployments seem to have geocoding functionality.