keithamoss / demsausage

Democracy Sausage
https://democracysausage.org
MIT License
9 stars 4 forks source link

Address geocoding cost issue #37

Open keithamoss opened 5 years ago

keithamoss commented 5 years ago

For the Victorian election users used our Google-based geocoder to make:

This didn't cost us anything because we get $200/month free from Google, but the total cost was $193 for the month. With NSW having a greater population we can expect this will start to actually cost money for their election.

Let's look at what we can do to address this.

e.g. By reducing how much we use the autocomplete geocoder e.g. Only when the user stops typing for a few seconds OR only when the user hits enter

If we want to look at other geocoders @stevage has a list here: https://getlon.lat

https://addressfinder.com.au came up at (you only get charged for geocoding requests, not autocompletes). Based on current pricing, we'd need the Pro 3 plan and pay $326 for 5.5k geocodes.

AB#116

davidjacooper commented 5 years ago

Is the autocomplete simply helping the user complete their own address? e.g. when I enter "swansea st" and get a short list of possible addresses?

If so, could we just ask them to enter their suburb or postcode?

keithamoss commented 5 years ago

Good thinking, yeah we could do suburb or postcode and then just host the data ourselves.

It may pay to implement it alongside https://github.com/keithamoss/demsausage-v3/issues/27 so that users can still pinpoint their house by eye on the map.

Or if we were feeling ambitious we could just grab GNAF via https://github.com/minus34/gnaf-loader and build our own geocoder :D

ineffyble commented 5 years ago

GNAF is surprisingly easy to set up and use for geocoding, from what I've heard.

stevage commented 5 years ago

Heh, depends what your expectations are. It's an incredibly complex dataset. And loading it (with the above tool) is only half the battle. Actual geocoding requires fuzzy text matching, search indexes blah blah. Even using the geocoder developed by Data61 is non-trivial. (I spent some time getting it running once. I got close.)

In any case, there is no need. If you don't need to store the geocodes (which, for privacy, I'm assuming is true), then the ArcGIS World Geocoding Service works well enough and is free. Mapbox's geocoder is good, but requires that you be using a 'mapbox map'.

You definitely shouldn't be paying anything with such "low" usage numbers on a free, public website.

keithamoss commented 5 years ago

Mapbox have tweaked their pricing structure and it looks (at first glance) to be significantly cheaper than what we're getting through the big G now.

https://blog.mapbox.com/new-pricing-46b7c26166e7

At the time of writing, details are still pending on the new "Session and user" prices for Geocoding APIs on the Pricing, but the API pricing (at the bottom) compares well to Google (Mapbox $75 vs Google ~$800 for 200k requests).

Looks like a good fit for the web UI places search and the data quality GUI idea. Not suitable for any sort of small geocoding jobs we want to do.

Edit: Actually, some of their pricing is now 10 times more than it was taken at face value. Need to factor in the difference in how they used to track map views (tiles) vs the new map loads (once per page load) though.

And the free tier for geocoding seems more generous vs the new (or rebadged?) Temporary Geocoding API.

Old pricing

MapBox Pricing Old

New pricing

Screen Shot 2019-05-26 at 17 51 34

keithamoss commented 4 years ago

Worth reviewing the Mapbox pricing changes to see if that makes it viable. But as always, it's the trade-off between "Run your own and add complexity" and "Hitch your wagon to someone else and risk changes beyond your control".

Via @andrewharvey at FOSS4G: https://github.com/komoot/photon http://photon.komoot.de

andrewharvey commented 4 years ago

Mapbox's geocoder is good, but requires that you be using a 'mapbox map'.

@stevage Not anymore, per the latest Terms of Service https://www.mapbox.com/legal/tos#[GAGA] if you're not using point of interest results (poi data type https://docs.mapbox.com/api/search/#data-types) then you don't need to use results on a Mapbox map, you can show results on any map.

@keithamoss Although the Mapbox geocoding engine is open source (carmen) the documentation to self host and ingest data is lacking so I wouldn't suggest it.

Photon works pretty well and not too hard to self host. It can ingest OpenAddresses.io data by way of https://github.com/openaddresses/oa2osm.

You're spot on though, always a balance between the maintenance cost, scalability, availability considerations of self hosting vs offloading this to another provider.

keithamoss commented 1 year ago

Hoping to tackle this one in 2023. Per earlier discussions, Mapbox still seems to be one of the best SaaS options available.

Tentatively hoping to get this in place before the NSW election in March.

  1. Analyse our current usage of Google's geocoded and map it to Mapbox's equivalent.
  2. Slot in Mapbox as a replacement.
  3. ...
  4. Save money?

Refs

https://docs.mapbox.com/api/search/geocoding/#geocoding-response-object

keithamoss commented 1 year ago

If we do one day go the "roll our own geocoder" route just for fun, consider the advice in Rules for Autocomplete.