Initial /api/searchLocations API

[simonw]

We need a basic search API with the following features:

• Export everything (maybe by paginating, maybe even in one big authenticated lump) - needed for all sorts of productive import/matching/processing scripts
• Search by term (initially probably with LIKE but we can add PostgreSQL FTS soon afterwards)
• Search by concordance IDs

This issue is to get a basic framework up and running with those features. Will be adding plenty more features later on as and when we need them.

[simonw]

Initial design:

GET /api/searchLocations

Arguments:

• ?q= - search term, applied against name (and maybe against address too)
• ?latitude=&longitude=&radius= - search within a specific radius - using https://til.simonwillison.net/postgresql/closest-locations-to-a-point initially, later sped up using GeoDjango and PostGIS
• ?state=CA
• ?idref=concordance_id - an ID within the concordances table - can be passed multiple times for an OR search (or can be passed a comma-separated list)
• ?format= specify output format - you can use geojson or nlgeojson here, default is regular JSON. Might even support CSV.
• ?size=10 - to retrieve the first ten results
• ?all=1 - to stream ALL results, using the streaming keyset pagination mechanism I built in c340bd2f30d7759238c3042ca4e549243c0d8630

[simonw]

Default JSON shape will be:

{
  "results": [
    {"id": "...", "name": "..."},
    {"id": "...", "name": "..."},
    {"id": "...", "name": "..."}
  ],
  "total": 1023
}

This leaves room to add extra keys for things like pagination later.

[simonw]

This will start out unauthenticated but we may want to start requiring Authorization: Bearer xxx authentication later on, especially for expensive operations like ?all=1.

[simonw]

Now live on staging, docs here: https://vial-staging.calltheshots.us/api/docs#get-apisearchlocations

[simonw]

Concordances are now populated, so this API can grow options to search by concordance ID reference.

[simonw]

For concordance filtering this works:

locations = Location.objects.filter(
    concordances__in=ConcordanceIdentifier.objects.filter(
      Q(authority = 'cvs', identifier='#11344') | Q(authority = 'google_places', identifier='ChIJ3S7I3CE9hYARLpUyyZ21Q2A')
    )
)

[simonw]

I'm going to try and get the streaming option to work next. I'm bumping the lat/lon/radius thing to another ticket.

[simonw]

New feature demos:

• https://vial-staging.calltheshots.us/api/searchLocations?state=OR
• https://vial-staging.calltheshots.us/api/searchLocations?idref=google_places:ChIJ4VAdzhscwVQRrr4ZR3NhnN8&idref=vaccinespotter:7528796

[simonw]

• https://vial-staging.calltheshots.us/api/searchLocations?state=OR&format=nlgeojson now works - only returns first 10 results though

[simonw]

?all=1 is now live on staging!

curl 'https://vial-staging.calltheshots.us/api/searchLocations?all=1' > /tmp/wat.json

... that seemed to work! It gave me a 8MB JSON file with 10,562 locations in it

~ % cat /tmp/wat.json| jq .results | jq length
10562