search

CAVaccineInventory / vial

The Django application powering calltheshots.us
https://vial.calltheshots.us
MIT License
13 stars 1 forks source link

Private API for loading in (and updating) locations #98

Closed simonw closed 3 years ago

simonw commented 3 years ago

Part of trying to move away from management commands (#27) - but also means that our teams that are finding new locations will have a way to pass them into the service.

POST /api/import-locations - then send it JSON with either a single location ({...}) or a list of locations ([{...}, {...}]).

Can use the import_ref key to update_or_create() items - that way we can run an import a second time to update the core details if we e.g. run a scraper again.

simonw commented 3 years ago

This will also let me rewrite ./manage.py import_vaccinefinder_locations from #73 as a client-side script that calls this API.

simonw commented 3 years ago

Here's the code I want to replace with this API:

https://github.com/CAVaccineInventory/django.vaccinate/blob/db68474114f87cce51e2a08dc3a67eb83c635b44/vaccinate/core/import_utils.py#L207-L231

And:

https://github.com/CAVaccineInventory/django.vaccinate/blob/db68474114f87cce51e2a08dc3a67eb83c635b44/vaccinate/core/import_utils.py#L32-L74

simonw commented 3 years ago

I'm going to limit the API to accepting a maximum of 100 locations at a time.

simonw commented 3 years ago

Initial documentation: https://github.com/CAVaccineInventory/django.vaccinate/blob/379cd7e4ea00fbc8eebd9d537ea60a306073956b/docs/api.md#post-apiimportlocations

simonw commented 3 years ago

This first version only supports the following fields:

https://github.com/CAVaccineInventory/django.vaccinate/blob/379cd7e4ea00fbc8eebd9d537ea60a306073956b/vaccinate/api/views.py#L347-L353

I need to support more fields - website and county and provider and suchlike.

simonw commented 3 years ago

Full list of properties from the model:

    name (done)
    phone_number
    full_address (the entire address, including city and zip code)
    street_address (the first line of the address) - I will derive this from full address
    city
    state (done)
    zip_code
    hours
    website
    location_type (done)
    google_places_id
    provider (ForeignKey to Provider)
    county
    latitude (done)
    longitude (done)
    airtable_id

I'm ignoring these for the moment:

    is_test_data
    soft_deleted
    soft_deleted_because
    duplicate_of
    provenance
    internal_notes
    do_not_call (boolean)
    do_not_call_reason
simonw commented 3 years ago

I can make county optional and fill it in from the latitude/longitude using the API trick from #76. Not in the next working version though.

obra commented 3 years ago

Over on Discord, I mooted a related kind of thing:

https://discord.com/channels/799147121357881364/813861006718926848/819081368348000256

@simonw (he/him) when you have breathing room, I'd love to chat a little about how to ingest location datasets of various stripes. sometimes we have nice lists from a government or a provider. that are places "known" to vaccinate sometimes, we have sketchy 3 year old federal datasets of likely locations sometimes we have geojson feeds from state health department arcgis servers. Do you think it would make sense to be bringing all of these things into a "places" table with the json magic bag of the original data on the side, sourcing info, and common characteristics and then magic happens and they're tied to locations in a locations table? that might let us slurp up feeds of potential locations and then reconcile them in-app.

On Tue, Mar 9, 2021 at 10:50 PM Simon Willison @.***> wrote:

I can make county optional and fill it in from the latitude/longitude using the API trick from #76 https://github.com/CAVaccineInventory/django.vaccinate/issues/76.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/CAVaccineInventory/django.vaccinate/issues/98#issuecomment-794980356, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALC2G47PI4EQ6UJ7IDFK3TC4CCZANCNFSM4Y4W5I3A .

simonw commented 3 years ago

Jesse: that's really interesting. Yeah, I think it makes sense for us to keep those independent of our core Location table, since that table is for de-duped, publication-ready data.

simonw commented 3 years ago

Still needed:

simonw commented 3 years ago

I had to add several APIs that return valid values that you might need to use in the importLocations API submissions:

simonw commented 3 years ago

This API is ready for people to start trying out now. Documentation here: https://github.com/CAVaccineInventory/django.vaccinate/blob/f78afbd1e5bbba4ddc4722c9a592dfd004cb143e/docs/api.md#post-apiimportlocations

Tomorrow I'm going to rewrite my Airtable and VaccineFinder location import management commands to talk to this API instead.

simonw commented 3 years ago

Idea: add a needs_review boolean column to Location and let people dump random stuff in there - then we can cleanup and deduplicate before it goes "live".

simonw commented 3 years ago

This is working well and has been used multiple times now.