Determine what loaders we can/should port from VaccineSpotter

[Mr0grog]

As VaccineSpotter shuts down its API this week, we’ve been working to finish and move into production any API loaders that we’d prototyped but never used because we got better data from VaccineSpotter. We should also look to see whether we want to port any of VaccineSpotter’s loaders and run them ourselves.

We should be pretty choosy here — we want to support what we reasonably can, but should not take on an especially large maintenance burden where our partners don’t need it. Focus on loaders that:

• Cover a larger number of locations.
• Aren’t too complex (e.g. some loaders do some really complex PostGIS geo calculations to figure out to how to effectively query stores by zip code, and we probably don’t want to take on maintaining that, even if it’s some clever work).
• Don’t require a proxy. VaccineSpotter had a good relationship with several proxy providers, which allowed them to do more scraping than the rest of us by having its HTTP requests come from many different locations. Loaders that require a proxy like this to be successful are ones we don’t want to run.

VaccineSpotter source: https://github.com/GUI/covid-vaccine-spotter

[Mr0grog]

Albertsons seems extremely portable (yay!) although it relies precariously on a file that Albertsons publishes to S3 (so people like us can use it instead of scraping 😝).

[Mr0grog]

Walmart is definitely not — it’s an extremely complex implementation even they had trouble maintaining (and it turns out it largely has not been working for a month, since VaccineSpotter’s soft shutdown).

[Mr0grog]

For the sake of prioritization, here’s how many locations we have that have ever sourced data from VaccineSpotter (based on our provider field):

provider	count	Already have alternative
cvs	9436	✅
walgreens	8713	✅
health_mart	4967
walmart	4670
kroger	2866	✅
rite_aid	2416	✅
publix	1194
albertsons	1115
heb	576
sams_club	573
safeway	565
costco	538	✅
hyvee	286
southeastern_grocers_winn_dixie	215
weis	162
comassvax (PrepMod in Colorado)	108
wegmans	103
thrifty_white	96
price_chopper	83
pharmaca	27
southeastern_grocers_harveys	11
centura_driveup_event	9
denver_ball_arena	7
kta_super_stores	6
southeastern_grocers_fresco_y_mas	4

[Mr0grog]

Health Mart has not been operational since June.

Publix looks pretty complex and in need of proxying. Similar lift to Walmart.

H-E-B Pharmacy looks very portable.

HyVee looks possible.

Southeastern Grocers looks like it requires proxying, and most locations have not been updated in months (only a handful of locations have recent updates). Probably not worthwhile.

[Mr0grog]

Weis is the same under the hood as Southeastern Grocers. Probably not worthwhile.

Same for Wegmans.

(These use Enliven Health, which is the problem.)

[Mr0grog]

Thrifty White looks possible, but on the complex and harder-to-maintain end. The are not active in states we actively partner with (they are in IA, MN, MT, ND, and SD). Debatable value.

Price Chopper doesn't seem too complex, but the endpoints it utilizes aren’t returning any data. May need to check again during daytime hours. Earlier today, VaccineSpotter was returning data with recent updates.

Pharmaca looks like medium complexity. It does not cover very many locations, though.

Centura Drive-Up and Denver Ball Arena are both closed. We should make sure those locations are marked as private on our side.

KTA is medium complexity, and looks like it’s actively working. Only 6 stores and not in a partner state, though, so may not be worth the effort.

[Mr0grog]

Ones we should definitely port:

• Albertsons (#420)
• H-E-B (#421)
• HyVee (#422)

Questionable:

• Price Chopper ~(Need to revisit whether this works tomorrow.)~ Update: can’t get this to work; may require special SSL certificates. Probably not going to pursue.
• Pharmaca (Low benefit vs. effort)
• KTA (Low benefit vs. effort)

Other:

• Mark centura_driveup_event and denver_ball_arena locations as private in our API! This should probably just be a migration. (#423)