Don't make duplicate requests for data in a given geographic area

[dmfenton]

• Koop-Yelp operates in a kind of pass-through mode currently. Where:
  • Each request for a different geography e.g. koop/yelp/pizza/FeatureServer/0/query?geometry={bbox} creates a new table in the db.
  • When loaded into AGOL, new tables are created whenever the map is panned or zoomed.
  • This is highly inefficient. Koop-Yelp should have access to what it already knows.

A strategy like the one pictured below would be one way to solve this.

Note: This functionality probably belongs in it's own Koop plugin but I think this might be a good place to prototype it.

Thoughts @chelm or @ajturner ?

[ajturner]

Making tables are comparably heavy. How about you make a single table for all Yelp or per tag (e.g. 'Pizza')

[dmfenton]

That's the idea. But Koop needs to be able to make requests and upsert the results into the table for any given map area. And it needs to do that with at least some level of awareness of the data it already has for the sake of efficiency

[chelm]

I think you can do this solely within the provider, and I agree that using query params to create tables is bad juju. We had a similar problem with the koop-vrbo provider where we ended up just using the the bbox as the table key. So the query to koop would only ever hit that bbox and agol requests would subset that. But that is not clean, or what you want to do here.

I think @ajturner is right that you should create a table for a url params like 'pizza' and add the smarts to the provider to basically negotiate inserts vs updates, etc.

But still this is really tricky and is going to get complex. Nevertheless its a problem we should and provide a patterned example for others to use.

I'll think more about it today.