Multiple task queues and second worker service

ResonantGeoData / RD-OpenGeo

Resonant GeoData primary deployment

https://www.resonantgeodata.com

Apache License 2.0

5 stars 3 forks source link

Multiple task queues and second worker service #1

Closed banesullivan closed 2 years ago

banesullivan commented 3 years ago

Continuation of ResonantGeoData/ResonantGeoData#285

Have two queues: one for running fast, high-priority ETL routines (e.g. populate_raster_entry) and one for running slow, low-priority ETL routines (e.g. populate_raster_footprint)

Note changes in ResonantGeoData/ResonantGeoData#294

On the Heroku deployment, we should also set it up so that we can easily spin up a second worker and handle each queue on separate workers (for now, we do not need this)

See ResonantGeoData/ResonantGeoData#326

banesullivan commented 2 years ago

Closing due to inactivity and no present need for this. The ETL routines in RGD have been greatly improved since this was originally proposed