search

mnich0ls / evee-sd

An event aggregation app for events in San Diego
1 stars 2 forks source link

Infrastructure: Event scrapers (event aggregators) deployment #16

Closed mnich0ls closed 5 years ago

mnich0ls commented 5 years ago

Create a DO droplet for all web scrapers that use a similar framework i.e. all web scrapers written by @jremi use the same language and have similar dependencies and to me it makes sense to deploy on the same server (but I'm open to other/better ways!)

Create a deployment script (or use a deployment framework so that we can easily deploy changes to the scrapers/aggregators).

Configure these scrapers to run in the early morning, but ensure that the frequency and time can be quickly changed if/when it becomes necessary.

jremi commented 5 years ago

All of this will be most likely related to the Evee - Transformation and Load Services google drive doc.