EssDeeDee / safe-eats-etl

0 stars 0 forks source link

Design etl #3

Open astrocaribe opened 6 years ago

astrocaribe commented 6 years ago

Requirements:

No self-hosting; must be cloud-based
Guaranteed to process all inputs, not necessarily in order

Possible technologies/providers:

Amazon SQS
Eventador (Kafka-as-a-Service)

Phase 1: Ingest raw HTML

(Maybe this is a Lambda or something?)
Input: URL for one day of inspections
Output: Raw page HTML
Sinks: S3 bucket, phase 2

Phase 2: Parse HTML into fields

Input: Phase 1 output
Output: Food inspection data objects
Sinks: database (or first, geocoder?)

astrocaribe commented 6 years ago

@mattgawarecki @tjgragg Question for the masses: What would be the best format to enter the beginning/end dates for a query to this server? For example, I currently have: http://localhost:5000/scores?b=3/1/2018&e=4/1/2018 with b and e signifying the beginning end dates. This is just a shortcut from ?BegDate=3/1/2018&EndDate=4/1/2018 to make querying easier, but I'm still somewhat dissatisfied with this current solution.

astrocaribe commented 6 years ago

I'm also working on the food inspection data object; is json the best format to use? Because that's where I'm leaning.

tjgragg commented 6 years ago

JSON, please!

On Sun, Apr 8, 2018 at 1:21 PM Tommy Le Blanc notifications@github.com wrote:

I'm also working on the food inspection data object; is json the best format to use? Because that's where I'm leaning.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EssDeeDee/safe-eats-etl/issues/3#issuecomment-379570696, or mute the thread https://github.com/notifications/unsubscribe-auth/AOHMi0PAXrJZImHUYMNejet1VUy7d-Rsks5tmlUWgaJpZM4Snzk6 .