joelansbro / pipeline

API Pipeline DB middleware
2 stars 0 forks source link

Pipeline: Separate Keyword job from database insert #33

Open joelansbro opened 2 years ago

joelansbro commented 2 years ago

The database insert at the end of the pipeline needs to be separated from the keyword job. Philosophically, the data that is cleaned in the pipeline should be stored "raw" into the sqlite db and they keyword job should be a job used afterwards, as part of the outbound API tasks.