Dedupe ward data, and only add data if it doesn't already exist

swb-ief / etl-pipeline

The Covid Lens

1 stars 10 forks source link

Dedupe ward data, and only add data if it doesn't already exist #179

Open aswinjayan94 opened 2 years ago

aswinjayan94 commented 2 years ago

On days when the Mumbai PDF isn't updated on time, we end up scraping the previous day's PDF and appending duplicate data to the ward csv. This change tries to fix that