richardogoma / bitcoin-rate-etl

An ETL pipeline to ingest near-real time data of Bitcoin rates across major currencies (USD/GBP/EUR) from the CoinDesk Bitcoin Price Index API.
MIT License
0 stars 1 forks source link

Duplicates in loaded data #6

Closed richardogoma closed 1 year ago

richardogoma commented 1 year ago

DUPLICATES This is due to a delay by the client in fetching the JSON payload before it is refreshed, so we have to use an UPSERT clause and UNIQUE constraint on the timestamp field to prevent duplicate records from being inserted.