Refactor to pull from a timestamp onward

DiamondDAO / discord-etl

This program takes the output from `discord-bot` then transforms and loads into a database

0 stars 1 forks source link

Open nathankoerschner opened 2 years ago

nathankoerschner commented 2 years ago

Refactor ETL to store last parsed timestamp, then on future parses only look over data from that timestamp onwards.

nathankoerschner commented 2 years ago

Addressed in this commit, but needs to be tested.

Note that this assumes the inclusion of TIMESTAMP=0 in the programs .env file

nathankoerschner commented 2 years ago

@clemp this issue is ready for review.