Ultimately, I want to run the script regularly so that an updated plot
can be generated daily. However, the program re-fetched all puzzle data
on every iteration, which is wasteful and poor behavior.
Summary of Changes
Allow loading of existing data to reduce the number of requests needed
for subsequent program runs
Add database abstraction to store puzzle stats and load
previously-stored data from file. The database continues to be backed by
a CSV file under the hood, but switching it out to something else later
should now be simpler.
Add function to calculate the minimal search space to query puzzle
statistics for, skipping any dates that are already in the database.
Once this is properly hooked in, the entire puzzle history doesn't need
to re-fetched on every run, greatly reducing the number of requests
needed for incremental updates.
Allow parameters to be set via arguments using structopt crate rather
than relying on environment variables
Add logging via pretty_env_logger and log crates
Bug fixes
Add connect timeout to reqwest client
Increase concurrency of puzzle data fetching. Some operations were
unintentionally being serialized.
Issues Addressed
Ultimately, I want to run the script regularly so that an updated plot can be generated daily. However, the program re-fetched all puzzle data on every iteration, which is wasteful and poor behavior.
Summary of Changes
structoptcrate rather than relying on environment variablespretty_env_loggerandlogcrates