chihacknight / chn-ghost-buses

"Ghost buses" analysis project through Chi Hack Night

https://github.com/chihacknight/breakout-groups/issues/217

MIT License

19 stars 14 forks source link

Update scraper script and automate aggregation #11

Closed lauriemerrell closed 1 year ago

lauriemerrell commented 2 years ago

Should merge after #8

Description

This PR:

[x] Updates the scraper script to write to both the public & private buckets (so data is more accessible but we still have a backup)
[x] Updates the scraper script to hit v3 of the CTA API so we can compare the results between the two
[x] Automates the creation of the daily aggregations

After this is merged, will need to take the steps outlined in #17

Resolves #10, resolves #9

Type of change

[ ] Bug fix
[x] New functionality
[ ] Documentation

How has this been tested?

I have done Lambda test runs for the new scraper & aggregator & verified that they appear to run as expected.

lauriemerrell commented 2 years ago

@dcjohnson24 -- I added a refactored version of the daily aggregations script that can be deployed & scheduled with Lambda -- let me know what you think?