search

CivicTechTO / ttc_subway_times

A scraper to grab and publish TTC subway arrival times.
GNU General Public License v3.0
40 stars 30 forks source link

S3 token expired #59

Closed carolyz closed 5 years ago

carolyz commented 5 years ago

Running python3 fetch_s3.py --bucket ttc.scrape --start_date 2019-04-01 --end_date 2019-05-01 --output_dir some_local_dir from the README throws a ClientError

Full error message:

botocore.exceptions.ClientError: An error occurred (ExpiredToken) when calling the ListObjectsV2 operation: The provided token has expired.

Attempted with different date ranges and network and ran into the same issue. I ended up getting data from the links in the channel instead so it's all good.

radumas commented 5 years ago

Interestingly.... this is different than the error I've previously gotten when attempting this. I'm not entirely sure how to properly set up environment variables for the fetch script to work. This should probably be a warning :warning: in the README

carolyz commented 5 years ago

Interesting indeed. I would suggest mentioning and/or including the links in the channel as an alternative in the warning. Something along the lines of

The above is known to cause errors in the past. As an alternative, monthly data dumps can be found here

This is part of another discussion but I would be happy to set up some tables in Athena so the S3 data is in a more familiar format to some. Caveat is billing is based on amount of data scanned so not sure how financially feasible that is