search

andreantonacci / everynoise_scraper

Two webscrapers to collect data from everynoise.com
4 stars 2 forks source link

s3 create bucket #7

Closed hannesdatta closed 5 years ago

hannesdatta commented 5 years ago

@andreantonacci, please prototype with a bucket of yours; let me then know when you're done and I will edit the bucket name accordingly. i created one, but I don't know (yet) how to give you permission to it.....

andreantonacci commented 5 years ago

@hannesdatta the script is ready, you just need to change the bucket name to one of yours. No authentication is needed on the script itself, you just need a properly configured AWS CLI on the machine you're running it. It uploads only new files (those from the current crawling) since I imagined you are going to keep them all locally too.