Fix datasets availability

okfn-brasil / serenata-toolbox

📦 pip module containing code shared across Serenata de Amor's projects | ** Este repositório não recebe atualizações frequentes **

MIT License

154 stars 69 forks source link

Fix datasets availability #187

Closed cuducos closed 6 years ago

cuducos commented 6 years ago

This PR changes the datasets from Amazon S3 to Digital Ocean Spaces.

Following that change, a it added a bit of clean up:

As the idea is to use Digital Ocean Spaces, we have a nice URL accessbile to the whole team in the Digital Ocean dashboard to manage datasets files, so this PR completely remove the Datasets.remote object
Due to the same reason, this PR removes everything related to authentication (i.e. the toolbox is as read-only tool to the storage now)

In spite of that it is flexible: when the envvar points to Digital Ocean Spaces, it works. If this envvar is not set it works with Amazon S3 right away.

anaschwendler commented 6 years ago

What I did to test this PR:

Clone the project
Change to @cuducos branch:
Run the python console, and tried to download one dataset

Try to see the content of the download and it returned an error:

➜  /tmp cat 2016-12-06-reimbursements.xz 
<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>serenata-de-amor-data</BucketName><RequestId>tx00000000000000002668d-005ab50a22-1632fc-nyc3a</RequestId><HostId>1632fc-nyc3a-nyc</HostId></Error>%

cuducos commented 6 years ago

Many thanks for the review, @anaschwendler : )

Three minor changes:

I added an error if the dataset is not done (b5f23ff), so we don't end up with a dumb file as you did
Also I did a little clean up (0b0efbd)
I edit the file permissions in Digital Ocean, so I expect there's no NoSuchKey anymore

Would you mind testing and reviewing it after these changes?

anaschwendler commented 6 years ago

Ok! Now it is working! 🎉

Good one @cuducos \o\