search

Data4Democracy / usa-dashboard

A dashboard of key metrics for the USA
69 stars 27 forks source link

data.world upload script #17

Open seanjtaylor opened 7 years ago

seanjtaylor commented 7 years ago

We'd like to be able to regularly scrape and upload data to data.world. This task is for a script that will push CSV file we scrape to data.world.

seanjtaylor commented 7 years ago

https://docs.data.world/documentation/api/addFiles.html

tdraebing commented 7 years ago

Hi, if it is ok, I would like to work on this. I think it would be nice to have a comprehensive toolkit to communicate with the data.world API. Thus my thoughts on designing the script are to keep it extensible, thus that further tasks like file syncing or getting files can easily be implemented. Also it might be nice to have two ways of interacting with the script, from code and from a CLI. For the first PR I would aim to implement the POST:/File-request and make it accessible by a simple function call from python code. More request types and command line usage would follow later. What do you think?

Also, I think such a tool/script might be useful for most projects here. Thus it might be a good idea to move it to separate repo in my opinion.

Cheers, Thomas

tdraebing commented 7 years ago

@tomkovarik Sorry maybe I did get something wrong, I am new to the project, but wasn't this issue just about having a script/tool to make it easy to upload files to data.world via its API? I don't see how this connects with a predictive value or so. Sorry I am a bit confused here.

tomkovarik commented 7 years ago

@tdraebing Whoops, wrong thread. I've deleted my comment to avoid further confusion. Please ignore.

tdraebing commented 7 years ago

Hi, I started implementing this. Since I am not yet a member of the D4D github organization, I created a repo on my own account for it for know. But I would be happy to transfer it to a repo here. Right now the python-module just wraps some data.world API calls, so that they can be performed by a single function call. There is still a lot to do (adding more request types, status_code handling, response formatting, building a CLI on top of it, etc....).

@seanjtaylor maybe you could have a quick look at it, if that is what you intended, when opening this issue.

Cheers, Thomas

rflprr commented 7 years ago

@tdraebing, to save you time, you may want to look at data.world's official Python package, as we've recently introduced wrappers for all our REST API endpoints, among other niceties that make it easy to download and work with data.world datasets. Look here: https://github.com/datadotworld/data.world-py and let me know if you have any questions.

tdraebing commented 7 years ago

Thanks @rflprr, that looks great. Then I will stop working on this.

@seanjtaylor I guess this issue can be closed in this case.