[Epic] Datahub JS and Python SDK

[zelima]

As an admin, I want to have Python and JS SDK so that users can use datahub programmatically

As a user, I want to be able to write up a small code in Python or JS to communicate with Datahub, so that I'm able to publish or get datasets without manual interaction.

Acceptance Criteria

• [x] JS SDK is finished and well documented
  • [x] Has complete documentation
  • [x] Tutorials, getting started etc...
  • [ ] blog Post
• [ ] Python SDK exists and well docmented
  • [ ] Has complete documentation
  • [ ] Tutorials, getting started etc...
  • [ ] blog Post
• [ ] Track downloads

Tasks

• [x] Common APIs defined
• [x] Write JS tutorial "Publish/Deploy Data on DataHub [with JS]" => changes @dima
• [ ] Write same tutorial for python => list of requirements for python datahub-client-py (before even library exists)
• [ ] Write Python library

Analysis

Define Common APIs

As a programmer I need APIs and docs to be able to:

• [x] Get data
• [x] Post data
• [ ] List my datasets
• [ ] (optional) Search for datasets

Information sources

• Developers docs: https://docs.datahub.io/developers/api/
• Repositories docs:
  • https://github.com/datahq/auth#api
  • https://github.com/datahq/bitstore#api
  • https://github.com/datahq/specstore#api
  • https://github.com/datahq/metastore#api
• https://github.com/datahq/data-cli code itself

Output

The SDK instructions are written in the hackmd document, and later will be extended to a blog-post. (ask @acckiygerman for details

[AcckiyGerman]

@zelima @akariv I have a question about the SDK:

I understand that we should use our existing tools (like datahub-client) and the API as much as possible, so I should focus on describing "what we have ATM".

But also I think the existing way to publish data is a bit tricky (especially when pushing a single file - the descriptor should be created as well), and I'd like to simplify this.

Should I imagine the "simplified" SDK interface and discuss it with you, so we could implement it during this EPIC task ?

[zelima]

@AcckiyGerman please include analysis comments inside the issue description under analysis section as we always do (not as a separate comment). It's more readable

I understand that we should use our existing tools (like datahub-client) and the API as much as possible, so I should focus on describing "what we have ATM".

Yes we should focus on that, indeed datahub-client is our JS SDK at the moment, I think.

But also I think the existing way to publish data is a bit tricky (especially when pushing a single file - the descriptor should be created as well),

Not sure what do you mean by "descriptor should be created as well", but at the moment push takes descriptor and options as an argument and does its job... see https://github.com/datahq/datahub-client/blob/master/lib/utils/datahub.js#L34 It does not really care you are pushing single file or bunch of them. it just takes those arguments, performs set of tasks behind the scene and makes appropriate API requests.

Our job is to document all of its arguments properly and provide some examples. Eg what descriptor is (generally), what may options be. Also, describe errors, what's happening behind the scenes etc...

[AcckiyGerman]

After our comments yesterday I've realised that there are actually two tasks in the issue

• [x] As a user, I want to be able to write up a small code in Python or JS to communicate with Datahub, so that I'm able to publish or get datasets without manual interaction. That says me to provide just few simple examples (quick start)
• [x] SDK has complete documentation Which means we should describe the datahub-clientlib in details, like in the data.js/README.md doc.

Both tasks are important. I'm starting with the first one and this is an issue for the second task: https://github.com/datahq/pm/issues/136

Irakli Mchedlishvili 08:19 :+1: @AcckiyGerman only thing to add is that SDK documentation should also have usage examples

[AcckiyGerman]

datahub-client complete documentation: https://github.com/datahq/datahub-client/pull/38

[AcckiyGerman]

js sdk tutorial: https://github.com/datahq/datahub-content/pull/6

[zelima]

@AcckiyGerman could you please close this Epic as FIXED in this milestone and open a separate one for python as it will include a big chunk of work. in upcoming milestone

[AcckiyGerman]

JS part is FIXED:

• the tutorial http://datahub.io/docs/tutorials/js-sdk-tutorial
• the datahub-client lib documentation https://github.com/datahq/datahub-client/blob/master/README.md

Python part is extracted in a separate issue: https://github.com/datahq/pm/issues/137