[epic] MVP in JS - Githubissues

Minimal nodejs based datahub-cli

Command Line

The command line is our MVP. It is also the primary tool for our primary audience of power data wranglers. The command line is power, speed, and simplicity (for those familiar with it).

Approach: mechanisms over policies => push and get from local files (or even stdin and stdout) without even need for data packaging locally.

Concepts

Data Package identifier - http://specs.frictionlessdata.io/data-package-identifier/
- Note: out of date now with data.okfn.org reference
DataHub package identifier (dhpkgid) = publisher/package-name
- Note: we could allow simple package-name on understanding that means core/package-name
Current working directory (CWD)
"In a data package" => CWD contains a datapackage.json
Stdout
Stdin
File path ...

Sketch of Command Line interface


# push current directory (must have datapackage.json)
data push

# retrieve to current directory ()
data get publisher/package

data config

## ======================
# extras

# ls my packages (for my default publisher)
data ls

data info {pkgid}

# user and default publisher
data whoami

## ======================
## data wrangling stuff (later)

# see datapipes
data pipe cut {file}

Acceptance Criteria

[x] push implemented as per above
[x] get implemented as per above
[x] frictionless install experience (e.g.. a simple download of a binary)

Tasks

Prep and Planning

[x] Define minimal datahub-cli
[x] Review existing code
- [x] https://github.com/frictionlessdata/dpm-js
- [x] https://github.com/zeit/now-cli
[x] Plan out MVP
- [x] Choose CLI library and approach - #26
- [x] Work out how we can compile for different platforms - https://github.com/zeit/pkg
- [x] Detailed analysis for key commands

Implementation

[x] create skeleton datahub command with version and help commands #2
- [x] create instructions #4
- [x] enable Travis #3
[x] implement datahub get #30
- [x] Extended version of get #5
[x] implement push (put / publish) - #6

Nice to have:

[x] info command to find out about a package #7
[ ] auto suggesting on mis-spelt commands #8
[ ] Bash command line completions #8
[x] [ux] coloring errors and output ... #9
[x] Naming the executable something good like data

Appendix

JS vs Python - SCQA

What's missing in the current command line

Situation: we have a python-based CLI that works

Complication: the python CLI does not cover these new features and has certain drawbacks like being hard to install reliably cross-platform

Question: should we extend the python CLI or build a new CLI using JS

Why use JS:

We can reuse much of our own JS code (that we are using in frontend)
- We can interuse JS code in CLI and app front-end
Easier installation for different OS distributions (node seems to be better than python here)
More tools to preview data packages before publishing
Possibility to reuse code to build Desktop app (with js and electron)
(node may be somewhat more popular than python for this type of thing)

Old Analysis

## push ==============

# stdin - to your scratchpad data package
cat xyz.csv | datahub push

# stdin to data package
cat xyz.csv | datahub push my-data-package

# explicit path (less important!)
datahub push xyz.csv my-data-package

## get  ==============
# by default simply outputs to stdin
datahub get {publisher}/{dataset}/{name}[.extension]

# (can also download to current directory (or even to stdin)
datahub get {publisher}/{dataset}/{name}[.extension] {dir or path}

Comments:

You don't even need a data package (or know data packages to get started)
- => server (or CLI) has to do job of generating this for you / updating datapackage.json as you add new files ...
would love to be able to get started with no config info at all (other than username?)
- => sandbox publisher anyone can publish to (if no username)
- publisher (user) but no data package => publish to scratchpad data package within your publisher

Config wanted:

username
token
server

datopian / data-cli

[epic] MVP in JS #1