The command line is our MVP. It is also the primary tool for our primary audience of power data wranglers. The command line is power, speed, and simplicity (for those familiar with it).
Approach: mechanisms over policies => push and get from local files (or even stdin and stdout) without even need for data packaging locally.
Note: we could allow simple package-name on understanding that means core/package-name
Current working directory (CWD)
"In a data package" => CWD contains a datapackage.json
Stdout
Stdin
File path ...
Sketch of Command Line interface
# push current directory (must have datapackage.json)
data push
# retrieve to current directory ()
data get publisher/package
data config
## ======================
# extras
# ls my packages (for my default publisher)
data ls
data info {pkgid}
# user and default publisher
data whoami
## ======================
## data wrangling stuff (later)
# see datapipes
data pipe cut {file}
Acceptance Criteria
[x] push implemented as per above
[x] get implemented as per above
[x] frictionless install experience (e.g.. a simple download of a binary)
[x] create skeleton datahub command with version and help commands #2
[x] create instructions #4
[x] enable Travis #3
[x] implement datahub get #30
[x] Extended version of get #5
[x] implement push (put / publish) - #6
Nice to have:
[x] info command to find out about a package #7
[ ] auto suggesting on mis-spelt commands #8
[ ] Bash command line completions #8
[x] [ux] coloring errors and output ... #9
[x] Naming the executable something good like data
Appendix
JS vs Python - SCQA
What's missing in the current command line
Situation: we have a python-based CLI that works
Complication: the python CLI does not cover these new features and has certain drawbacks like being hard to install reliably cross-platform
Question: should we extend the python CLI or build a new CLI using JS
Why use JS:
We can reuse much of our own JS code (that we are using in frontend)
We can interuse JS code in CLI and app front-end
Easier installation for different OS distributions (node seems to be better than python here)
More tools to preview data packages before publishing
Possibility to reuse code to build Desktop app (with js and electron)
(node may be somewhat more popular than python for this type of thing)
Old Analysis
## push ==============
# stdin - to your scratchpad data package
cat xyz.csv | datahub push
# stdin to data package
cat xyz.csv | datahub push my-data-package
# explicit path (less important!)
datahub push xyz.csv my-data-package
## get ==============
# by default simply outputs to stdin
datahub get {publisher}/{dataset}/{name}[.extension]
# (can also download to current directory (or even to stdin)
datahub get {publisher}/{dataset}/{name}[.extension] {dir or path}
Comments:
You don't even need a data package (or know data packages to get started)
=> server (or CLI) has to do job of generating this for you / updating datapackage.json as you add new files ...
would love to be able to get started with no config info at all (other than username?)
=> sandbox publisher anyone can publish to (if no username)
publisher (user) but no data package => publish to scratchpad data package within your publisher
Minimal nodejs based
datahub-cli
Command Line
The command line is our MVP. It is also the primary tool for our primary audience of power data wranglers. The command line is power, speed, and simplicity (for those familiar with it).
Approach: mechanisms over policies => push and get from local files (or even stdin and stdout) without even need for data packaging locally.
Concepts
publisher
/package-name
package-name
on understanding that meanscore/package-name
Sketch of Command Line interface
Acceptance Criteria
push
implemented as per aboveget
implemented as per aboveTasks
Prep and Planning
datahub-cli
Implementation
datahub
command with version and help commands #2datahub get
#30push
(put / publish) - #6Nice to have:
info
command to find out about a package #7data
Appendix
JS vs Python - SCQA
What's missing in the current command line
Situation: we have a python-based CLI that works
Complication: the python CLI does not cover these new features and has certain drawbacks like being hard to install reliably cross-platform
Question: should we extend the python CLI or build a new CLI using JS
Why use JS:
Old Analysis
Comments:
sandbox
publisher anyone can publish to (if no username)scratchpad
data package within your publisherConfig wanted: