Refactor the existing dsv->json code and add CLI

[tomgp]

Based on feedback

Revisions:

• fixed problem with command not working with Node12
• switched to use .datasupplyrc for default config
• switched to JSON for the metadata
• renamed targetDirectories to dataSourceDirectories

Deferred:

• adding integration testing to catch problems with the CLI
• switching to use .datasupplyignore
• switching config loading to use cosmiconfig

Headlines

To see it working

• check out the branch and type npm link to "install" the module
• type supply this should run the script using the supply-config.json file at the root. output appears in ./example-output. Note the slightly different structure for output JSON and that the 'polling-data' example now includes metadata
• type supply --config ./test/example-data/config-fixture.json to try a different configuration file
• try out excluding directories or including different file extensions vis these config files

Because of the change to the JSON output format this PR is a breaking change

The detail

This PR has a number of changes from the current version. Mostly under the hood to do with refactoring but also adding the supply command and adding a means to add metadata

• Use fs and path methods where possible for checking file extension etc.
• The walking through of the directories to find target data files is now self contained i.e. it doesn't save the intermediate results to a global variable which can be a bit risky with this kind of recursive function)
• Added the 'supply' command to the package json
• make the configuration know about different different data delimiters and associate them with arbitrary file extensions
• Replace csv-parser with d3-dsv reasoning: we don't need to use streams 99% of the time and they make the code more involved/ difficult to reason with than just loading the hole file into memory in one gulp.
• add the ability to specify metadata (as YAML) and attach it to resulting JS object / JSON output file
• Split the code up differently to allow for easier reuse e.g. of file finding and configuration parsing
• add commander for passing in configuration
• add test data directories
• make tests that use that directory structure as a kind of fixture instead of mocking FS where appropriate

[marcelkornblum]

I was on Node 12 without realising and got

> supply
(node:4709) ExperimentalWarning: The ESM module loader is experimental.
file:///home/marcel/projects/datasupply/src/supply-cmd.js:8
import { Command } from 'commander';
         ^^^^^^^
SyntaxError: The requested module 'commander' is expected to be of type CommonJS, which does not support named exports. CommonJS modules can be imported by importing the default export.
For example:
import pkg from 'commander';
const { Command } = pkg;
    at ModuleJob._instantiate (internal/modules/esm/module_job.js:97:21)
    at async ModuleJob.run (internal/modules/esm/module_job.js:136:20)
    at async Loader.import (internal/modules/esm/loader.js:179:24)```

[marcelkornblum]

Firstly, with now I'm running v14, it seems to work really nicely. Could we please find a way to test for the v12 error I found so we don't get passing tests for that situation?

I've got a few syntactical / naming type thoughts... :)

1. I'm not a huge fan of supply-config.json - it feels more idiomatic to either use .datasupplyrc (my preference) or datasupply.config.json. Maybe we could allow a couple of different ones
2. is the targetDirectories config key the relative location of the source data files? If so it feels misnamed from the point of view of the user
3. IMO excludeDirectories would be better served with a distinct .datasupplyignore file (again idiomatically), perhaps optionally
4. Although I understand the reasoning, I personally find YAML very easy to make errors in, and I wonder if we ought not to start with all config in JSON rather than using different formats for different types of config? I suggest it's called e.g. polling-data.metadata.json
5. I think the data-specific config could do with a little more structure - it would be more flexible and future-proof if e.g. the fields were inside a fields object, with the title on a higher level object.
6. Specifically for that Date format in the data config, the way it's a second leaf on the branch for that field looks odd to my structural eye :)

[tomgp]

Firstly, with now I'm running v14, it seems to work really nicely. Could we please find a way to test for the v12 error I found so we don't get passing tests for that situation?

Think I should be able to fix that (I thought I had )

I've got a few syntactical / naming type thoughts... :)

1. I'm not a huge fan of supply-config.json - it feels more idiomatic to either use .datasupplyrc (my preference) or datasupply.config.json. Maybe we could allow a couple of different ones

Agree .datasupplyrc as the default is nice

2. is the targetDirectories config key the relative location of the source data files? If so it feels misnamed from the point of view of the user

relative to the the location of the configuration file or in absence of that relative to the place where the command is invoked (I think, though let me check :) )

3. IMO excludeDirectories would be better served with a distinct .datasupplyignore file (again idiomatically), perhaps optionally

I'm not sure about this, I like the idea of being able to have 2 different configs and of having as much config as possible in one place. we could also have an ignore file but then we'd need to to think about which took precedence

4. Although I understand the reasoning, I personally find YAML very easy to make errors in, and I wonder if we ought not to start with all config in JSON rather than using different formats for different types of config? I suggest it's called e.g. polling-data.metadata.json

Agree, I've done it with YAML in the past because it works well when using things like Jekyll, with our typical stack json makes more sense

5. I think the data-specific config could do with a little more structure - it would be more flexible and future-proof if e.g. the fields were inside a fields object, with the title on a higher level object.

this should be easier to do with JSON, I think there's a discussion to be had here about how prescriptive we want to be

6. Specifically for that Date format in the data config, the way it's a second leaf on the branch for that field looks odd to my structural eye :)

[marcelkornblum]

Think I should be able to fix that (I thought I had )

I'm surprised the tests passed though - do you think they are missing something or was my setup at fault?