Data Supply

Data Supply is an opinionated data processing, provision and presentation library of components for Javascript

❗ Please note this project is under early active development and probably not suitable for use yet.

Data supply supports NodeJS v12 and above.

Why use Data Supply

The data supply toolset helps you to create a front end Javascript application for data visualisation and manipulation quickly and easily. Using data supply will make it easier to spend time on the presentation of your data, by making predictable decisions in order to get your data from files to components.

If you use data supply on many projects your display components will have more predictable, consistent interfaces, and will therefore be more reusable.

What Data Supply is

Data Supply is a set of opinionated libraries and configurations that save time for front end Javascript applications that deal with presenting data sets.

It is extensible and flexible, and should be usable with any modern browser-based Javascript project regardless of framework, with any file format and with any display component or library.

Data Supply is usable with no knowledge of how it works, but it's also easy to reason about, investigate and extend should you need to.

What Data Supply is not

Data Supply is not an ORM or a database. It's not magic, and it's not a great leap forward. It does not do anything you can't do yourself using popular libraries. It just handles the boring stuff predictably so you don't have to.

Data Supply is not closed or restrictively licensed. Use it as much as you like on anything you like. If you do something useful with it, please consider opening a PR back into the librbary so others can benefit.

What Data Supply does

Data Supply consumes data from static file sources such as JSON and CSV files and allows you to predictably use that data in display components with a minimum of boilerplate. By using clear conventions and pragmatic defaults, Data Supply cuts out a lot of the boiletplate connecting data sources to stores and then via filtering and related functionality to display components.

Data Supply looks for a configuration file, and falls back to pragmatic defaults for any options that aren't configured. All the important decisions it takes can be overridden with configuration. All the important format- or framework-specific code it uses is pluggable and can be swapped out.

When Data Supply runs, it:

looks for data files in the project directory and converts them to its internal store using a sensible representation
creates filters that can be easily combined and extended to retrieve slices of data
creates components and functions that can be used to connect the data (filtered or not) to the component of your choice
uses headings in the data file as column / parameter names for predictable data retrieval

What Data Supply does not do

Data Supply does not make data relational, or know about relational data. It doesn't store data persistently. It doesn't make or receive network requests. It doesn't do any "magical" things that are hard to understand.

What Data Supply may come to do

Please see the roadmap.

Read more documentation

Usage

Installation

Install the package (or if you're developing locally npm link it)

Command line

data supply provides a CLI.

Type supply.

Starting at the place where you typed supply data supply will visit all subdirectories find all CSV and TSV data files (and possibly JSON metadata files named THE_SAME_AS_THE_DATA_FILE.meta.json c.f. conventions) and convert them to a single JSON file in the standard array-of-objects format:

{
  "data":[
    {
      "spreadsheet column 1":"value one",
      "spreadsheet column 2":"value two",
      "spreadsheet column 3":"value three"
    },
    ...
  ],
  "metadata":{
    ...
  }
}

supply will look for a .datasupplyrc file in the location from which it was invoked. If it doesn't find one it runs with default options...

{
  "excludeDirectories": [".next", "node_modules"],
  "fileTypes":[
    {"extension": "csv", "delimiter": ","},
    {"extension": "tsv", "delimiter": "\t"}
  ],
  "dataSourceDirectories": ["."]
}

To specify your own config file you can type supply --config ./path/to/my/supply-config.json

Paths specified in supply-config files are relative to the location of that file so for example if in the config you set

{
  "dataSourceDirectories":["spreadsheet-stash", "/global-data"]
}

then the script will look to source data from /path/to/the/config-file-directory/spreadsheet-stash, /global-data and all their subdirectories.

API

getFilePaths

parseDataFiles

Contributors ✨

Thanks goes to these wonderful people (emoji key):

_{Marcel Kornblum}
📖 🤔

_{Tom Pearson}
📖

_{Chris Camplin}
💻

This project follows the all-contributors specification. Contributions of any kind welcome!

signal-noise / datasupply

readme