search

dataproofer / Dataproofer

A proofreader for your data
http://dataproofer.org/
691 stars 53 forks source link
cli command-line csv data-analysis data-mining data-science excel nodejs spreadsheet

readme

Dataproofer

""

A proofreader for your data

Every day, more and more data is created. Journalists, analysts, and data visualizers turn that data into stories and insights.

But before you can make use of any data, you need to know if it’s reliable. Is it weird? Is it clean? Can I use it to write or make a viz?

This used to be a long manual process, using valuable time and introducing the possibility for human error. People can’t always spot every mistake every time, no matter how hard they try.

Data proofer is built to automate this process of checking a dataset for errors or potential mistakes.

Getting Started (Desktop)

Download a .zip of the latest release from the Dataproofer releases page.

Drag the app into your applications folder.

Select your dataset, which can be either a CSV on your computer, or a Google Sheet that you’ve published to the web.

Once you select your dataset, you can choose which suites and tests run by turning them on or off.

Proof your data, get your results, and feel confident about your dataset.

Getting Started (Command Line)

npm install -g dataproofer

Read the documentation

dataproofer --help
>  Usage: dataproofer <file>

  A proofreader for your data

  Options:

    -h, --help          output usage information
    -V, --version       output the version number
    -o, --out <file>    file to output results. default stdout
    -c, --core          run tests from the core suite
    -i, --info          run tests from the info suite
    -a, --stats         run tests from the statistical suite
    -g, --geo           run tests from the geographic suite
    -t, --tests <list>  comma-separated list to use
    -j, --json          output JSON of test results
    -J, --json-pretty   output an indented JSON of test results
    -S, --summary       output overall test results, excluding pass/fail results
    -v, --verbose       include descriptions about each column
    -x, --exclude       exclude tests that passed

  Examples:

    $ dataproofer my_data.csv

Run a test

dataproofer data.csv

Save the results

dataproofer --json data.csv --out data.json

Learn how to run specific test suites or tests and output longer or shorter summaries, use the --help flag.

Found a bug? Let us know.

Table of Contents

""

Test Suites

Information & Diagnostics

A set of tests that infer descriptive information based on the contents of a table's cells.

Core Suite

A set of tests related to common problems and data checks — namely, making sure data has not been truncated by looking for specific cut-off indicators.

Geo Suite

A set of tests related to common geographic data problems.

Stats Suite

A set of test related to common statistical used to detect outlying data.

""

Development

git clone https://github.com/dataproofer/Dataproofer.git
cd Dataproofer
yarn

How You Can Help

Write a test

See our test to-do list and leave a comment

Add a feature

See our features list and leave a comment

Short on time?

See our smaller issues and leave a comment

Got more time?

See our medium-sized issues and leave a comment

Plenty of time?

See our larger issues and leave a comment

Creating a new test

Tests are made up of a few parts. Here's a brief over-view. For a more in-depth look, dive into the documentation.

.name()

This is the name of your test. It shows up in the test-selection screen as well as on the results page

.description()

This is a text-only description of what the test does, and what it is meant to check. Imagine you are explaining it to a remarkably intelligent 5-year-old.

.methodology()

This is where the code your test executes lives. Pass it a function that takes in rows and columnHeads

rows is an array of objects from the data. The object uses column headers as the key, and the row’s value as the value.

So if your data looks like this:

President Year
George Washington 1789
John Adams 1797
Thomas Jefferson 1801

Then the first object in your array of rows will look like this:

{ president: ‘George Washington’, year: ‘1789’ }

and so on.

Generally, to run a test, you are going to want to loop over each row and do some operations on it — counting cells and using conditionals to detect unwanted values.

Helper Scripts

Helper scripts help you test and display the results of Dataproofer tests. These are a small set of functions we've found ourselves reusing.

For more information, please see the full util documentation

""

Troubleshooting a test that won't run

Tests are run inside a try catch loop in src/processing.js. You may wish to temporarily remove the try/catch while iterating on a test. Otherwise, for now we recommend heavy doses of console.log and the Chrome debugger.

Iterating on tests

Dataproofer saves a copy of the most recently loaded file in the Application Data directory provided to it by the OS. You can quickly load the file and run the tests by typing loadLastFile() in the console. This saves you several clicks for loading the file and clicking the run button while you are iterating on a test. If you want to temporarily avoid any clicks you can add the function call to the ipc.on("last-file-selected", event handler in electron/js/controller.js

Release a new version

We can push releases to GitHub manually for now:

git tag -a 'v0.1.1' -m "first release"
git push && git push --tags

The binary (Dataproofer.app) can be uploaded to the releases page for the tag you pushed, and should be zipped up first (Right click and choose "Compress Dataproofer")

""

Sources

Thank You

vocativ-logo
knight-logo

A huge thank you to the Vocativ and the Knight Foundation. This project was funded in part by the Knight Foundation's Prototype Fund.

Special Thanks

... and the countless journalists who've encouraged us along the way. Thank you!