frictionlessdata / example-continuous-data-integration

[DEPRECATED] Please use https://goodtables.io
12 stars 0 forks source link

Continuous Data Integration!

Build Status

It goes like this:

  1. You have a data package - like the one here.
  2. You install a small script for testing the data package - see scripts/test_data.py
  3. Configure travis CI - see .travis.yml
  4. Hey presto - your data will be checked on every commit or pull request

Examples

Here's some example of actual builds:

We also list the sample outputs below. You can configure error output to be in JSON or CSV rather than TXT.

Bad Structure

This "bad" CSV (won't get any dinner):

Date,Country,Number,
2015-01-01,3,20.3
2015-02-01,United States,23.5,,
2015-02,United States,x23.5,,,
,,

Results in:

+----------------+---------------+-------------+-------------------------------------------------------+
| result_name    | result_id     |   row_index | result_message                                        |
+================+===============+=============+=======================================================+
| Missing Header | structure_001 |           0 | Headers column is empty.                              |
+----------------+---------------+-------------+-------------------------------------------------------+
| Defective Row  | structure_003 |           0 | The row dimensions are incorrect compared to headers. |
+----------------+---------------+-------------+-------------------------------------------------------+
| Defective Row  | structure_003 |           1 | The row dimensions are incorrect compared to headers. |
+----------------+---------------+-------------+-------------------------------------------------------+
| Defective Row  | structure_003 |           2 | The row dimensions are incorrect compared to headers. |
+----------------+---------------+-------------+-------------------------------------------------------+
| Empty Row      | structure_005 |           3 | Row is empty.                                         |
+----------------+---------------+-------------+-------------------------------------------------------+
| Defective Row  | structure_003 |           3 | The row dimensions are incorrect compared to headers. |
+----------------+---------------+-------------+-------------------------------------------------------+

Bad Schema

This "bad" CSV (must be punished):

Date,Country,Number
2015-01-01,3,20.3
2015-02-01,United States,23.5
2015-02,United States,x23.5

Results in:

+----------------+----------------+-------------+-------------------------------------------------------------+
| result_name    |   column_index |   row_index | result_message                                              |
+================+================+=============+=============================================================+
| Incorrect Type |              0 |           2 | The value "2015-02" in column "Date" is not a valid Date.   |
+----------------+----------------+-------------+-------------------------------------------------------------+
| Incorrect Type |              2 |           2 | The value "x23.5" in column "Number" is not a valid Number. |
+----------------+----------------+-------------+-------------------------------------------------------------+