[import] guess csv dialect

[rufuspollock]

As a Publisher I want to be able to provide a CSV with semicolon delimiters or other csv variations and have the app handle this automatically so that I don't have to set this stuff by hand in a datapackage.json

See e.g. https://github.com/datahubio/qa/issues/35

Acceptance criteria

• [x] data push my.csv with semi-colons, colons, pipes, tabs just works
• [x] data init also handles this ok

Tasks

• [x] Find a csv "sniffer" library - https://www.npmjs.com/package/csv-sniffer
• [x] Identify what variations you want to test for (and do we have sample files already somewhere e.g. tabulator-py)
  • [x] Separator
  • [x] Line endings
  • [x] quote character
• [x] Identify relevant place for its use - probably data.js
• [x] Use this in relevant places e.g. cli, desktop

Analysis

All cases failed, except comma separated file. Please, check url https://datahub.io/Mikanebu/test-data-for-different-separators/v/2.

Sample is located here: https://github.com/Mikanebu/qa-test-datasets Other samples: https://github.com/frictionlessdata/test-data/tree/master/data-files/csv/separators

Related issue: https://github.com/datahq/datahub-qa/issues/35

Also, if there is datapackage.json, that describes all files:

Error! The column header names do not match the field names in the schema

Variations

We want following delimiters to be supported: [',', ';', ':', '|', '\t', '^', '*', '&'] We also want to guess quote character along with delimiter - but still default to double quote

As per line endings, we don't want to do anything as CSV parser library we're using handles it internally.

Identify relevant place for its use

this should be implemented in data.js library:

• first implement a function that guesses parser options in /parser/csv.js which is the most relevant place
• we also want to re-use that function when adding schema so it's inferred correctly
• we also want to add "dialect" property into resource descriptor when adding schema

[anuveyatsu]

Working delimiters:

• comma: https://datahub.io/anuveyatsu/comma-delimited
• semicolon: https://datahub.io/anuveyatsu/semicolon-delimited
• pipe: https://datahub.io/anuveyatsu/pipe-delimited
• tab: https://datahub.io/anuveyatsu/tab-delimited
• tsv format with tab delimiters: https://datahub.io/anuveyatsu/tsv-extension
• colon: https://datahub.io/anuveyatsu/colon-delimited
• caret: https://datahub.io/anuveyatsu/caret-delimited

[anuveyatsu]

FIXED, available from v0.7.0 of the CLI tool - try the latest version here http://datahub.io/download

[AcckiyGerman]

Manual test with multi-delimeters dataset (from test-data repo) FAILED

• pushing is ok
• showcase page shows:

  File "/usr/local/lib/python3.6/site-packages/tableschema/schema.py", line 138, in cast_row
  raise exceptions.CastError(message)
  tableschema.exceptions.CastError: Row length 1 doesn't match fields count 3

May be the test dataset is not valid, anyway we need to investigate.

[AcckiyGerman]

Now works, it was an invalid descriptor in a test dataset FIXED:
User could push datasets and files with different delimeters.