frictionlessdata / datapackage-js

A JavaScript library for working with Data Package.
http://frictionlessdata.io/
MIT License
42 stars 15 forks source link

Set Delimiter Dialect #100

Closed Donskelle closed 6 years ago

Donskelle commented 6 years ago

Hey Dude, awesome work you've done. Thanks for that.

I have issues using csv data like this:

name;type;node_parameter;predecessor;successor;edge_parameter
wind-SH;volatile-generator;;;SH-electrictiy;{ub: 12, sequence: wind_profile, edge_type: [electricity]}
demand-el-DE;demand;;DE-electrictiy;;{sum: 2300, sequence: demand_profile, edge_type: [electricity]}

After i call infer on this csv, it tells me that:

Error: Number of columns is inconsistent on line 2

I guess it uses "," and ";" as delimiter. I tried this to solve my issues:

wind-SH;volatile-generator;;;SH-electrictiy;"{ub: 12, sequence: wind_profile, edge_type: [electricity]}"

`This resolves in :

Error: Invalid opening quote at line 2

Any idea to resolve it?

Donskelle commented 6 years ago

I fixed in kinda dirty way. I added a function to tableschema table class using firstline to read first line and set delimiter accordingly.

async function getDelimiter(source) {
  try {
    const headerLine = await firstline(source)
    const test1 = (headerLine.match(/;/g) || []).length;
    const test2 = (headerLine.match(/,/g) || []).length;
    if (test1 > test2)
      return ';'
    else
      return ','
  } catch(e) {
    return ','
  }
}