csv-import does the wrong thing when data includes unicode BOM

attic-labs / noms

The versioned, forkable, syncable database

Apache License 2.0

7.44k stars 266 forks source link

csv-import does the wrong thing when data includes unicode BOM #2044

Open aboodman opened 8 years ago

aboodman commented 8 years ago

I'm not sure whose fault this is, but:

!? noms show http://demo.noms.io/cli-tour::sf-bicycle-parking.value[0]
struct Row {
  Address: String,
  Geom: String,
  LastQ20EditedQ20Date: String,
  Location: String,
  MonthQ20Installed: String,
  NumberQ20ofQ20Racks: String,
  NumberQ20ofQ20Spaces: String,
  ObjectQ20ID: String,
  Placement: String,
  QEFBBBFYearQ20Installed: String,
  Street: String,
}

aboodman commented 8 years ago

Note the QEFBBB in front of Year installed

aboodman commented 8 years ago

Possible solutions:

Take off the ?includeBOM when fetching from socrata. I thought I saw that not having that caused other encoding issues, but we could try it again.
Fix the glitch.

arv commented 8 years ago

At this point we only support UTF-8 without a BOM. We should support UTF-8 with an optional BOM.