search

kite-sdk / kite

Kite SDK
http://kitesdk.org/docs/current/
Apache License 2.0
394 stars 263 forks source link

Kite errors out when quoted csv header contains space #433

Open dipeshtech opened 8 years ago

dipeshtech commented 8 years ago

I tried kite sdk to create avro schema out of csv file using below command: kite-dataset csv-schema test.CSV --class Sample

It fails (when column header has spaces): "field 1","field 2","field 3" "Agwam","Agwam, MA","25007"

with following error:

Unknown error: Bad header for field, should start with a character or and can contain only alphanumerics and 0: "field 1"

Success example: "field1","field2","field3" "Agwam","Agwam, MA","25007"

Not sure if there is a way to handle it.

yawaramin commented 8 years ago

So, it looks like this is because Avro requires field names to obey the above specs. See https://github.com/kite-sdk/kite/blob/release-1.1.0/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/filesystem/CSVUtil.java#L179 for the exact code where this is happening.

If the field name doesn't match the specs, it's impossible to generate an Avro schema using that name. So this is a case of that more general issue. We can call this something like 'Kite CSV schema inference errors when headers don't obey Avro specs'.

One solution would be to slightly change the inferSchema and inferNullableSchema methods that currently take the Set<String> requiredFields parameter. We could make this Map<String, String> requiredFieldMappings to let the user specify Avro-safe names for all the required fields. Then inferSchemaInternal uses the mappings to generate the schema.

sshikov commented 6 years ago

We are often have files with just text in the header row, containing spaces, cyrillic letters etc, so field names inferred also doesn't match Avro's specification.

Our team proposal is to do some automatic transformation on such field names, like base32 encoding, plus generate schema doc using original column value.