Closed gygabyte closed 3 years ago
Hi Pedro,
This is because I use org.apache.parquet.avro.AvroParquetReader
to read files, and hyphens (-
) are invalid characters in Avro schemas.
I'm afraid that this isn't something that I can change, but replacing hyphens with underscores in your column names would resolve this.
Thanks,
Ben
https://github.com/mjakubowski84/parquet4s/issues/131 - if you update to parquet4s 1.1.0 then this issue will be resolved.
I hope this helps.
Ben
ok, sorry for the disturbance :) thanks a lot
Not at all, happy to help. It's surprising how inconsistent and weird all of the Parquet and Avro libraries are. When I made this plugin I thought it would be 5 lines of code and I'd never have to worry about it again!
I am getting this error when processing a file that is produced by the the scala library parquet4s. In this case it was written using snappy compression, but the same issue occurs for an uncompressed file
Thanks Pedro