Closed sananguliyev closed 3 years ago
Hi Sanan,
Thanks for adding a sample file, it really helps me.
I use the latest version of the parquet-avro Java library to read data - the issue is that it outputs the date field (received_at
) without any surrounding quotes, which in turn breaks the JSON library I use to parse the data:
The received_at
field is defined in the schema as:
"name" : "received_at",
"type" : {
"type" : "long",
"logicalType" : "timestamp-millis"
}
and I add support for this Logical Type with:
genericData.addLogicalTypeConversion(new TimeConversions.TimeMillisConversion());
I'm not sure why this library would output invalid JSON.
I will dig into it but I suspect I'll have to raise an issue on parquet-avro. In the meantime I will look into having the plugin not load the table
pane if invalid JSON is found.
It seems like the answer is for me to decode data into JSON rather than using toString()
. https://issues.apache.org/jira/browse/AVRO-2343. You'd expect this to be trivial but nothing is ever easy in Avro and Parquet. I'll take a decent stab at it this weekend.
Thanks for your investigation. I still could continue with int64
but not sure what will be the side effects.
I've asked Stack Overflow - https://stackoverflow.com/questions/63655421/writing-parquet-avro-genericrecord-to-json-while-maintaining-logicaltypes and I'll ask the Parquet mailing list if that doesn't get anywhere.
On Sat, Aug 29, 2020 at 8:36 PM Sanan Guliyev notifications@github.com wrote:
Thanks for your investigation. I still could continue with int64 but not sure what will be the side effects in the future.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benwatson528/intellij-avro-parquet-plugin/issues/43#issuecomment-683333424, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPNI2NJNELUQ4O3LGLIS63SDFKCJANCNFSM4QNMDBYQ .
I've uploaded a new version of the plugin that will at least let you view the raw data and schemas for affected files. The update will be available in the IntelliJ Marketplace within a couple of business days, or if you're very eager you can download it here.
Thank you very much @benwatson528
Hi,
Not sure the problem is in my parquet file or this plugin has some issues. The parquet file works fine when I read the file with Presto. I quickly checked the file and even works here in this site: http://parquet-viewer-online.com/
You add sample file and exception here: 405580a3-a4b9-4602-b576-84e6b20f2c3a.parquet.zip
Thanks in advance.