search

databricks / spark-avro

Avro Data Source for Apache Spark
http://databricks.com/
Apache License 2.0
539 stars 310 forks source link

Read bytes and fixed avro types with decimal logicalType as DecimalType #268

Open peter-ungvari opened 6 years ago

peter-ungvari commented 6 years ago

From avro version 1.7.7 released in mid-2014, decimal numbers are supported with decimal logical type of bytes or fixed types: Avro spec. v1.7.7, current

Decimal values from avro files matching this specification are unable to read using current avro-spark releases.

Related issues:

257 - similar logicalType handling for string type

246 - writing decimals in this format

248

121 - PR

mzitnik commented 6 years ago

Hi @peter-ungvari

Having a problem with this issue spark-avro save data in BigDecimal type and reading back to DecimalType any idea when it will fixed or is there already a branch with this fix.

peter-ungvari commented 6 years ago

Hi @mzitnik ,

Branch is branch-3.2, but PR was not accepted.

If you are using version 3.2, you may try https://bintray.com/peter-ungvari/maven/spark-avro/3.3.0 This was released from my fork.

I don't think anybody will merge my PR anytime soon, see #276

After the project built successfully on my fork, Travis failed to download sbt api on PR in some cases: https://travis-ci.org/databricks/spark-avro/builds/356975870?utm_source=github_status&utm_medium=notification

[FAILED     ] org.scala-sbt#api;0.13.13!api.jar: The HTTP response code for https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/api/0.13.13/jars/api.jar did not indicate a success. See log for more detail. (294ms)

Thanks.