search

saurfang / sparksql-protobuf

Read SparkSQL parquet file as RDD[Protobuf]
http://spark-packages.org/package/saurfang/sparksql-protobuf
Apache License 2.0
93 stars 36 forks source link

Fix schema translation for empty Messages #9

Closed AmailP closed 5 years ago

AmailP commented 5 years ago

When translating a schema containing an empty message the library was issuing an error of type

java.lang.IllegalStateException: Cannot build an empty group

Empty messages are sometimes used to fill old deprecated fields.

Since parquet does not support empty structs, IMO empty messages can only be ignored. Link to a related spark issue

codecov[bot] commented 5 years ago

Codecov Report

Merging #9 into master will increase coverage by 0.04%. The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master      #9      +/-   ##
=========================================
+ Coverage   98.46%   98.5%   +0.04%     
=========================================
  Files           5       5              
  Lines          65      67       +2     
  Branches        6       6              
=========================================
+ Hits           64      66       +2     
  Misses          1       1
Impacted Files Coverage Δ
...fang/parquet/proto/spark/sql/ProtoReflection.scala 100% <100%> (ø) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9febd8f...8330fd4. Read the comment docs.