databricks / spark-avro

Avro Data Source for Apache Spark

http://databricks.com/

Apache License 2.0

539 stars 310 forks source link

Reader writer improvements #283

Open smiklos opened 6 years ago

smiklos commented 6 years ago

This pr saves on string conversions by explicitly making the generic datum reader read strings instead of utf-8. Also, the converter copies every object from the record read by the reader but the iterator never actually reused any record instance. Now record is reused per iterator for efficient memory reuse

codecov-io commented 6 years ago

Codecov Report

Merging #283 into master will decrease coverage by 0.26%. The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master     #283      +/-   ##
==========================================
- Coverage   92.23%   91.97%   -0.27%     
==========================================
  Files           5        6       +1     
  Lines         322      324       +2     
  Branches       43       39       -4     
==========================================
+ Hits          297      298       +1     
- Misses         25       26       +1