search

NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

add wire format for fixed avro type #301

Closed jendap closed 10 years ago

jendap commented 11 years ago

FixedSpecific (and GenericData.Fixed) do not extend SpecificRecordBase so there's no implicit WireFormat for it.

BTW: This is not needed but one has to manually call .bytes() on the fixed fields but that's not too intuitive for most people. So we can just add it.

etorreborre commented 11 years ago

Can you create a pull request with this code please?

etorreborre commented 11 years ago

Sorry I forgot to ask. Can you please also add a small test case showing that the wire format works? Something along those lines. Thanks.

jendap commented 11 years ago

I was afraid you're going to ask for it! :-)

There's no test for avro record wire format. So I guess I shall put there sample avro schema into src/test/avro and enable the avro plugin ... pretty much the same way as it is in examples/avro, right?

jendap commented 11 years ago

I see, I'll to it the same way as you have in that AvroFileSpec.scala

jendap commented 11 years ago

I finally got to this...

There are trivial tests for SpecificRecord and SpecificFixed now. But - the patch now include generated classes. We should probably drop them and enable the sbt-avro plugin. Eric, can you do that please?

BTW: I've tried to do it without java or avro plugin. But avro looks for a static field SCHEMA$ and I don't know how to create static field in scala (see https://issues.scala-lang.org/browse/SI-4581)

There's no support for GenericFixed mainly because I'm not sure where to get the schema in fromBytes method (without serializing it with every single record) and the way it's being hacked in GenericRecord is an interesting hack to say at least ;-)