search

elodina / go-avro

Apache Avro for Golang
http://elodina.github.io/go-avro/
Apache License 2.0
129 stars 55 forks source link

Has go<>java binary avro interop been verified? #54

Closed mrwilby closed 9 years ago

mrwilby commented 9 years ago

I have a basic avro schema that includes some nested records, enums, and some arrays & maps.

We are observing no issues when roundtripping this data purely in go. By this, I mean serializing a record to binary data using go and then deserializing the same data back into memory.

However, a simple java test program using the java 1.7.7 avro libraries cannot deserialize binary avro data written in go. The inverse is not working either, i.e. we are not able to deserialize (in go) data generated using the same schema (by java).

As far as I can grok from looking at the binary data generated by each runtime, java appears to generate more densely packed 'array' and 'map' count values which proceed the actual data.

I have a suspicion that java is generating one byte 'count' values and this library is generating two byte values.

Has anybody tried this kind of java <> go interop for binary serialized data (specifically, maps/arrays)?

I can possibly supply some sample java & go code.

serejja commented 9 years ago

Hi, we've been testing a rather simple case here but this does not include maps/arrays interop. I don't actually remember if we tested maps/arrays interoperability between Go and JVM.

If you attach an example we can sure debug and fix this issue. Thanks

mrwilby commented 9 years ago

Thanks. Here's some test code and sample output. Looks like this library can decode the java output, but not vice versa.

Here's the output from the go program (there is a similar program for java included):

go build; ./go-avro Generated binary avro data (Go)

Binary avro data: ../binary-avro/avro.go.bin

00000000 00 00 22 73 72 31 2e 74 65 73 74 2d 73 74 72 69 |.."sr1.test-stri| 00000010 6e 67 2d 31 22 73 72 32 2e 74 65 73 74 2d 73 74 |ng-1"sr2.test-st| 00000020 72 69 6e 67 2d 31 b4 0a 22 73 72 33 2e 74 65 73 |ring-1.."sr3.tes| 00000030 74 2d 73 74 72 69 6e 67 2d 31 22 73 72 33 2e 74 |t-string-1"sr3.t| 00000040 65 73 74 2d 73 74 72 69 6e 67 2d 32 22 73 72 33 |est-string-2"sr3| 00000050 2e 74 65 73 74 2d 73 74 72 69 6e 67 2d 33 22 73 |.test-string-3"s| 00000060 72 33 2e 74 65 73 74 2d 73 74 72 69 6e 67 2d 34 |r3.test-string-4| 00000070 00 00 00 08 31 32 33 34 22 73 72 34 2e 74 65 73 |....1234"sr4.tes| 00000080 74 2d 73 74 72 69 6e 67 2d 31 |t-string-1|

Read binary avro data (Go)

Binary avro data: ../binary-avro/avro.java.bin

00000000 00 00 22 73 72 32 2e 74 65 73 74 2d 73 74 72 69 |.."sr2.test-stri| 00000010 6e 67 2d 31 b4 0a 22 73 72 33 2e 74 65 73 74 2d |ng-1.."sr3.test-| 00000020 73 74 72 69 6e 67 2d 31 22 73 72 33 2e 74 65 73 |string-1"sr3.tes| 00000030 74 2d 73 74 72 69 6e 67 2d 32 22 73 72 33 2e 74 |t-string-2"sr3.t| 00000040 65 73 74 2d 73 74 72 69 6e 67 2d 33 22 73 72 33 |est-string-3"sr3| 00000050 2e 74 65 73 74 2d 73 74 72 69 6e 67 2d 34 00 00 |.test-string-4..| 00000060 08 31 32 33 34 22 73 72 34 2e 74 65 73 74 2d 73 |.1234"sr4.test-s| 00000070 74 72 69 6e 67 2d 31 |tring-1|

Read binary avro data (Java)

The java program blows up with an exception: Generated binary avro data (Java) {"subRecord1":{"map1":{},"string1":""},"subRecord2":{"string1":"sr2.test-string-1","int1":666},"subRecord3":{"string1":"sr3.test-string-1","string2":"sr3.test-string-2","string3":"sr3.test-string-3","string4":"sr3.test-string-4","enum1":"e1","map1":{}},"subRecord4":{"bytes1":"1234","string1":"sr4.test-string-1"}} Read binary avro data (Java) Exception in thread "main" org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -58 at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:363) at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:355) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157) at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) at avrotest.TestRunner.readBinaryAvro(TestRunner.java:85) at avrotest.TestRunner.main(TestRunner.java:109) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

Process finished with exit code 1

mrwilby commented 9 years ago

Committed the code here: https://github.com/mrwilby/avro-interop-test

serejja commented 9 years ago

Cool, thanks. Will probably take a look at it tomorrow on or Monday when I get other things done.

Will let you know on any progress.

mrwilby commented 9 years ago

Thank you. Looks good now!