beam-community / avro_ex

An Avro Library that emphasizes testability and ease of use.
https://hexdocs.pm/avro_ex/AvroEx.html
67 stars 27 forks source link

Doesn't support Avro 1.8.2 #10

Closed hats closed 2 years ago

hats commented 6 years ago

Hi! I'm having an issue when trying to decode a previously encoded message from another library. Schema:

{
  "type": "record",
  "namespace": "lead",
  "name": "CreateLead",
  "fields": [
    {
      "name": "lead",
      "type": {
        "type": "record",
        "name": "Lead",
        "fields": [
          {
            "name": "id",
            "type": {
              "name": "uuid",
              "type": "fixed",
              "size": 36
            }
          },
          {
            "name": "phone",
            "type": {
              "name": "phone",
              "type": "fixed",
              "size": 11
            }
          },
          {
            "name": "name",
            "type": "string"
          },
          {
            "name": "utm_source",
            "type": [
              "null",
              "string"
            ]
          },
          {
            "name": "utm_medium",
            "type": [
              "null",
              "string"
            ]
          },
          {
            "name": "utm_campaign",
            "type": [
              "null",
              "string"
            ]
          },
          {
            "name": "utm_content",
            "type": [
              "null",
              "string"
            ]
          },
          {
            "name": "utm_term",
            "type": [
              "null",
              "string"
            ]
          },
          {
            "name": "referrer",
            "type": [
              "null",
              "string"
            ]
          }
        ]
      }
    }
  ]
}

Encoded message:

payload = "Obj\u0001\u0004\u0014avro.codec\bnull\u0016avro.schema\xEE\t{\"type\":\"record\",\"name\":\"CreateLead\",\"namespace\":\"lead\",\"fields\":[{\"name\":\"lead\",\"type\":{\"type\":\"record\",\"name\":\"Lead\",\"namespace\":\"lead\",\"fields\":[{\"name\":\"id\",\"type\":{\"type\":\"fixed\",\"name\":\"uuid\",\"namespace\":\"lead\",\"size\":36}},{\"name\":\"phone\",\"type\":{\"type\":\"fixed\",\"name\":\"phone\",\"namespace\":\"lead\",\"size\":11}},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"utm_source\",\"type\":[\"null\",\"string\"]},{\"name\":\"utm_medium\",\"type\":[\"null\",\"string\"]},{\"name\":\"utm_campaign\",\"type\":[\"null\",\"string\"]},{\"name\":\"utm_content\",\"type\":[\"null\",\"string\"]},{\"name\":\"utm_term\",\"type\":[\"null\",\"string\"]},{\"name\":\"referrer\",\"type\":[\"null\",\"string\"]}]}}]}\u0000\xE7ƶ\xEAҀ\xE4\u001E\xBA\xB3\x972ޚ\xA1\xC6\u0002\x94\u000190ff2231-68fe-43fe-a1c2-ac24e12bd38279854538053\u0018Сергей\u0002\u000Eorganic\u0000\u0000\u0000\u0000\u0000\xE7ƶ\xEAҀ\xE4\u001E\xBA\xB3\x972ޚ\xA1\xC6"

When I try to dencode this object:

{:ok, json_schema} = @schema_path |> Path.absname |> File.read
{:ok, parsed_schema} = AvroEx.parse_schema(json_schema)
message = AvroEx.decode(parsed_schema, payload)

** (MatchError) no match of right hand side value: <<114, 100, 34, 44, 34, 110, 97, 109, 101, 34, 58, 34, 80, 101, 114, 102, 111, 114, 109, 101, 114, 67, 114, 101, 97, 116, 101, 76, 101, 97, 100, 34, 44, 34, 110, 97, 109, 101, 115, 112, 97, 99, 101, 34, 58, 34, 114, 117, 46, 115, ...>>                                                                                                                              lib/avro_ex/decode.ex:121: AvroEx.Decode.do_decode/3                                                                                                                                                        lib/avro_ex/decode.ex:127: AvroEx.Decode.do_decode/3                                                                                                                                                        lib/avro_ex/decode.ex:139: anonymous fn/3 in AvroEx.Decode.do_decode/3                        
    (elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
    lib/avro_ex/decode.ex:138: AvroEx.Decode.do_decode/3
    lib/avro_ex/decode.ex:139: anonymous fn/3 in AvroEx.Decode.do_decode/3
    (elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
    lib/avro_ex/decode.ex:138: AvroEx.Decode.do_decode/3

How do I fix this error?

doomspork commented 5 years ago

@hats can you check this against the latest master? We're working on improving the support for Avro.

davydog187 commented 2 years ago

Confirming that this is still an issue with the given data

davydog187 commented 2 years ago

Taking a closer look here, the issue is that we currently do not support decoding Object Container Files.

Will close as this is technically not a bug, will revisit after launching v2.0.0