AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
230 stars 75 forks source link

Feature Request: Use JSON Avro Spec for 'ToAvroConfig' #234

Closed 1ambda closed 2 years ago

1ambda commented 3 years ago

Hi, Thanks for the great open source. This project saved me today :)

It would be nice to support the provideReaderSchema function while building ToAvroConfig. Currently it's only supported by FromAvroConfig.

The reason why it's useful is,

Thanks!

cerveada commented 3 years ago

Hello, I am trying to understand.

1ambda commented 3 years ago

Hi! Thanks for the reply.

I am using auto-generated avro class which embeds JSON avro schema. For instance,

@org.apache.avro.specific.AvroGenerated
public class Inventory extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
  private static final long serialVersionUID = -3307908315669412535L;
  public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"Inventory\"... "); // JSON schema
  public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }

So I thought, we can provide the embedded schema like

    val configAvro: ToAvroConfig = AbrisConfig
      .toConfluentAvro
      .provideSchema(Inventory.getStringifiedJsonSchema(), registryId) // use auto-generated class's JSON schema here
      ...

I think it's better to describe the current use-case in my team for understanding.

Since I am new to confluent SR, I might misunderstand the architecture. Please let me know If It's impossible.

Thanks

cerveada commented 2 years ago

Sorry for late response.

I think you can use:

ToAvroConfig
  .withSchema(Inventory.getStringifiedJsonSchema())
  .withSchemaId(registryId)

This will create Abris configuration without trying to connect to schema registry at all. So it's your responsibility to make sure the reader is able to get the schema by the id you provided.

1ambda commented 2 years ago

Thanks for the information. Would be really helpful.