Feature Request: Use JSON Avro Spec for 'ToAvroConfig'

[1ambda]

Hi, Thanks for the great open source. This project saved me today :)

It would be nice to support the provideReaderSchema function while building ToAvroConfig. Currently it's only supported by FromAvroConfig.

• https://github.com/AbsaOSS/ABRiS/blob/master/documentation/confluent-avro-documentation.md

The reason why it's useful is,

• Some users use auto-generated avro classes by downloading their Avro JSON schema from the registry once and create classes using CLI tools.
• Since schema registry's LIVE schema could change by topic provider, it forces consumer side code change too. Even though consumers don't want to update it now :( They just want to use previous (not updated) version of the schema until they decide to update it by them selves.
• But they don't want to update registry by calling 'provideAndRegisterSchema' since they are consumers, not provider of the topic.

Thanks!

[cerveada]

Hello, I am trying to understand.

• The consumers are reading form Avro and use FromAvroConfig therefore they can use provideReaderSchema, right?

• When using ToAvroConfig you are the producer and there is no reading happening, so don't know what would be the reader schema used for?

[1ambda]

Hi! Thanks for the reply.

I am using auto-generated avro class which embeds JSON avro schema. For instance,

@org.apache.avro.specific.AvroGenerated
public class Inventory extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
  private static final long serialVersionUID = -3307908315669412535L;
  public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"Inventory\"... "); // JSON schema
  public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }

So I thought, we can provide the embedded schema like

    val configAvro: ToAvroConfig = AbrisConfig
      .toConfluentAvro
      .provideSchema(Inventory.getStringifiedJsonSchema(), registryId) // use auto-generated class's JSON schema here
      ...

I think it's better to describe the current use-case in my team for understanding.

• We have the original source topic consumed by many teams. (Producer 1, Topic A)
• Usually we consume the avro payload from the original source topic (Consumer 1, Topic A)
• But somehow, we need to send the same payload for previous period restored from cold layer only for my team (HDFS or S3, ..)
• So operational producer sends messages to different topic, but with the same payload. But the additional topic doesn't need to be registered in SR since payload is the same (Producer 2, Topic B)
• (Consumer 1 for Topic A + B) process the messages by union data streams with the original topic for instance, streamOriginal.union(StreamOperation).foreachPartition( ... )

Since I am new to confluent SR, I might misunderstand the architecture. Please let me know If It's impossible.

Thanks

[cerveada]

Sorry for late response.

I think you can use:

ToAvroConfig
  .withSchema(Inventory.getStringifiedJsonSchema())
  .withSchemaId(registryId)

This will create Abris configuration without trying to connect to schema registry at all. So it's your responsibility to make sure the reader is able to get the schema by the id you provided.

[1ambda]

Thanks for the information. Would be really helpful.