CosmosDb Sink Connector appends `_0` to key of protobuf `oneOf` object

[AlexKDawson]

Description

• Using the sink connector. Protobuf with a oneOf field causes field to have _0 appended to it's key.

Expected Behavior

• I would expect the key for the nested object to be the same in cosmos as in the protobuf.

Reproduce

• Configure the Cosmos Sink Connector with the following:

  config_nonsensitive = {
    "connector.class" : "CosmosDbSink",
    "kafka.auth.mode" : "SERVICE_ACCOUNT",
    "kafka.service.account.id" : var.SA_CONNECTOR_ID,
    "tasks.max" : "1",

    "name" : "CosmosDbSink_namespace-project-tripintegrations-dev",
    "input.data.format" : "PROTOBUF",

    "topics" : "namespace-project-tripintegrations-byvehicle-dev-public",
    "connect.cosmos.databasename" : "Project.Demand",
    "connect.cosmos.containers.topicmap" : "namespace-project-tripintegrations-dev-public#TripStates",
    "cosmos.id.strategy" : "ProvidedInValueStrategy",

    "transforms" : "idTransform",
    "transforms.idTransform.type" : "org.apache.kafka.connect.transforms.ReplaceField$Value",
    "transforms.idTransform.renames" : "TripId:id",
  }

• Feed data into the source topic with the following protobuf schema

  Project/Execution/TripByVehicle.proto

  syntax = "proto3";
  
  option csharp_namespace = "Namespace.Project.Protobuf.Execution";
  import "google/protobuf/timestamp.proto";
  import "Project/CommonMessages.proto";
  package Project.Execution;
  
  message TripByVehicle {
  string TripId = 1;
  .google.protobuf.Timestamp LastUpdateUtc = 2;
  ServiceLocation Origin =3;
  ServiceLocation Destination =4;
  .google.protobuf.Timestamp ScheduledDeparture = 5;
  .google.protobuf.Timestamp ScheduledArrival = 6;
  }

  Project/CommonMessages.proto // Supporting proto imported above

  syntax = "proto3";
  
  option csharp_namespace = "Namespace.Project.Protobuf";
  package Project;
  
  import "google/protobuf/timestamp.proto";
  
  message ServiceLocation {
  bool IsLocationCode = 1;
  oneof LocationDetail {
    string LocationCode = 2;
    .Project.Address Address = 3;
  }
  }
  
  message Address {
   string AddressLine1 = 1;
   string AddressLine2 = 2;
   string City = 3;
   string Country = 4;
   string FirstName = 5;
   string LastName = 6;
   string PostalCode = 7;
   string State = 8;
  }

  • Send data into the topic

• Cosmos will create the LocationDetail field with the key LocationDetail_0 in the container.

...
    "Origin": {
        "LocationDetail_0": {                                 <------------- The issue occurs here
            "LocationCode": "ABCDE",
            "Address": null
        },
        "IsCarvanaLocation": true
    },
    "Destination": {
        "LocationDetail_0": {                                 <------------- And here
            "LocationCode": "FGHIJ",
            "Address": null
        },
        "IsCarvanaLocation": true
    },
 ...

Additional Context

• If this is an intended behavior due to the use of a oneOf, could you point me toward the documentation please. I was not able to find any information on this happening

[kushagraThapar]

@xinlian12 please take a look.

[AlexKDawson]

Has there been any movement on this? Has this been reproducible?

[xinlian12]

@AlexKDawson this issue is in our backlog, plant o pick it up in the next few weeks

[AlexKDawson]

Thanks for the update!