apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
919 stars 147 forks source link

Test avro version compatability #484

Closed vinishjail97 closed 3 months ago

vinishjail97 commented 4 months ago

Important Read

What is the purpose of the pull request

(For example: This pull request implements the sync for delta format.)

Avro schema not being parsed with recursive update.

{
  "type" : "record",
  "name" : "HoodieCleanMetadata",
  "namespace" : "org.apache.hudi.avro.model",
  "fields" : [ {
    "name" : "startCleanTime",
    "type" : {
      "type" : "string",
      "avro.java.string" : "String"
    }
  }, {
    "name" : "timeTakenInMillis",
    "type" : "long"
  }, {
    "name" : "totalFilesDeleted",
    "type" : "int"
  }, {
    "name" : "earliestCommitToRetain",
    "type" : {
      "type" : "string",
      "avro.java.string" : "String"
    }
  }, {
    "name" : "lastCompletedCommitTimestamp",
    "type" : {
      "type" : "string",
      "avro.java.string" : "String"
    },
    "default" : ""
  }, {
    "name" : "partitionMetadata",
    "type" : {
      "type" : "map",
      "values" : {
        "type" : "record",
        "name" : "HoodieCleanPartitionMetadata",
        "fields" : [ {
          "name" : "partitionPath",
          "type" : {
            "type" : "string",
            "avro.java.string" : "String"
          }
        }, {
          "name" : "policy",
          "type" : {
            "type" : "string",
            "avro.java.string" : "String"
          }
        }, {
          "name" : "deletePathPatterns",
          "type" : {
            "type" : "array",
            "items" : {
              "type" : "string",
              "avro.java.string" : "String"
            }
          }
        }, {
          "name" : "successDeleteFiles",
          "type" : {
            "type" : "array",
            "items" : {
              "type" : "string",
              "avro.java.string" : "String"
            }
          }
        }, {
          "name" : "failedDeleteFiles",
          "type" : {
            "type" : "array",
            "items" : {
              "type" : "string",
              "avro.java.string" : "String"
            }
          }
        }, {
          "name" : "isPartitionDeleted",
          "type" : [ "null", "boolean" ],
          "default" : null
        } ]
      },
      "avro.java.string" : "String"
    }
  }, {
    "name" : "version",
    "type" : [ "int", "null" ],
    "default" : 1
  }, {
    "name" : "bootstrapPartitionMetadata",
    "type" : [ "null", {
      "type" : "map",
      "values" : "HoodieCleanPartitionMetadata",
      "avro.java.string" : "String",
      "default" : null
    } ],
    "default" : null
  } ]
}

(for example:)

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)