apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
474 stars 97 forks source link

`field-id`'s missing in generated Avro files #353

Open Fokko opened 2 months ago

Fokko commented 2 months ago

Looks like the field-id is missing in the Avro metadata generated by iceberg-rust. This should conform the spec: https://iceberg.apache.org/spec/#avro

avro-tools getschema snap-0-1-a0c0a37d-6828-47c6-80e8-64b0a013fe44.avro
24/04/25 14:58:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{
  "type" : "record",
  "name" : "manifest_file",
  "fields" : [ {
    "name" : "manifest_path",
    "type" : "string"
  }, {
    "name" : "manifest_length",
    "type" : "long"
  }, {
    "name" : "partition_spec_id",
    "type" : "int"
  }, {
    "name" : "content",
    "type" : "int"
  }, {
    "name" : "sequence_number",
    "type" : "long"
  }, {
    "name" : "min_sequence_number",
    "type" : "long"
  }, {
    "name" : "added_snapshot_id",
    "type" : "long"
  }, {
    "name" : "added_data_files_count",
    "type" : "int"
  }, {
    "name" : "existing_data_files_count",
    "type" : "int"
  }, {
    "name" : "deleted_data_files_count",
    "type" : "int"
  }, {
    "name" : "added_rows_count",
    "type" : "long"
  }, {
    "name" : "existing_rows_count",
    "type" : "long"
  }, {
    "name" : "deleted_rows_count",
    "type" : "long"
  }, {
    "name" : "partitions",
    "type" : [ "null", {
      "type" : "array",
      "items" : {
        "type" : "record",
        "name" : "r508",
        "fields" : [ {
          "name" : "contains_null",
          "type" : "boolean"
        }, {
          "name" : "contains_nan",
          "type" : [ "null", "boolean" ],
          "default" : null
        }, {
          "name" : "lower_bound",
          "type" : [ "null", "bytes" ],
          "default" : null
        }, {
          "name" : "upper_bound",
          "type" : [ "null", "bytes" ],
          "default" : null
        } ]
      }
    } ],
    "default" : null
  }, {
    "name" : "key_metadata",
    "type" : [ "null", "bytes" ],
    "default" : null
  } ]
}
s-akhtar-baig commented 2 months ago

@Fokko, can you please assign this to me? Thanks!

Fokko commented 2 months ago

@s-akhtar-baig Certainly 🙌