search

Netflix / iceberg

Iceberg is a table format for large, slow-moving tabular data
Apache License 2.0
476 stars 59 forks source link

Fix record name conflict when converting schemas to Avro. #38

Closed rdblue closed 6 years ago

rdblue commented 6 years ago

Using the last field name for the record name breaks when Avro tries to index a schema by record names, which happens in toString. The fix is to create record names using the record's field ID. This maintains stable record names, but ensures records within a schema never conflict.

rdblue commented 6 years ago

@arron-green, this is a fix for the problem you hit.

arron-green commented 6 years ago

Awesome! I’ll try it out and let you know

rdblue commented 6 years ago

Ah, this breaks things that rely on the old naming scheme. I kind of thought that might happen. I'll have to look into how to fix this, then... maybe just do this for nested layers.

rdblue commented 6 years ago

The fix was to update the aliasing that is done by the manifest reader. Everything works now, so I'll merge this.

arron-green commented 6 years ago

worked exactly as expected https://gist.github.com/arron-green/ae5147751a6c367583e9b273d8f00635