feat: validate airbyte metadata added to iceberg schema

[jdpgrailsdev]

What

• Validate that Airbyte metadata columns are added to the Iceberg schema
• Actually detect insert vs update operation

How

• Add tests to verify that Airbyte metadata columns are added to the Iceberg schema
• Enable CDK tests to validate metadata is added to a schema when withMetadata() is called
• Convert IcebergUtil to a Micronaut singleton to better facilitate testing
• Detect insert/update when converting to Iceberg record

Review guide

1. IcebergUtil.kt
2. IcebergUtilTest.kt
3. IcebergV2WriterTest.kt
4. AirbyteTypeToAirbyteTypeWithMetaTest.kt

Can this PR be safely reverted and rolled back?

• [X] YES 💚
• [ ] NO ❌

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 22, 2024 6:03pm

[edgao]

this looks a lot like https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/data/AirbyteTypeToAirbyteTypeWithMeta.kt ? which we can maybe reuse in some way

[jdpgrailsdev]

this looks a lot like https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/core/load/src/main/kotlin/io/airbyte/cdk/load/data/AirbyteTypeToAirbyteTypeWithMeta.kt ? which we can maybe reuse in some way

@edgao It does and I wasn't sure how to reuse that given that it returns Airbyte types and not Iceberg types. I suspect what is really being pointed out here is some refactor to this in the CDK to make the common parts more common.

[jdpgrailsdev]

PR has been updated to validate that the metadata is added to the schema. I have removed the code that I originally added to do this, as the existing logic already adds the fields. This is confirmed via new tests added to this PR.