[Versioning]: Explore Kedro + Iceberg for versioning

ElenaKhaustova commented 1 month ago

Description

At the current stage by versioning we assume mapping a single version number to the corresponding versions of parameters, I/O data, and code. So one is able to retrieve a full project state including data at any point in time.

The goal is to check if we can use Iceberg to map a single version number to code, parameters, and I/O data within Kedro and how it aligns with Kedro’s workflow.

As a result, we expect a working example of kedro project used with Iceberg for versioning and some assumptions on:

whether it solves the main task and what are the constraints;
how easy is to set up;
how the workflow looks like;
whether any changes are required on the kedro side;
what data formats are supported;
how easy is to work with local/remote storage;
how demanding is it in terms of dependencies.

Context

https://github.com/kedro-org/kedro/issues/4199

Market research

datajoely commented 1 month ago

This is slightly unscientific, but I trust the vibes in the industry enough to say Iceberg will clearly be the winner in long term.

Plus people saying things like this: https://www.linkedin.com/posts/michaelrosam_the-five-phases-of-a-successful-ai-data-strategy-activity-7252579389664587776-TgMo?utm_source=share&utm_medium=member_desktop

In my opinion this is a situation where we should really go all in on the technology rather than be super agnostic / on-size-fits-all. I'd love a future for Kedro where without much configuration persisted data defaults to this model.

noklam commented 1 month ago

@datajoely I actually took a stab on this a while ago. My experience with it is Delta has a more mature support than Iceberg at the moment in the Python ecosystem. for example the integration of ibis with iceberg is suboptimal. So from there I think Delta is gonna have a better performance with anything database related, AFAIK with iceberg it always load things in memory first.

One thing to note that these "versioning" are not as effective as we want. For example, an incremental change of adding 1 row will result in a complete rewrite in current Kedro dataset with Delta as well. For high-level versioning, they works very well with dataframe/table format.

The main challenge here I see is how to unify the "versioning" in Kedro, Kedro use a customisable timestamp, while Delta use a incremental version number (0, 1, xxx) or timestamp. Iceberg probably user something similar but I haven't checked.

datajoely commented 1 month ago

Delta is 100% more mature, Iceberg is the horse to back.

This is the thread I was trying to find earlier:

https://x.com/sean_lynch/status/1845500735842390276

I also don't think we should be wedded to that timestamp decision. It was made a long time ago and also has a non-trivial risk of collision. If we were doing that again we'd be better off using a ULID...

noklam commented 3 weeks ago

^ To be more specific, I was referring mainly to the python binding, i.e. PyIceberg and rust-delta(python). Iceberg itself is fairly mature, especially with the catalog etc, but the python binding seems to be lacking behind a little bit.

noklam commented 3 weeks ago

Any chance I can take this ticket or work together on this? I have explored this a little bit a while ago and would be a great opportunities to continue on it.

@merelcht @ankatiyar

deepyaman commented 3 weeks ago

I agree with @datajoely is the horse to back, at least from an API perspective. PyIceberg is maturing (it has moved significantly in the past couple years).

Realistically, I don't think Kedro should dictate whether you use Iceberg or Delta (or Hudi); that is a user choice, just like whether to use Spark or Polars. This is where unified APIs will ideally make implementation easier.

datajoely commented 3 weeks ago

So I'm actually being bullish and saying we should pick one of these when it comes to our idea of versioned data. We simply don't have capacity to integrate everywhere properly.

datajoely commented 1 week ago

Super cool application of these concepts

https://datamonkeysite.com/2024/11/10/smart-data-pipeline-design-check-for-delta-table-changes-with-minimal-overhead/

astrojuanlu commented 5 days ago

Realistically, I don't think Kedro should dictate whether you use Iceberg or Delta (or Hudi); that is a user choice, just like whether to use Spark or Polars.

I'm with @deepyaman on this one. There should be a layer in Kedro that is format-agnostic. We can be more opinionated in a higher layer.

astrojuanlu commented 5 days ago

What's clear though is that the Apache Iceberg’s REST catalog API has won for sure https://github.com/kedro-org/kedro-devrel/issues/141#issuecomment-2264794234

datajoely commented 3 days ago

I'm with @deepyaman on this one. There should be a layer in Kedro that is format-agnostic. We can be more opinionated in a higher layer.

I just want to warn against the noble pursuit of generalisation when there are times to pick a winner, I'd much rather pick a horse and do it well.

noklam commented 3 days ago

@ElenaKhaustova I have left some questions at the end since it's not a PR yet.

https://noklam.github.io/blog/posts/pyiceberg/2024-11-18-PyIcebergDataset.html

# Questions
- What does it means when we said " if we can use Iceberg to map a single version number to code, parameters, and I/O data within Kedro and how it aligns with Kedro’s workflow." Versioning code & parameters sounds more like versioning artifacts.
- How to efficiently version data? `overwrite` is a completely re-write. For SQL engine this is implemented by the engine that utilise API like `append`, `replace`. With pandas/polars it is unclear if it's possible. (Maybe be possible if it's using something like `ibis`)
- Incremental pipeline (and incremental data)
- Version non-table type, i.e. parameters, code(?), Iceberg support only these three formats out of the box: Apache Parquet, Apache ORC, and Apache Avro. Parquet is the first-class citizen and the only format that people use in practice.

ElenaKhaustova commented 3 days ago

@ElenaKhaustova I have left some questions at the end since it's not a PR yet.

https://noklam.github.io/blog/posts/pyiceberg/2024-11-18-PyIcebergDataset.html

# Questions
- What does it means when we said " if we can use Iceberg to map a single version number to code, parameters, and I/O data within Kedro and how it aligns with Kedro’s workflow." Versioning code & parameters sounds more like versioning artifacts.
- How to efficiently version data? `overwrite` is a completely re-write. For SQL engine this is implemented by the engine that utilise API like `append`, `replace`. With pandas/polars it is unclear if it's possible. (Maybe be possible if it's using something like `ibis`)
- Incremental pipeline (and incremental data)
- Version non-table type, i.e. parameters, code(?), Iceberg support only these three formats out of the box: Apache Parquet, Apache ORC, and Apache Avro. Parquet is the first-class citizen and the only format that people use in practice.

From the versioning research (https://miro.com/app/board/uXjVK9U8mVo=/?share_link_id=24726044039) pain points and summary, we concluded that users mention two major problems—versioning and experiment tracking. At first, we decided to focus on versioning. With it, the main user pain point was not to version a specific artefact as current kedro versioning allows so (not in an optimal way though) but to be able to retrieve a whole experiment/run. Meaning being able to travel back in time with your code and data and checkout to a specific version for the whole kedro project not just for the individual artifact.

Please see Kedro + DVC example for better understanding: https://github.com/kedro-org/kedro/issues/4239#issuecomment-2479038489

It's clear we can easily version artifacts (tabular data), but what about versioning catalogs/projects—more high-level entities and non-tabular data?

So the main questions are:

Does it make sense to use it for anything else but versioning tabular data?
Can we map artifacts snapshots with the git commit hash for example to retrieve the full project state or is there any other mechanism for that?

datajoely commented 3 days ago

My view:

Does it make sense to use it for anything else but versioning tabular data?

I'm willing to bet >95% of use cases fall into this.

Can we map artifacts snapshots with the git commit hash for example to retrieve the full project state or is there any other mechanism for that?

Now I've seen how elegant the dvc integration can be, maybe that's the right paradigm?

noklam commented 2 days ago

It's clear we can easily version artifacts (tabular data), but what about versioning catalogs/projects—more high-level entities and non-tabular data?

Iceberg (table) was designed for tabular data, so non-tabular is not supported.
Version object isn't something that Iceberg can do.
While version artifacts data is possible, Iceberg can do it in a much more efficient way (though not supported in my current implementation)

So the main questions are:

Does it make sense to use it for anything else but versioning tabular data?

My short answer is no.

Can we map artifacts snapshots with the git commit hash for example to retrieve the full project state or is there any other mechanism for that?

You can create "branch" and "tags" with Iceberg Table, but again that's for tabular data only. https://py.iceberg.apache.org/api/#snapshot-management

Other comments

Many experiment tracking tools start with metric tracking + git hash, then slowly adding data versioning & lineage as well. In general, a full reproducible experiment (whether or not this is important is a different story):

Python environment (freeze version + OS)
Code, this often mean git hash + uncommited changes. This can be combined with generating a git patch file + some GitOps.
Data versioning. This is the most tricky one. With artifacts, this is usually simpler, but it's extremely hard with tables or anything stored in database simply because state is not preserved (or anything that can be mutated externally). What I am seeing here is that DVC solves the artifact problem but cannot do much with tables.

With Iceberg, the metadata needed to be handled externally, i.e. a SQLite db that keep tracks of runs git hash + load_version of all the data (or their snapshot_id from the Iceberg table). So when user need to do time travel, it needs to specify the load version like this:

SELECT * FROM your_table_name TIMESTAMP AS OF 'YYYY-MM-DD HH:MM:SS'; # can be id as well,

kedro-org / kedro