Support for Pydantic V2 [Planned for Q1 2024]

[ravi-kumar-pilla]

Description

• This will bring pydantic v2 features to Kedro Viz

Context

• Pydantic v2 has changed its implementation to use Rust under the hood which makes it faster

Possible Implementation

https://docs.pydantic.dev/2.0/migration/

Timelines - We will work on the migration to pydantic v2 next year (Q1, 2024)

Possible Alternatives

NA

Checklist

• [x] Include labels so that we can categorise your feature request

[antonymilne]

I think this is not the right way to do things (or at least not for a long time). While the addition of model_post_init in pydantic v2 is very nice and would make it very easy to convert from dataclasses __post_init__, doing this would mean dropping support for pydantic v1, which I don't think we want to do for e.g. 3-6 months at least. The performance benefit of pydantic v2 will really be insignificant for kedro-viz and we don't especially need any of the features that pydantic v2 brings.

Besides, many of the things that are doing in __post_init__ in dataclasses are actually better suited to validators (note v1 documentation). Let me leave a few comments on https://github.com/kedro-org/kedro-viz/pull/1565/ to illustrate.

So what I would recommend is to support both pydantic v1 and v2 but use only the v1 API, as in https://github.com/kedro-org/kedro-viz/pull/1529. Then in due course convert from pydantic v1 to v2 API, which means you can then use model_post_init where useful, and drop support for pydantic v1 entirely.

[astrojuanlu]

This is because apache-airflow has moved to pydantic v2 in 2.8.0

https://github.com/kedro-org/kedro-plugins/pull/511#issue-2071944428

There is more pressure from upstream dependencies to support Pydantic v2.

[astrojuanlu]

Also to make things clear, supporting Pydantic v2 is not about performance (which for Kedro-Viz will have a negligible impact), but about allowing Kedro-Viz to coexist with Pydantic v2 in the same environment.

[butterlyn]

@astrojuanlu Hi, big fan of Kedro (and your latest YouTube video series), trying to convince my organisation to adopt the platform in day-to-day operations.

Is it possible for this to be released earlier in 2024 (perhaps Q2 or late Q1?)? Even in a non-stable development version? 2025 Q2 timeframe is a deal-breaker since our tech stack heavily uses Pydantic v2, and if I say to upper management that we'll need to wait to 2025 then I'll be fighting a losing battle when pitching Kedro. Thanks

[astrojuanlu]

Hi @butterlyn , thanks for your kind words! I'm almost sure 2025 is a typo, we definitely want to tackle this in 2024. Watch this space.

[ravi-kumar-pilla]

Hi @butterlyn , Thank you for your patience. As @astrojuanlu pointed out, we will be prioritizing pydantic v2 support in Q1, 2024.

[butterlyn]

@ravi-kumar-pilla Got the green light to trial Kedro after saying this feature was planned for Q1 2024. Thank you!! 😀 Will prepare to migrate in anticipation of the update

[antonymilne]

FYI vizro now supports pydantic 1 and 2 and uses the v1 API throughout - see https://github.com/mckinsey/vizro/pull/189.

Don't be put off by the size of the diff in the PR. It was actually very easy to do 🙂 Although we did not have any complications with having FastAPI as a dependency which I know was an issue on kedro-viz.

[astrojuanlu]

@markussagen also proposed migrating to Pydantic v2 in https://github.com/kedro-org/kedro-viz/issues/1719

[MarkusSagen]

Anything I can help with to speed up the timeline for this? Would you accept PRs for this?

@astrojuanlu

[astrojuanlu]

@rashidakanchwala any ongoing work that would be blocking this?

@MarkusSagen we'd be happy to have an external contribution for this! be mindful that there are several aspects to consider (FastAPI compatibility among them). you can read past discussion in https://github.com/kedro-org/kedro-viz/pull/1529, https://github.com/kedro-org/kedro-viz/pull/1634

[astrojuanlu]

In any case, let's wait for confirmation from the team - there are a couple of refactors ongoing and it might not make sense for you to open a PR against current main @MarkusSagen

[MarkusSagen]

Thanks for letting me know!

[rashidakanchwala]

Hi @MarkusSagen -- we will be doing this work in our next sprint which is Monday onwards. So hopefully we shld get this merged by 3rd or 4th week of Feb.

[noklam]

For the record, we decide to support V2 only going forward. For pydantic v1 user they should be able to use the older version of kedro-viz.