pydantic_vs_protobuf_vs_namedtuple_vs_dataclasses

[utterances-bot]

Pydantic vs Protobuf vs Namedtuples vs Dataclasses

Personal blog and resume for Jan-Hendrik Ewers.

https://janhendrikewers.uk/pydantic_vs_protobuf_vs_namedtuple_vs_dataclasses.html

[bainsla]

Excellent comparison. 4 months later, are you still in love with Protobuf? The choice between Pydantic and dataclass doesn't look easy.

[iwishiwasaneagle]

Thanks @bainsla ! I still am, however my current project didn't end up needing it. The primary reason for looking into the different data types was because simply storing raw pydantic models was super inefficient. Now, protobuf would have been a decent solution buuut I then stumbled upon PostGIS which completely solved all my problems. See, storing millions of points as float64 was quite expensive (oops) but PostGIS has already thought of this and uses very efficient data types. I currently use SQLAlchemy and I'm still planning on exploring using protobuf, but for now it's Pydantic with until classes to do the conversions.

[4e1e0603]

"One of the largest downsides of using a data class is the additional performance overhead." Have you tried dataclass(slots=True), frozen=True?

[bryancole]

How about taking a look at Atom (https://atom.readthedocs.io/en/latest/). This is an observer framework (=data binding) usually used in the context of GUI-development but the emphasis on memory-efficiency and validation makes it ideal for data-classes. I use it everywhere I can (GUI or non-GUI).

[iwishiwasaneagle]

@bryancole cheers for the heads up! Will look into it.

[craig8]

@iwishiwasaneagle now that it's 2023 have you looked into pydantic-dataclasses... From what I read its dataclasses with the validation thrown in...not a full pydantic base class. Thoughts?

[iwishiwasaneagle]

I'll have a look into it! Planning an updated version of this blog anyway. Cheers!

-------- Original Message -------- On 30 Jun 2023, 03:12, Craig wrote:

@.***(https://github.com/iwishiwasaneagle) now that it's 2023 have you looked into pydantic-dataclasses... From what I read its dataclasses with the validation thrown in...not a full pydantic base class. Thoughts?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[juanmirocks]

In particular, Pydantic v2 promises a 5-50x speedup.

It would be nice to see your updated benchmark :-)

[EdmundsEcho]

Thank you for these useful benchmarks "as is". I'm often having to process large data sets so need lightweight solutions. I wonder if newer versions will benefit from more and more introduction of Rust hybrid solutions. Thoughts?

[iwishiwasaneagle]

@juanmirocks thank you for your comment! You inspired me to do a v1 vs v2 comparison which you can find here.

[iwishiwasaneagle]

@EdmundsEcho Funny you should comment on this post at the same time that I was working on this exact comparison! My findings are that yes, the rust-python hybrid has given the library massive speed gains.

https://janhendrikewers.uk/pydantic-1-vs-2-a-benchmark-test.html