search

mitchelllisle / sparkdantic

✨ A Pydantic to PySpark schema library
https://mitchelllisle.github.io/sparkdantic/
MIT License
53 stars 10 forks source link

Difference with pydantic-spark? #378

Closed timvancann closed 2 months ago

timvancann commented 2 months ago

There already exists a library that seems to have similar features: https://github.com/godatadriven/pydantic-spark Please provide some reason / differences to choose this over the other, or vice versa.

mitchelllisle commented 2 months ago

Hi @timvancann Thank you for the link! I wasn't aware of this project. It's always beneficial for the community to have options and nice to see others tackling the same challenges. I think its always positive where different solutions can coexist and cater to varied preferences and requirements and I don't think that just because something exists that excludes others from tackling similar problems.

timvancann commented 2 months ago

I never meant for this repo not to exist; I apologize if that was how my message came across. From a use perspective, I was merely looking for the differences between the two and when to choose which. Since this one is the newer (and more active repo), I incorrectly assumed that you've tried the other repo and found it lacking :).

dan1elt0m commented 2 months ago

Hey @timvancann, sparkdantic has feature parity with pydantic-spark and recommend to use sparkdantic. pydantic-spark is hard to maintain, because the code isn't clean and its author is not maintaining any of the libraries inside the godatadriven space anymore.