search

asuiu / SparkORM

ORM for Apache Spark and DataFrames schema manager
https://pypi.org/project/sparkorm/
MIT License
12 stars 3 forks source link

Interesting idea #1

Open pycal opened 3 months ago

pycal commented 3 months ago

Hey @asuiu , I think this is a very interesting idea and attractive for a project I'm managing.

I'd be interested to know if you have plans to implement the various MigrationStrategies?

Also I was taking a look at some examples and if I'm not mistaken, in 1.2.24 some are no longer working because of recent changes (unless I'm mistaken?) Can you let me know if I should be able to do this?

from sparkorm import Struct

e.g. https://github.com/asuiu/SparkORM/blob/master/examples/conferences_extended/conferences.py#L3

asuiu commented 3 months ago

Hi! You are right, the from sparkorm import Struct doesn't work due to not exported Struct in the init. I have fixed this, and the example should work now, in the version 1.2.25, uploaded on PyPi as well.

Yes, I have plans to implement more MigrationStrategies. Do you have any specific examples in mind?

pycal commented 3 months ago

Nice.

Yes, I have a couple concrete use cases for migration strategies right now actually!

The simplest is "add new column to existing schema."

The other immediate use is "modify existing column type, given a specified cast":