crflynn
commented
1 year ago I think that's just what the latest version was when I created this.
I just tested locally with python 3.8 and pyspark 3.1.1 and the tests all passed. I can relax it and make a new release for you a bit later today.
jelos98
Hello!
I'm interested in using
pbsparkin a project I'm working on. Unfortunately, I'm running in an spark environment that's not in my control, and as a result I need to be able to run with pyspark==3.1.2, which is problematic given that the requiremetns forpbsparklistpyspark>=3.2.0I've tested locally (though admittedly non-comprehensively) by modifying the requirements to allow pyspark==3.1.2, and it appears to work fine, at least for the cases I need.
Was the requirement set at 3.2.x because you have specific reason to believe it doesn't work in all cases on 3.1.x? Or was it simply a matter of 3.2.x being the version that it was developed against, and thus it's not tested against earlier versions?
If it's the latter, would you consider relaxing the requirements (perhaps with a README caveat that 3.1.x isn't thoroughly tested?) If it's the former, do you recall what doesn't work on earlier versions, so I don't wind up doing something that's going to come back to haunt me?
Either way - I wanted to thank you for your work on this package. I'm working in a world where I'm consuming literally dozens of distinct, large protobuf messages in Spark, and this package as you might imagine, becomes an enormous time saver.
Kudos, and thanks! -James