Closed kunaljubce closed 4 months ago
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of quinn.
[X] I have confirmed this bug exists on the main branch of quinn.
This is mainly a CI error, so to reproduce it, you would need to install `PySpark 2.4.8`.
Happening due to tests failing on the below functions as df.transform() was only introduced in Spark 3.0.0 -
df.transform()
test_transform_with_lambda()
test_transform_with_no_arg_fun()
test_transform_with_one_arg_fun()
test_chain_transforms()
test_transform_with_closure()
test_transform_with_functools_partial()
For PySpark 2.4.x, this test should not get executed.
Quinn version checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of quinn.
[X] I have confirmed this bug exists on the main branch of quinn.
Reproducible Example
Issue Description
Happening due to tests failing on the below functions as
df.transform()was only introduced in Spark 3.0.0 -test_transform_with_lambda()test_transform_with_no_arg_fun()test_transform_with_one_arg_fun()test_chain_transforms()test_transform_with_closure()test_transform_with_functools_partial()Expected Behavior
For PySpark 2.4.x, this test should not get executed.
Installed Versions