Feature request: pyspark 3.0.0 version support

aws / sagemaker-spark

A Spark library for Amazon SageMaker.

https://aws.github.io/sagemaker-spark/

Apache License 2.0

298 stars 126 forks source link

Feature request: pyspark 3.0.0 version support #120

Open halilduygulu opened 4 years ago

halilduygulu commented 4 years ago

Please fill out the form below.

System Information

Spark or PySpark:
SDK Version:
Spark Version:
Algorithm (e.g. KMeans):

Describe the problem

When is pyspark 3.0.0 version support coming? Can we have an estimate to plan update in projects? Right now sagemaker-pyspark lib is not working with latest pyspark as expected.

Exact command to reproduce:

metrizable commented 4 years ago

@halilduygulu Thanks for using Amazon SageMaker.

Currently, we don't have an estimate on upgrading our pyspark dependency to 3.0.0.

We are always re-evaluating our backlog of features based on customer requests, so we appreciate the feedback on this feature.

Let us know if there is anything else we can be an assistance of.

odesiuz commented 4 years ago

Just to second what @halilduygulu has asked, I have been waiting for sagemaker to support pyspark 3.0.0. It will be nice to update the dependency as I guess some people will be eager to try out some new feat to see the improvements in spark 3.0.0.

Thanks