Support for Spark 3 - Githubissues

springml / spark-sftp

Spark connector for SFTP

Apache License 2.0

100 stars 98 forks source link

Support for Spark 3 #83

Open adityashekhar146 opened 3 years ago

adityashekhar146 commented 3 years ago

Currently, the spark-sftp package doesn't work when trying to use in Spark 3.0. Are there any plans to extend the support for Spark 3?

shaikmanu797 commented 3 years ago

@adityashekhar146 take a look at https://github.com/arcizon/spark-filetransfer repository (supports Spark 3) which I had developed if it helps to solve your issue.

satishetc4288 commented 3 years ago

@shaikmanu797 your spark-filetransfer has issues. I tried it with spark 3 and i am seeing file content got changed after sending it into SFTP server

satishetc4288 commented 3 years ago

@adityashekhar146 Currently, the spark-sftp package doesn't work when trying to use in Spark 3.0. Are there any plans to extend the support for Spark 3? We are waiting for spark 3 Support .

hadoopBrogrammers commented 3 years ago

We are also looking for same sftp connector for spark3.x, so is there any plan to have this soon.

kumgaurav commented 3 years ago

Why there is no commit or improvement on this repo. I also see scala 2.12 pull request is open from long time too.

maleksan commented 3 years ago

@adityashekhar146 take a look at https://github.com/arcizon/spark-filetransfer repository (supports Spark 3) which I had developed if it helps to solve your issue.

This connector library perfectly works in my case. (Spark 3.1.2, Scala 2.12), @shaikmanu797 thanks for great work.

rajancherma commented 11 months ago

@adityashekhar146 are there any plans for provide support for scala 2.13