search

springml / spark-sftp

Spark connector for SFTP
Apache License 2.0
100 stars 98 forks source link

Incorrect file lookup | Actual file written in worker node and during transmit it is checking on driver node. #45

Open lkolla opened 5 years ago

lkolla commented 5 years ago

Incorrect file lookup | Actual file written in worker node and during transmit it is checking on driver node.

We tried to use this library, but we could not. we noticed following issue.

Issue., Actual file written in one of worker node during dataframe write and during transmit, it is checking in driver node.

Our use case, read data from kafka topic, transform data into a csv and transmit to sftp location.

environment: spark 2.2.1 and spark-sftp v1.1.3.

samuel-pt commented 5 years ago

@lkolla

Can you try providing temp folder as a parameter ? You can use "tempLocation" parameter for pass the tempFolder location