Open lkolla opened 5 years ago
Incorrect file lookup | Actual file written in worker node and during transmit it is checking on driver node.
We tried to use this library, but we could not. we noticed following issue.
Issue., Actual file written in one of worker node during dataframe write and during transmit, it is checking in driver node.
Our use case, read data from kafka topic, transform data into a csv and transmit to sftp location.
environment: spark 2.2.1 and spark-sftp v1.1.3.
@lkolla
Can you try providing temp folder as a parameter ? You can use "tempLocation" parameter for pass the tempFolder location
Incorrect file lookup | Actual file written in worker node and during transmit it is checking on driver node.
We tried to use this library, but we could not. we noticed following issue.
Issue., Actual file written in one of worker node during dataframe write and during transmit, it is checking in driver node.
Our use case, read data from kafka topic, transform data into a csv and transmit to sftp location.
environment: spark 2.2.1 and spark-sftp v1.1.3.