incorrect directories - Githubissues

francopatanojll commented 6 years ago

Using Azure Databricks, the connector dropped the file in the root tmp folder "/tmp/" but when accessing the file later it errored out with file not found because it was reading from "/dbfs/tmp/".

samuel-pt commented 6 years ago

Can you try specifying the temp folder using tempLocation option? Something like

val df = spark.read.
            format("com.springml.spark.sftp").
            option("host", "SFTP_HOST").
            option("username", "SFTP_USER").
            option("password", "****").
            option("fileType", "csv").
            option("delimiter", ";").
            option("inferSchema", "true").
            option("tempLocation", "/dbfs/tmp/").
            load("/ftp/files/sample.csv")

francopatanojll commented 6 years ago

That works, thanks!

From: samuel-pt [mailto:notifications@github.com] Sent: Monday, May 28, 2018 12:29 AM To: springml/spark-sftp spark-sftp@noreply.github.com Cc: Patano, Franco Franco.Patano@am.jll.com; Author author@noreply.github.com Subject: [EXTERNAL] Re: [springml/spark-sftp] incorrect directories (#23)

Can you try specifying the temp folder using tempLocation option? Something like

val df = spark.read.

        format("com.springml.spark.sftp").

        option("host", "SFTP_HOST").

        option("username", "SFTP_USER").

        option("password", "****").

        option("fileType", "csv").

        option("delimiter", ";").

        option("inferSchema", "true").

        option("tempLocation", "/dbfs/tmp/").

        load("/ftp/files/sample.csv")

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/springml/spark-sftp/issues/23#issuecomment-392422964, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AltRIbSFlp4-twew3X_JxSqOWw6y5oX4ks5t24sBgaJpZM4UOJ4-.

This email is for the use of the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. We have taken precautions to minimize the risk of transmitting software viruses, but we advise you to carry out your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege. If you are the intended recipient and you do not wish to receive similar electronic messages from us in the future then please respond to the sender to this effect.

springml / spark-sftp

incorrect directories #23