springml / spark-sftp

Spark connector for SFTP
Apache License 2.0
100 stars 98 forks source link

Null(Blank) columns are returned with quotes(" ") when writing to csv #78

Open ucrkarthik opened 3 years ago

ucrkarthik commented 3 years ago

Would we be able to add a "nullValue" option to the writeToTemp method?

Here is some more info on the nullValue: nullValue – sets the string representation of a null value. If None is set, it uses the default value, empty string. Since 2.0.1, this nullValue param applies to all supported types including the string type.

https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html?highlight=drop

Thanks, Karthik

ucrkarthik commented 3 years ago

Here are the changes that I am proposing to add nullValue to the code. This will be in the DefaultSource.scala file.

image

ucrkarthik commented 3 years ago

@samuel-pt would you be able to review this request?