search

mlverse / pysparklyr

Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect
https://spark.posit.co/deployment/databricks-connect.html
Other
14 stars 3 forks source link

Date not converting properly #115

Closed edgararuiz closed 5 months ago

edgararuiz commented 6 months ago 
library(sparklyr)
sc <- spark_connect(method = "databricks_connect")
sdf_sql(sc, "SELECT date('2023-01-01') AS date_value")
#> # Source:   table<`sparklyr_tmp_table_90883880_812a_49f2_a071_639c35330ee3`> [1 x 1]
#> # Database: spark_connection
#>   date_value
#>   <chr>     
#> 1 19358
spark_disconnect(sc)
edgararuiz commented 6 months ago

Tracked it down to an problem with to_pandas from Python

d1 <- invoke(sc, "sql", "SELECT  date('2023-01-01') AS date_value")
d2<- d1$pyspark_obj
d2$toPandas()
#>   date_value
#> 1      19358