search

mitre / sparklyr.nested

A sparklyr extension for nested data
Apache License 2.0
31 stars 4 forks source link

wrong default name on multi-level deep select #21

Closed mattpollock closed 5 years ago

mattpollock commented 5 years ago

see https://stackoverflow.com/a/49410602/1785752

Reprex:

write

[{"a":"jkl","b":"mno","c":{"c1":"ghi","c2":{"c21":"abc","c22":"def"}}}]

to a file and get a spark session sc

spk_df <- spark_read_json(sc, "tmp", "file:///path/to/test.json")
spk_df %>% sdf_select(c.c2.c22)
# # Source:   table<sparklyr_tmp_7431373dca00> [?? x 1]
# # Database: spark_connection
#   c2
#   <chr>
# 1 def

but the column name should be c22

mattpollock commented 5 years ago

a work-around is to use spk_df %>% sdf_select(c22 = c.c2.c22)