sparklyr.nested features

mitre / sparklyr.nested

A sparklyr extension for nested data

Apache License 2.0

31 stars 4 forks source link

sparklyr.nested features #25

Closed sbottelli closed 1 year ago

sbottelli commented 3 years ago

Hi all, I really appreciate your job on extend sparklyr world with nested data! I want to ask you a question about how to use sparklyr.nested in detail. Can you report here some example about:

calculating mean or median on nested tbl_spark
using spark_apply function on nested tbl_spark Thanks for your support!

mattpollock commented 1 year ago

Those features are beyond the scope of what I tried to accomplish with sparklyr.nested. However I believe that both of those operations are now possible using the "hof" (higher order functions) and using local tydyr functions in spark_apply (the complication there being you must ensure that the package is available on the worker nodes).