Open mrericrichter opened 4 months ago
I was looking at how PySpark implements the ML functions, it seems that they use numpy and do some of the work on the client but I'm not 100% - will carry on trying to figure it out!
Typically, these functions run on worker nodes. However, it seems that Spark Connect currently supports SQL functions only. All supported functions contain a flag in their documentation that states 'Supports Spark Connect', e.g.
Functions from the MLLib do not contain this flag, e.g.
This seems to be a major limitation of Spark Connect as of today. The documentation says that more functions will be added in future versions of Spark.
i'll keep this open and add them in when they are available
Could you please provide an example (or even an implementation) of how to use functions from the ML package, e.g. the
Microsoft.Spark.ML.Feature.Bucketizer
class?