Create wrapper for Spark standalone runtime models

[ukclivecox]

• expose Spark standalone runtime models with a thin REST server or gRPC server that respects the internal model API

• Build Docker image

[Maximophone]

Use the python wrapper https://github.com/SeldonIO/seldon-core/blob/master/docs/wrappers/python.md as a reference

[holdenk]

This seems related to https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/119 / https://github.com/kubeflow/kubeflow/issues/155 although model serving in Python with Spark is not very well supported right now (it works but it involves copying the data to the JVM and using a cluster like environment which isn't super well suited to serving.

Starting with a Python wrapper at the start could be a good first place to go, since if we end up needing to re-implement predict function anyways to avoid the cluster-like behaviour might as well fit in with everything else.

[ukclivecox]

Ideally, the idea referenced in https://pages.databricks.com/Real-time-Prediction-Serving.html would be good where you can export your runtime as a standalone Java class without Spark dependencies. Then it could be wrapped for running in seldon-core as a prediction microservice. However, I have not heard of much progress on this from the Spark community. It seems to be an enterprise feature at present in Databricks.

We are planning to support easy Java wrappers soon for easy Java/Scala integration.

[holdenk]

That code is proprietary and developed internally at Databricks as one of their differentiators, so not suitable for us to try and integrate.

Although we can follow that same approach, and there is some work on progress to that in the OSS side by DB Tsai (eg the local linalg package for prediction).

[holdenk]

So correct me if I'm wrong, but it looks like right now in the two wrappers added by @cliveseldon for Java we assume the inputs are all Doubles? Is that inherent in the design?

[holdenk]

Oh wait, when I look at the prediction protos it seems to assume only double inputs -- which seems limiting unless I'm missreading it?

[ukclivecox]

The prediction protos allow you to send in a Tensor (doubles) or NdArray for easy use from JSON/REST and for mixed type values. You can also send in custom binary or string as well.

[holdenk]

So if I want to do a prediction on a mixed collection of types JSON/REST interface is the only one? What would be the right way to implement a model which needs to serve on a mix of input types (e.g. a general Spark pipeline model)?

Would it be to have people give a JSON representation of the Row object as a string and parse it manually?

[ukclivecox]

The NDArray option can be used from gRPC as well as JSON/REST. We don't have a current example using gRPC for this use case. It would be great to have one.

[holdenk]

Ok, so gRPC can be used for any of them, but NDArray won't work for an arbitrary pipeline right (e.g. something with a mix of strings/doubles) - I'm more or less stuck encoding something else inside of the protos right?

[ukclivecox]

Yes, NDArray can be used for mixed types which is why it was chosen. Would we be great to have an example using this in the codebase if we can help you create it.

[jomach]

Is there any active development on this ? I think I have a use case for this. We have data scientist that develop on jupyter notebooks + pyspark. The problem: They have custom libs in their udfs that they are developing on the go. We want to provide a way of iterative development with the cluster, this means let's say the udf A get's changed with import pandas as pd how we let the spark cluster know that he now needs to include pandas ? Would google flow solve this problem ?

[ukclivecox]

We are working on an example to export Spark models as PMML and then provide a Java wrapper to do runtime predictions using a PMML model evaluator JPMML. Would this fit your needs?

[jomach]

@cliveseldon Thanks for the quick reply. But I'm still one step before that, so the answer would be no it does not fit. I want to have a way when data scientists develop a function with a third party dependency it would get somehow deployed / installed into the cluster.

[ukclivecox]

You can use our standard python wrappers to wrap code into an image for use. At present we support tools such as RedHat's source-to-image. Once the new image is created and deployment image version updated in the manifest a rolling update can be done using kubectl. We are working on demos to show how the CI/CD for this can be automated. If you wish to start a new thread discussing your particular needs please do.

[ukclivecox]

Closing as PMML examples exists. We also have ONNX examples. For other ways to use Spark models we should open a new issue.

[Kuoxen]

We are working on an example to export Spark models as PMML and then provide a Java wrapper to do runtime predictions using a PMML model evaluator JPMML. Would this fit your needs?

@cliveseldon Sorry to dig this up, but I can't find the "example" in the project. Could you please share the link

[ukclivecox]

You will find a Triton ONNX example in https://github.com/SeldonIO/seldon-core/blob/master/notebooks/triton_examples.ipynb

[Kuoxen]

many thanks

[ukclivecox]

https://docs.seldon.io/projects/seldon-core/en/latest/examples/triton_examples.html