search

SeldonIO / MLServer

An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
https://mlserver.readthedocs.io/en/latest/
Apache License 2.0
685 stars 177 forks source link

Multi-language Runtimes #183

Open axsaucedo opened 3 years ago

axsaucedo commented 3 years ago

Create multi-language wrappers that can run Java, C++ and R models. For this, we can leverage the existing research in Seldon Core which leverages tools like JNI and PyBind to bridge from Python to other runtimes.

indranilr commented 1 year ago

Adding to the above issue as indicated in https://github.com/SeldonIO/seldon-core/issues/4531

MLeap is used to deploy Spark mllib models (including Spark Xgboost and others) for inference through Java runtime. With Seldon core the Java language wrapper could be used to wrap MLeap format models and deploy using Java JNI wrapper. The JNI wrapper however had many vulnerabilities https://github.com/SeldonIO/seldon-core/issues/4103.

Please support deployment of MLeap serialized Spark MLlib models in MLServer so that Java JNI wrapper vulnerabilities can be avoided.

adriangonz commented 1 year ago

Hey @indranilr ,

Thanks for expressing interest on this one!

Just as a head up, this is on the roadmap, but probably won't come through until the 1.4.0 release.

dtpryce commented 1 year ago

Just out of interest would this include Julia support?

adriangonz commented 1 year ago

We've already done something along these lines for C++ and Java for SCv1 so that would be the initial priority. We are not planning to explore adding support for Julia yet.

dtpryce commented 1 year ago

Totally understood! Will be good to see those first few languages and we could always try to mimic for Julia if we decide we really need it :) thank @adriangonz!

indranilr commented 6 months ago

Hey @indranilr ,

Thanks for expressing interest on this one!

Just as a head up, this is on the roadmap, but probably won't come through until the 1.4.0 release.

@adriangonz since 1.4.0 is released now, has the support for MLeap serialized Spark Xgboost model incorporated ? Could you please point to relevant documentation ?

adriangonz commented 6 months ago

Hey @indranilr,

I don't work on Seldon or this project anymore. Best would be to check with @ramonpzg on what's the status of this one.

indranilr commented 6 months ago

Hey @indranilr,

I don't work on Seldon or this project anymore. Best would be to check with @ramonpzg on what's the status of this one.

Thanks Adrian, @ramonpzg appreciate if you can provide guidance w.r.t this question

ramonpzg commented 5 months ago

Hi @indranilr -- We recently moved things around in the roadmap of MLServer and have started dedicating more capacity to improving different features of it and its runtime. It might take us a couple more versions before we can fully dedicate time for multi-language wrappers (it would definitely be very valuable to add these :raised_hands: ).