Proposal: Create KFServing Python SDK

[yuzisun]

/kind feature

CUJ

• For out-of-box prediction, user can call KFServing python sdk to launch KFService with the model uri on notebook after model is trained, this can be integrated with kubeflow pipeline.
• Fairing integration for custom prediction code and fairing can call KFServing python sdk to launch KFService custom container.
• Can be hooked up with production deployment UI backend service to launch, promote canary, rollback KFService.

Describe the solution you'd like

• Use openapi-gen to generate openapi definitions and swagger.json from KFService go struct.
• Use swagger-codegen to generate KFServing python sdk for the python object models.
• API to deploy KFService

  model_spec =  V1alpha1ModelSpec(tensorflow=V1alpha1TensorflowSpec(model_uri='gs://flower'))
  kfsvc = V1alpha1KFService(metadata=V1ObjectMeta(name='flower'),spec= V1alpha1KFServiceSpec(default=model_spec))
  KFServingApi.deploy(kfsvc)

• API to promote canary, default is reset with canary spec and 100 % traffic is routed to default.

  model_spec =  V1alpha1ModelSpec(tensorflow=V1alpha1TensorflowSpec(model_uri='gs://flower'))
  canary_model_spec =  V1alpha1ModelSpec(tensorflow=V1alpha1TensorflowSpec(model_uri='gs://flower/v2'))
  kfsvc = V1alpha1KFService(metadata=V1ObjectMeta(name='flower'),spec= V1alpha1KFServiceSpec(default=model_spec, canary= canary_model_spec, canary_traffic_percent=20))
  KFServingApi.deploy(kfsvc)
  KFServingApi.promote_canary(kfsvc)

[rakelkar]

Might be worth looking to see if this can align with something like MLFlow? @aronchick thoughts?

[rakelkar]

MLFlow has the concept of flavors for frameworks and talks about this:

MLflow Models: A model packaging format and tools that let you easily deploy the same model (from any ML library) to batch and real-time scoring on platforms such as Docker, Apache Spark, Azure ML and AWS SageMaker.

Would be cool to have KFServing on that list :-) and not require data scientists to learn something different... also as I was trying to debug stuff, would be nice to have this SDK/CLI have the smarts to pull out logs from the right places so that users dont need to Kubernetes...

[rakelkar]

Opened related issue in MLFlow to get feedback: https://github.com/mlflow/mlflow/issues/1465

[jinchihe]

@yuzisun Great idea! Recently I created a PR to deploy kfservice in the kubeflow Fairing. If we have kfserving, that's better to intergrate kfserving with kubeflow pipeline, fairing, and MLFlow as @rakelkar mentioned above.

I'm interested in doing this :-)

/assign @jinchihe

[jinchihe]

Followed instruction from @yuzisun, I generated the openapi model and swagger.json by using openapi-gen and generate KFServing python sdk for the python object models using swagger-codegen.

Will have a short discussion with @yuzisun before taking next steps, seems Dan has done some preliminary work, need to confirm more with Dan, thanks.

[jinchihe]

Just discussed with @yuzisun, I will do more tesing for the python SDK, and the deliver a PR for reviewing later, thanks.

[ellistarn]

/area infrastructure-feature