I use the image pytorch/torchserve:0.5.3-cpu as the environment.
Installed using source? [yes/no]: No
Are you planning to deploy it using docker container? [yes/no]: yes
Is it a CPU or GPU environment?: CPU
Using a default/custom handler? [If possible upload/share custom handler/model]: custom, see below.
What kind of model is it e.g. vision, text, audio?: vision
I test with two workflows:
dog_breed_classification in the examples
default mnist model, and make a testing workflow with the following files:
mnist_workflow_handlers.py
import json
import base64
def print_data(data, context):
print("+" * 50)
print("node: print_data")
print("data:", data)
print("context:", context)
if data is None:
return data
output = []
for row in data:
d = row.get("data") or row.get("body")
# if isinstance(d, (bytes, bytearray)):
# d = d.decode('utf-8')
output.append(d)
return output
def test_data(data, context):
print("+" * 50)
print("node: test_data")
print("data:", data)
print("context:", context)
if data is None:
return data
output = []
for row in data:
d = row.get("data") or row.get("body")
# if isinstance(d, (bytes, bytearray)):
# d = d.decode('utf-8')
output.append(d)
return output
def combine_data(data, context):
if data is None:
return data
print("+" * 50)
print("node: combine_data")
print("data:", data)
print("context:", context)
output = []
for row in data:
data_dict = {}
for k in row:
d = row.get(k)
# if isinstance(d, (bytes, bytearray)):
# d = d.decode('utf-8')
data_dict[k] = d
output.append(data_dict)
print([d.get("print_data") for d in output])
return [d.get("print_data") for d in output]
If a node has multiple parent nodes, for example node A with result ["a"] and node B with result ["b"], the output of these node will be combined by the frontend into:
However, because there is no key data or body (only A and B)in this json, the envelope class cannot process the data.
Even if the data is found, the kserve v2 envelope can only handle inputs key of inputs other than outputs.
Also in order to pass it to the model properly as it does with null envelope, the request data needs to be converted into the form:
[
"data": {
"A": ["a"],
"B": ["b"]
}
]
Possible Solution
It is possible to set all the workflow model/functions' envelope to null, whatever the service_envelope specifed in the config.properties. However, this creates an inconsistence between the model api standard and the workflow api standard. And all the models in the workflow use null envelope while other models use another envelope. Therefore, it is not a desired solution.
In order to solve the issue and maintain the consistency between APIs of workflow and standalone models:
Set the envelope of workflow functions to the default envelope as regular models (those have .mar files)
Identify internal workflow requests between nodes, and let kserve v2 envelope be able to parse outputs data. For example, pass a special header from frontend, telling the backend this is an internal workflow request.
Identify internal workflow requests type, whether it is a flat or nested data. For example, pass a special header from frontend, telling backend the request is flat or nested.
Kserve v2 envelope handler will handle the data according to the header.
I've prepared a pull request and will submit it asap for you review
I divide the changes into separate commits targeting different problems with description in the commit message. Once all of the commits fulfill the requirements, I will add more unittests, rebase and squash the commits.
If possible, please help to give suggestions whether I'm on the right track of this problem.
Steps to Reproduce
Pull the image pytorch/torchserve:0.5.3-cpu
Don't set the envelope in config.properties. Run the container with the workflow mnist I provided above, or the example workflow dog_breed_classification. The prediction will succeed.
Set the service_envelope=kservev2, run the container with the workflow in 2.. Now the prediction will fail
Failure Logs [if any]
I add some print in ts/model_service_worker.py:
class TorchModelServiceWorker(object):
@staticmethod
def load_model(load_model_request):
try:
model_dir = load_model_request["modelPath"].decode("utf-8")
model_name = load_model_request["modelName"].decode("utf-8")
handler = load_model_request["handler"].decode("utf-8") if load_model_request["handler"] else None
envelope = load_model_request["envelope"].decode("utf-8") if "envelope" in load_model_request else None
envelope = envelope if envelope is not None and len(envelope) > 0 else None
print("=" * 50)
print("model name:", model_name)
print("envelope: ", envelope)
Below are the logs when loading the dog_breed_classification:
Context
0.5.3
1.10.0
Your Environment
I use the image
pytorch/torchserve:0.5.3-cpu
as the environment.I test with two workflows:
mnist_workflow_handlers.py
mnist_workflow.yaml
Expected Behavior
The workflow api should work out of box with kserve v2 envelope.
Current Behavior
The workflow api doesn't work with kserve v2 envelope.
Reasons
1.The envelope of the workflow function is not set correctly.
While the models' envelope are set to the default envelope specified in config.properties, the workflow funcitons' envelope is null.
2. The request envelope
kservev2.py
cannot internal workflow requests and nested requests properly.If a node has multiple parent nodes, for example node
A
with result["a"]
and nodeB
with result["b"]
, the output of these node will be combined by thefrontend
into:However, because there is no key
data
orbody
(onlyA
andB
)in this json, the envelope class cannot process the data.Even if the data is found, the kserve v2 envelope can only handle inputs key of
inputs
other thanoutputs
.Also in order to pass it to the model properly as it does with null envelope, the request data needs to be converted into the form:
Possible Solution
It is possible to set all the workflow model/functions' envelope to null, whatever the service_envelope specifed in the config.properties. However, this creates an inconsistence between the model api standard and the workflow api standard. And all the models in the workflow use null envelope while other models use another envelope. Therefore, it is not a desired solution.
In order to solve the issue and maintain the consistency between APIs of workflow and standalone models:
.mar
files)internal workflow requests
between nodes, and let kserve v2 envelope be able to parseoutputs
data. For example, pass a special header from frontend, telling the backend this is aninternal workflow request
.flat
ornested
data. For example, pass a special header from frontend, telling backend the request isflat
ornested
.I've prepared a pull request and will submit it asap for you review
I divide the changes into separate commits targeting different problems with description in the commit message. Once all of the commits fulfill the requirements, I will add more unittests, rebase and squash the commits.
If possible, please help to give suggestions whether I'm on the right track of this problem.
Steps to Reproduce
pytorch/torchserve:0.5.3-cpu
config.properties
. Run the container with the workflow mnist I provided above, or the example workflow dog_breed_classification. The prediction will succeed.service_envelope=kservev2
, run the container with the workflow in2.
. Now the prediction will failFailure Logs [if any]
I add some print in
ts/model_service_worker.py
:Below are the logs when loading the
dog_breed_classification
: