search

SeldonIO / seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
https://www.seldon.io/tech/products/core/
Other
4.37k stars 831 forks source link

A mirror in pipeline experiments has a different request ID from the default route #4918

Open jklaise opened 1 year ago

jklaise commented 1 year ago

Describe the bug

Running an experiment with a mirror between pipelines and inspecting each pipeline after sending a single request reveals that the request IDs between the default route and the mirror are different.

To reproduce

Follow the "pipeline mirror" example here: https://docs.seldon.io/projects/seldon-core/en/v2/contents/examples/local-experiments.html#pipeline-mirror-experiment

After sending a single request, inspect both pipelines, e.g.:

seldon pipeline inspect pipeline-add10
seldon.default.model.add10.inputs   ci5cr6d8gs9c73b1tea0    {"inputs":[{"name":"INPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[1, 2, 3, 4]}}]}
seldon.default.model.add10.outputs  ci5cr6d8gs9c73b1tea0    {"modelName":"add10_1", "modelVersion":"1", "outputs":[{"name":"OUTPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[11, 12, 13, 14]}}]}
seldon.default.pipeline.pipeline-add10.inputs   ci5cr6d8gs9c73b1tea0    {"inputs":[{"name":"INPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[1, 2, 3, 4]}}]}
seldon.default.pipeline.pipeline-add10.outputs  ci5cr6d8gs9c73b1tea0    {"outputs":[{"name":"OUTPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[11, 12, 13, 14]}}]}
seldon pipeline inspect pipeline-mul10
seldon.default.model.mul10.inputs   ci5cr6d8gs9c73b1teag    {"inputs":[{"name":"INPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[1, 2, 3, 4]}}]}
seldon.default.model.mul10.outputs  ci5cr6d8gs9c73b1teag    {"modelName":"mul10_1", "modelVersion":"1", "outputs":[{"name":"OUTPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[10, 20, 30, 40]}}]}
seldon.default.pipeline.pipeline-mul10.inputs   ci5cr6d8gs9c73b1teag    {"inputs":[{"name":"INPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[1, 2, 3, 4]}}]}
seldon.default.pipeline.pipeline-mul10.outputs  ci5cr6d8gs9c73b1teag    {"outputs":[{"name":"OUTPUT", "datatype":"FP32", "shape":["4"], "contents":{"fp32Contents":[10, 20, 30, 40]}}]}

The default route add10 has been assigned an ID ci5cr6d8gs9c73b1tea0 whereas the mirror has an ID ci5cr6d8gs9c73b1teag.

Expected behaviour

The request ID between the default route and the mirror is the same.

Environment

Local deployment via docker-compose running on latest v2 branch.

Model Details

Models provided via Seldon examples.

After chatting to @cliveseldon, this may be because the request IDs are only assigned after requests hit the pipeline gateway. Therefore, in an experiment with a mirror the traffic split using Envoy would be done before a request ID is assigned and so each downstream pipeline would be assigned a different ID.

agrski commented 1 year ago

You can also define your own request ID, I believe, in which case the pipeline gateway won't assign a new one