Length of custom field or response from MLServer causes missing payload with lack of helpful logs

[dtpryce]

Describe the bug

When using a custom MLServer model we have created a JSON input (JSON str) and output (JSON dict) via curl the length of one of the custom fields would cause Seldon Core to return status 200 but no payload. Testing directly against MLServer everything appears to be fine, then once the field used is longer than 18 characters this behaviour appears only in Seldon Core.

To reproduce

1. create a custom JSON in / out model similar to https://mlserver.readthedocs.io/en/stable/examples/custom-json/README.html ensure there is at least one string field
2. Test curl against MLServer infer endpoint with the string field at length 18, 19, 20 and more (should all be fine)
3. Deploy within Seldon core (V1 - not tried in V2)
4. Test curl against Seldon core infer endpoint with the string field at length 18, 19, 20 and more (for 20 or greater should return 200 but no payload)

Expected behaviour

See above plus screenshots below

One more thing to note is that the rest of the entries in data dictionary were fixed in testing but might be contributing to overall length of payload, in screenshots you can see content-length of successful responses getting near to 500 characters before failure. If hard to reproduce try making sure the response dict is almost 500 chars long and then vary one field up to that threshold (it might be total content size?)

Environment

• Cloud Provider: AWS
• Kubernetes Cluster Version:
  • Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4a3b558c52eb6995b3c5c1db5e54111bd0645a64", GitTreeState:"clean", BuildDate:"2021-12-15T14:52:11Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
  • Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:11:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
• Deployed Seldon System Images: docker.io/seldonio/seldon-core-operator:1.10.0

Model Details

• Images of your model: [Output of: kubectl get seldondeployment -n <yourmodelnamespace> <seldondepname> -o yaml | grep image: where <yourmodelnamespace>]

• Logs of your model: [You can get the logs of your model by running kubectl logs -n <yourmodelnamespace> <seldonpodname> <container>]

• Images are custom built and private so you will need to build something similar

• No clear logging from either MLServer or Seldon Core on this - also part of the issue

The following screenshots show MLServer response all fine, then attempts of curling Seldon with various ResponseOutput / fields within lengths to get missing payload and no helpful logging.

[dtpryce]

@cliveseldon would this behaviour be different in V2? We are about to look at upgrading

[ukclivecox]

It would be a bug on either version. We have not had a chance to replicate on v1 yet. Trying the same model on V2 would be useful.

[dtpryce]

Ok well we should be trying some V2 stuff out soon but would love to hear about a fix, we are seeing this more and more now with our JSON output models.

[dtpryce]

We have also discovered that gRPC endpoint do not suffer from this bug. So this is only relevant to REST and appears to be on both long JSON string input and output.

[adriangonz]

Hey @dtpryce ,

So far we haven't been able to replicate this one internally. Could you share more details about your environment that can help us replicate it (e.g. what ingress are you using, is there a LB on front of that, etc.)?

Also, could you try testing this with the latest SC?

[dtpryce]

Hey @adriangonz Sorry for slow response. I will let @Kolajik comment more on the infrastructure but I do know that we were going through a slightly different ingress.

We are also about to deploy V2 and will keep this issue in mind before rolling out.