"POST v2/repository/models/${MODEL_NAME}/load" failed on 22.05 but works fine on 21.08

[NonStatic2014]

Description I used triton to create a service locally with command below:

tritonserver --model-repository=/host/models/ --strict-model-config=false --model-control-mode "explicit" --load-model fashion --load-model densenet --load-model bidaf-9

Both 22.05 and 21.08 can load the model.

Then i try to unload the model "fashion" with

curl -X POST http://127.0.0.1:8000/v2/repository/models/fashion/unload -v

Both 22.05 and 21.08 can unload the model. Using curl -X POST http://127.0.0.1:8000/v2/repository/index -v verified.

Then try to load the fashion mode back with below json. It works fine on 21.08 but failed on 22.05.

curl -X POST -d @"fashion_load.json" http://127.0.0.1:8000/v2/repository/models/fashion/load -v

{
  "parameters": {
    "config": {
      "name": "fashion",
      "versions": [
        1
      ],
      "platform": "tensorflow_savedmodel",
      "inputs": [{
          "name": "Conv1_input",
          "datatype": "FP32",
          "shape": [
            -1,
            28,
            28,
            1
          ]
        }
      ],
      "outputs": [{
          "name": "Dense",
          "datatype": "FP32",
          "shape": [
            -1,
            10
          ]
        }
      ]
    },
    "file": "1/saved_model.pb"
  }
}

The error message is

{"error":"attempt to access JSON non-string as string"}

I also tried if load the model without the json payload, since the file name is default name, it can be loaded and scoring can work.

Triton Information triton image: 22.05 triton version: 2.22.0

Are you using the Triton container or did you build it yourself? -> container

To Reproduce See above

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

Expected behavior 22.05 can load the model as what we see on 21.08.

[GuanLuo]

Note that for "config", the value should be a string as described here, can you double quote the actual configuration (may need to escape " within the config as well), i.e. "config" : "{...}"

Another note is that the feature to provide config via load API is available since 22.03, and it is optional. So the above load invocation in 21.08 is actually loading the model with pre-existing model config (the request body is ignored), the equivalent in 22.05 is simply invoking the API with empty request body

[NonStatic2014]

Would you mind update below documents to mention which triton version or triton image version we should use?

• https://github.com/triton-inference-server/server/blob/main/docs/model_management.md
• https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_model_repository.md

I'm also looking for a real sample like what I mentioned the issue in the second link above. That will be very helpful for the first time user. Thanks!

[GuanLuo]

Note that you are looking at the main branch where the documentation is reflecting the latest development, you can switch to the corresponding release branch to checkout the available feature in the release that you are using.

I agree that the documentation can be improved with example usage of the API, as for now, you may look at the test coverage for the usage

[NonStatic2014]

The test coverage is using triton sdk. Do we have any sample using REST API directly?

[GuanLuo]

Not yet, but the comment I mentioned earlier should help with the issue that you are facing. Can you verify it?

[NonStatic2014]

Thank you @GuanLuo ! Update the config can make it through in the latest image.

Moving forward, can we add something like "new from 22.01" in the first line of the doc? Here is a sample from Python document.

[zengqingfu1442]

I still come across the same error message on triton 2.39.0. I use postman to launch the post requst,

[zengqingfu1442]

/reopen

[zengqingfu1442]

Note that for "config", the value should be a string as described here, can you double quote the actual configuration (may need to escape " within the config as well), i.e. "config" : "{...}"

Another note is that the feature to provide config via load API is available since 22.03, and it is optional. So the above load invocation in 21.08 is actually loading the model with pre-existing model config (the request body is ignored), the equivalent in 22.05 is simply invoking the API with empty request body

I tried to double quote the actual configuration but still failed with another error:

[Manikandan-t]

Hi @zengqingfu1442 Try with Escaping double quotes of config json similar to this

[zengqingfu1442]

Hi @zengqingfu1442 Try with Escaping double quotes of config json similar to this

Thanks a lot.