CORS config not working

[koking0]

🐛 Describe the bug

I followed the HuggingFace example to deploy the GPT2 model, but a CORS error occurred when the front-end was called.

I configured it in config.properties according to the tutorial on the official website, and then restarted the Docker container, but it didn't seem to work.

my config.properties file:

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082

number_of_netty_threads=32
job_queue_size=1000
model_store=/home/model-server/model-store
workflow_store=/home/model-server/wf-store

cors_allowed_origin=*
cors_allowed_methods=GET, POST, PUT, DELETE
cors_allowed_headers=X-Custom-Header

inside Docker container:

The magic is that not all requests will report errors. The GET request to obtain the model list and the POST request to register the model will not report an error, but the PUT request to modify the model configuration and the DELETE request to delete the model will report an error. The most fatal thing is to make predictions. The POST request reports an error.

The front end uses axios to request to delete the model

      axios.delete(url + "/" + row["modelName"]).then(_ => {
        this.getTorchServeModelList();
      }).catch(err => {
        console.log(err);
      })

Error logs

front-end request error:

Installation instructions

docker pull pytorch/torchserve:latest-gpu
docker run --rm -it -d --name torchserve --gpus '"device=0"' -p 18080:8080 -p 18081:8081 -p 18082:8082 -p 17070:7070 -p 17071:7071 -v $(pwd)/config.properties:/home/model-server/config.properties -v $(pwd)/model-store:/home/model-server/model-store pytorch/torchserve:latest-gpu

Model Packaing

torch-model-archiver --model-name insurance_chat --version 1.0 --serialized-file saved_model/conversation/pytorch_model.bin --handler ./torchserve_handler.py --export-path model-store --extra-files "saved_model/conversation/config.json"

config.properties

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082

number_of_netty_threads=32
job_queue_size=1000
model_store=/home/model-server/model-store
workflow_store=/home/model-server/wf-store

cors_allowed_origin=*
cors_allowed_methods=GET, POST, PUT, DELETE
cors_allowed_headers=X-Custom-Header

Versions

pip list:

torch                           1.12.0+cu113
torch-model-archiver            0.6.0
torch-workflow-archiver         0.2.4
torchaudio                      0.12.0
torchmetrics                    0.9.1
torchserve                      0.6.0
torchvision                     0.13.0

docker images:

pytorch/torchserve                     latest-gpu                 fb6d4b85847d   4 weeks ago     4.49GB

Repro instructions

I followed the HuggingFace example to deploy the GPT2 model.

Possible Solution

No response

[xyang16]

I was able to reproduce the issue. The cause is that the browser (tested Firefox, Chrome and Safari) was using OPTIONS to do a preflight check before PUT and DELETE request.

In this case, the OPTIONS request is:

OPTIONS http://123.125.8.44:18081/models/insurance_chat?minWorkers=2&maxWorkers=3

TS code currently doesn't handle OPTIONS for management API and will throw MethodNotAllowedException.

https://github.com/pytorch/serve/blob/master/frontend/server/src/main/java/org/pytorch/serve/http/api/rest/ManagementRequestHandler.java#L96

[xyang16]

@koking0 Please change the cors_allowed_methods in config.properties to:

cors_allowed_methods=GET, POST, PUT, DELETE, OPTIONS