allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.61k stars 651 forks source link

webserver nginx proxy config incorrect #1082

Open chriswue opened 1 year ago

chriswue commented 1 year ago

Describe the bug

When running the official clearml docker image with the webserver argument then an nginx instance is spun up. The proxy config rewrites /api paths to forward to the apiserver instance. As part of that it sets the proxy forward Host header to $host which is incorrect because $host refers to the original Host header or original server name (http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host). Instead the variable $proxy_host should be used.

Why is this important: We have ClearML deployed as an Azure Container App and the ingress controller will see a request that goes to the IP of the apiserver container but with a Host header of the webserver and will deny access with a 403. Setting the header like this instead: proxy_set_header Host $proxy_host; solves this problem.

I suspect that this wasn't noticed until now because in a docker-compose environment there is no ingress controller that performs sanity checking an the apiserver doesn't care about the Host either.

Environment

chriswue commented 1 year ago

Sorry, I should probably raise these issues against the clearml-server repo. Closing this in favor of https://github.com/allegroai/clearml-server/issues/209