ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
When running the official clearml docker image with the webserver argument then an nginx instance is spun up. The proxy config rewrites /api paths to forward to the apiserver instance. As part of that it sets the proxy forward Host header to $host which is incorrect because $host refers to the original Host header or original server name (http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host). Instead the variable $proxy_host should be used.
Why is this important: We have ClearML deployed as an Azure Container App and the ingress controller will see a request that goes to the IP of the apiserver container but with a Host header of the webserver and will deny access with a 403.
Setting the header like this instead: proxy_set_header Host $proxy_host; solves this problem.
I suspect that this wasn't noticed until now because in a docker-compose environment there is no ingress controller that performs sanity checking an the apiserver doesn't care about the Host either.
Environment
Server type (self hosted \ app.clear.ml): self-hosted via Azure Container Apps
ClearML SDK Version: N/A
ClearML Server Version (Only for self hosted). 1.11.0-373
Describe the bug
When running the official clearml docker image with the
webserver
argument then annginx
instance is spun up. The proxy config rewrites/api
paths to forward to theapiserver
instance. As part of that it sets the proxy forwardHost
header to$host
which is incorrect because$host
refers to the originalHost
header or original server name (http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host). Instead the variable$proxy_host
should be used.Why is this important: We have ClearML deployed as an Azure Container App and the ingress controller will see a request that goes to the IP of the
apiserver
container but with aHost
header of thewebserver
and will deny access with a 403. Setting the header like this instead:proxy_set_header Host $proxy_host;
solves this problem.I suspect that this wasn't noticed until now because in a docker-compose environment there is no ingress controller that performs sanity checking an the
apiserver
doesn't care about theHost
either.Environment