allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
364 stars 132 forks source link

Nginx Not Loading Plotly.js Resource: ClearML Self-Hosted Docker #227

Closed Betagmr closed 5 months ago

Betagmr commented 5 months ago

Nginx Not Loading Plotly.js Resource: ClearML Self-Hosted Docker

I have deployed a self-hosted instance of Clearml on Ubuntu 20.04 virtual machine with Docker. Initially, everything was working as expected, and I could view plots and scalars on the web-ui without any problems. However, at some point, the ui stop showing plots and scalars. As you can see in Image 2, the Plotly resource is not loading. It seems that Nginx is refusing to serve the file containing the Plotly JavaScript. The only change I made on Docker-Compose file was disabling the binding with IPv6.

Image Image
shyallegro commented 5 months ago

Hi @Betagmr , if I understand correctly you are not using the clearml docker compose. How are you serving the UI?

Betagmr commented 5 months ago

Hi @shyallegro, I'm using docker compose to serve the UI. I disabled the nginx IPv6 binding by adding an environment variable. Here is my Docker Compose configuration on the webserver, the rest remains the same.

webserver:
  command:
    - webserver
  container_name: clearml-webserver
  environment:
    - DISABLE_NGINX_IPV6=true
  #  CLEARML_SERVER_SUB_PATH : clearml-web # Allow Clearml to be served with a URL path prefix.
  image: allegroai/clearml:latest
  restart: unless-stopped
  depends_on:
    - apiserver
  ports:
    - "8080:80"
  networks:
    - backend
    - frontend
shyallegro commented 5 months ago

can you verify that it's possible to pull plotly js from the webserver? curl -v http://[server URL]:8080/app/webapp-common/assets/plotly-2.25.2.min.js > p.js

Betagmr commented 5 months ago

Okay, I tried but I got time out error. I also tested it with smaller files, and it seems to work if the file is less than 3 megs.

shyallegro commented 5 months ago

the vendor chunk is over 3mb as well, not sure it's a size issue. you can open a terminal into the running container to make sure the file is there, but since this is the release image there's no way it can't be. do you have some gateway between the serving Ubuntu host and your PC? maybe try to fetch the file from the host directly to localhost:8080

Betagmr commented 5 months ago

Yes, the file exists in the container. When I fetch the file from the server it works fine, the problem occurs when I fetch it from outside the server. We run the server on a VPN.

shyallegro commented 5 months ago

since this is your companies network, there's nothing else I can do. closing this ticket