Panel, Nginx Load Balancing and Custom Extensions breaks

[govinda18]

ALL software version info

panel 0.14.0, nginx 1.22.0

Description of expected behavior and the observed behavior

I am using the architecture discussed in the bokeh docs for load balancing with panel server - Load balancing with Nginx . This does not work well with custom bokeh extensions such as ipywidgets_bokeh or any other extension for that matter.

The error here is that the javascript extensions are not loaded with a 404 error on the server. Here is a sample error:

Failed to repull session Error: Model 'my_custom_ext.models.chart.chart_model.Chart' does not exist. This could be due to a widget or a custom model not being registered before first usage.

and in the networks tab the js file at /static/extensions/my_custom_ext/my_custom_ext.js?v=6b13789e43e5485634533de16a65d8ba9d34c4c9758588b665805435f80eb115 throws 404.

As I debugged a bit, the issue here is that nginx with least connection strategy sends each request to a different server and hence a different server is asked to load the extensions and a different one is requested for the javascript file of the extension. I wondered how is panel loading and it seems like at this line, hardcoding the value to override extension_dirs.

For now, a workaround for me is to use something like extension_dirs['my_custom_ext'] = str(Path(__file__).parent / "dist"). Not sure what should be the idea fix here though - feels like it should be bokeh and not panel but logging it here as a starting point.

Complete, minimal, self-contained example code that reproduces the issue

Follow https://docs.bokeh.org/en/test/docs/user_guide/server.html#load-balancing-with-nginx to setup and use a simple code like below:

import panel as pn
import ipywidgets as w
pn.Row(w.HTML("asd")).servable()

[ndmlny-qs]

I've tried to reproduce this error, and I just cannot seem to get it give me the same errors you have gotten. My process is a bit terse, but this is the method I used to try and reproduce your error.

Create a docker/podman container

This is my Containerfile as I am using podman, but I think the same will work with docker.

FROM ubuntu:latest
RUN apt-get update && \
    apt-get install -y build-essential && \
    apt-get install -y wget && \
    apt-get install -y nginx && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    -O ~/miniconda.sh && \
    /bin/bash ~/miniconda.sh -b -p /opt/conda
ENV PATH=$CONDA_DIR/bin:$PATH
RUN rm /etc/nginx/sites-available/default
COPY default /etc/nginx/sites-available/
CMD ["nginx"]
RUN conda install -y pip
RUN pip install ipywidgets==7.7.0 ipywidgets_bokeh==1.3.0 panel==0.14.0
COPY app.py .
EXPOSE 5100
EXPOSE 5101
EXPOSE 5102
COPY panel-serve.sh .
RUN chmod +x panel-serve.sh
CMD ["/bin/bash"]

You will need the following files in order to make the above Containerfile work.

The below python file called app.py is the application we wish to serve.

# app.py
import ipywidgets
import panel as pn

pn.Row(ipywidgets.HTML("asd")).servable()

The nginx configuration that uses the documentation from Bokeh.

upstream app {
    least_conn;
    server 127.0.0.1:5100;
    server 127.0.0.1:5101;
    server 127.0.0.1:5102;
}
server {
    location / {
        proxy_pass http://app;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host:$server_port;
        proxy_buffering off;
    }
}

The final component is the bash script that we will run indefinitely while serving the app.

#!/bin/bash

panel serve app.py --port 5100 &
panel serve app.py --port 5101 &
panel serve app.py --port 5102 &

wait -n

We will use podman to build this image using the following command, assuming you are in the same directory with all the above files.

podman build -t my-container .

Once the image has been built, we can run a container (tagged as my-container) with the following command.

podman run -td -p 5100:5100 -p 5101:5101 -p 5102:5102 localhost/my-container /panel-serve.sh

You can check to see the container is running with the command podman ps. It should show a container running. The next thing to do is to go to your favorite browser and enter in the url localhost:5100 in one tab, and in another tab open localhost:5101 etc.

My results showed the widget display in all tabs. @govinda18 can you try the above procedure to see if this works for you?

[hoxbro]

I don't know if it is related, but it seems like you have a misspelling in your Containerfile here: COPY default /etc/gninx/sites-available/

[ndmlny-qs]

thanks @Hoxbro that was a typo, which has now been fixed. I'm curious if you tried the example out and if it worked

[govinda18]

Hi Andy,

I think one of the issues with your setup is that you are directly exposing 5100, 5101 and 5102 from your container and when you hit them, nginx does not come into play. You might have to map some 8080:80 while running your image so that the request actually goes to nginx.

With that said, there is actually a simple way to reproduce this. Run the following:

panel serve app.py --port 5100
panel serve app.py --port 5101

Check the networks tab after opening and check the url for ipywidgets-bokeh. It will look something like: http://localhost:5100/static/extensions/ipywidgets_bokeh/ipywidgets_bokeh.js?v=6366d0c09db5d0dcc36e8cdc85c270e95e4cc21feb9987cc947fbb63c9dabee3

Try the same URL with the second port (make sure you do not open that app so that it is not initialized) - http://localhost:5101/static/extensions/ipywidgets_bokeh/ipywidgets_bokeh.js?v=6366d0c09db5d0dcc36e8cdc85c270e95e4cc21feb9987cc947fbb63c9dabee3

It should give 404. Now open the app at http://localhost:5101 and then open the above URL again and it will work. The hash argument seems to be machine dependent or something and is same across instances for a particular machine.

I did not try your setup yet but the above is very easy to reproduce. You can clearly see that nginx with least connection will send each request to a different server (including the get request at /static/ipywidgets_bokeh) and will run into the issue like above.

Let me know if this serves as a reproducer.

[ndmlny-qs]

@govinda18 I was able to reproduce your steps above. I am outlining below what I did to ensure I understand it, and others can reproduce it easily.

1. Install the conda environment conda env create --file environment.yaml using the environment.yaml file below.
2. Open two terminals and activate the environment in each one conda activate panel-nginx.
3. Serve the app.py module with panel in both terminals choosing a different port for each one.
   1. panel serve app.py --port 5100
   2. panel serve app.py --port 5101
4. Load http://localhost:5100 in your favorite browser, and open the dev console. Navigate to the Network tab and look for the ipywidets_bokeh JavaScript. Click on this item and copy the URL shown.
5. Open a new tab, and replace the port of the URL from 5100 -> 5101. When you request this URL, you should get a 404.
6. Finally open a new tab and load http://localhost:5101.
7. Navigate back to the tab where the ipywidgets_bokeh JavaScript gave you a 404. Reload the page and it will return the JavaScript as expected.

The steps above do not use nginx, but the result shows that panel/bokeh is not resourcing the static components correctly, as can be see in the Network tab of the dev console with this example.

I think I have modified the nginx/podman workflow to reproduce bokeh/panel error. See the section nginx below for the updated files. They now map all ports to 80, which is where nginx is listening, and nothing will load in the browser when you navigate to http://localhost:5100. The dev console does show panel and ipywidgets_bokeh not loading because the static folder can not be found, which is what the above example shows.

Question

You stated above that

For now, a workaround for me is to use something like extension_dirs['my_custom_ext'] = str(Path(file).parent / "dist"). Not sure what should be the idea fix here though - feels like it should be bokeh and not panel but logging it here as a starting point.

Are you building a custom bokeh extension that results in a dist folder? If this is the case, then we can use nginx to serve those static resources with another location block in the config. I'll try to create a custom bokeh widget, but I'll have to admit I've never ever with much frustration been able to get past the dreaded error of

Javascript Error: Model '...' does not exist.
This could be due to a widget or a custom model not being registered before first usage.

and I've spent a lot of time attempting to get past it, see the discussion here https://github.com/bokeh/bokeh/discussions/12301. If I can't get past the registered model error, I will need your help with an example that does build so we can try to make nginx serve static resources from the built model.

Alternatively, you can add this to your nginx config and see what happens with your built models, where the ... is your other config items.

server {
    ...
    location /static/ {
        alias path/to/static/dist/files;
    }
}

---

Files

# environment.yaml
name: panel-nginx
channels:
  - conda-forge
dependencies:
  - pip
  - pip:
      - ipywidgets==7.7.0
      - ipywidgets_bokeh==1.3.0
      - panel==0.14.0

# app.py
import ipywidgets
import panel as pn

pn.Row(ipywidgets.HTML("asd")).servable()

---

nginx

# Containerfile
FROM ubuntu:latest
RUN apt-get update && \
    apt-get install -y build-essential && \
    apt-get install -y wget && \
    apt-get install -y nginx && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    -O ~/miniconda.sh && \
    /bin/bash ~/miniconda.sh -b -p /opt/conda
ENV PATH=$CONDA_DIR/bin:$PATH
RUN rm /etc/nginx/sites-available/default
COPY default /etc/nginx/sites-available/
CMD ["nginx"]
RUN conda install -y pip
RUN pip install ipywidgets==7.7.0 ipywidgets_bokeh==1.3.0 panel==0.14.0
COPY app.py .
EXPOSE 80
COPY panel-serve.sh .
RUN chmod +x panel-serve.sh
WORKDIR /
CMD ["/bin/bash", "panel-serve.sh"]

# app.py
import ipywidgets
import panel as pn

pn.Row(ipywidgets.HTML("asd")).servable()

# default
upstream app {
    least_conn;
    server 127.0.0.1:5100;
    server 127.0.0.1:5101;
    server 127.0.0.1:5102;
}

server {
    listen 80 default_server;
    server_name _;

    access_log /tmp/bokeh.access.log;
    error_log /tmp/bokeh.error.log debug;

    location /app {
        proxy_pass http://app;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host:$server_port;
        proxy_buffering off;
    }
}

[govinda18]

Are you building a custom bokeh extension that results in a dist folder? If this is the case, then we can use nginx to serve those static resources with another location block in the config. I'll try to create a custom bokeh widget, but I'll have to admit I've never ever with much frustration been able to get past the dreaded error of

While I am building my own custom bokeh extension but the error is not limited to that. Any bokeh extension such as ipywidgets_bokeh will give you this error. There are a couple of issues with serving the static assets via nginx:

1. This is not scalable. For every bokeh extension you use, you will have to go and add a path here. So things like awesome-panel and all the extensions provided by them will have to be manually added to the nginx config.
2. The path from where the extensions should be served is dynamic. I may be using a virtual env and the path will depend on the python path. It might be tricky to write str(Path(file).parent / "dist") in nginx config.

and I've spent a lot of time attempting to get past it, see the discussion here https://github.com/bokeh/bokeh/discussions/12301. If I can't get past the registered model error, I will need your help with an example that does build so we can try to make nginx serve static resources from the built model.

I would suggest here that we ignore custom extension and even if we can make ipywidgets bokeh work in a generic way, the solution should likely solve it for custom extensions too. If we still need a custom extension that builds, we can simply take any one from the the awesome-panel org. @MarcSkovMadsen also wrote a detailed guide on developing custom bokeh model here - this was the one I followed for the first time while building a custom extension.

[ndmlny-qs]

Thanks for the further clarification.

[ndmlny-qs]

@govinda18 I have found where the version hash for JavaScript extensions is created in Bokeh, and how to prevent it from being appended to resources being served by the bokeh/panel server. You can read more about why it was implemented in the below PR.

https://github.com/bokeh/bokeh/pull/11573

overall gist to remove version hashes

This works for bokeh 3, panel 1, and ipywidgets_bokeh 1.4.

1. Use nginx to serve static files https://docs.bokeh.org/en/latest/docs/user_guide/server/deploy.html#load-balancing.
2. Move all your static files to a shared location https://docs.bokeh.org/en/latest/docs/user_guide/server/app.html#directory-format.
3. Ensure you use the correct environment variables https://docs.bokeh.org/en/latest/docs/dev_guide/setup.html#set-environment-variables.

tldr;

I am going to outline the same container based solution above, with the changes needed to remove the hashed version, and how to point to and serve static resources. All the code is found below in the section labeled "code". The steps to get custom extensions without version hashing working are as follows.

1. Create the container: podman build -t my-container .
2. Run the container: podman run -td -p 5100:5100 -p 5101:5101 -p 5102:5102 localhost/my-container
3. Navigate to http://localhost:5100/app

The important changes to the files include:

1. Adding bokeh server variables to the panel-serve.sh script. Without these, panel will append a version hash to the static JavaScript.
2. Changes to the the nginx.conf file include a new endpoint, called /static. This enpoint maps to a folder in the container at /static The Containerfile creates this folder, and copies all the required static JavaScript from bokeh, panel and ipywidgets_bokeh to the places the panel server expects to find them. Without these, the app will fail and more information about why it is a good idea to move your static files to a directory can be found in the bokeh documentation https://docs.bokeh.org/en/latest/docs/user_guide/server/deploy.html#load-balancing.

NOTE that the steps outlined in the above https://github.com/holoviz/panel/issues/4074#issuecomment-1522266913 still hold. So if you replicate those steps, you will get a 404 on a different port for static resources if you have not access the app at that port yet. This is to be expected because whe you issue the command panel serve ..., panel will start a server, but it will not serve anything until you access the document. This means that the 404 is to be expected, until you actually navigate to the the app at that port. Then the JS will get served, and you will no longer have a 404.

custom-bokeh-extension.webm

code

# Containerfile
# Load the official Debian nginx container.
FROM docker.io/library/nginx

# Update the container OS.
RUN apt-get update

# Copy our nginx configuration to the container.
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d/default.conf

# Download, install, set up conda, and create the virtual environment.
WORKDIR /
COPY panel-nginx-env.yaml .
RUN apt-get install -y wget \
    && wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /miniconda.sh \
    && /bin/bash /miniconda.sh -b -p /opt/conda
ENV CONDA_DIR /opt/conda
ENV PATH=$CONDA_DIR/bin:$PATH
ENV SHELL=/bin/bash
RUN conda env create --file panel-nginx-env.yaml

# Create a static folder and move all the needed JavaScript for Bokeh, Panel, and
# ipywidgets_bokeh in to it.
WORKDIR /
RUN mkdir -p /static/js \
    && mkdir -p /static/extensions/panel \
    && mkdir -p /static/extensions/ipywidgets_bokeh
WORKDIR /opt/conda/envs/panel-nginx/lib/python3.11/site-packages/bokeh/server/static/js
RUN cp bokeh.js /static/js \
    && cp bokeh-gl.js /static/js \
    && cp bokeh-tables.js /static/js \
    && cp bokeh-widgets.js /static/js
WORKDIR /opt/conda/envs/panel-nginx/lib/python3.11/site-packages/panel/dist
RUN cp panel.js /static/extensions/panel
WORKDIR /opt/conda/envs/panel-nginx/lib/python3.11/site-packages/ipywidgets_bokeh/dist
RUN cp ipywidgets_bokeh.js /static/extensions/ipywidgets_bokeh

# Copy the app to the container and make it executable.
WORKDIR /
COPY app.py .
COPY panel-serve.sh .
RUN chmod +x panel-serve.sh

# Clean up apt.
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# Expose ports.
EXPOSE 80

# Start nginx.
RUN /usr/sbin/nginx

# Serve the panel app.
ENTRYPOINT ["/bin/bash"]
CMD ["panel-serve.sh"]

# nginx.conf
upstream panel_servers {
    least_conn; # See https://nginx.org/en/docs/http/load_balancing.html
    server 127.0.0.1:5100;
    server 127.0.0.1:5101;
    server 127.0.0.1:5102;
}

server {
    # Catch all server name.
    # See https://nginx.org/en/docs/http/server_names.html section "Miscellaneous names"
    listen 80 default_server;
    server_name _;

    # Bokeh server logging.
    # See https://docs.bokeh.org/en/latest/docs/user_guide/server/deploy.html#nginx
    access_log /tmp/bokeh.access.log;
    error_log /tmp/bokeh.error.log debug;

    location / {
        # WebSocket proxying
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host:$server_port;
        proxy_buffering off;

        proxy_pass http://panel_servers;
    }

    # Bokeh, Panel, ipywidgets_bokeh static files.
    # See https://docs.bokeh.org/en/latest/docs/user_guide/server/deploy.html#nginx
    # Also look at the Container file where static resources are copied to the /static
    # folder.
    location /static {
        alias /static;
    }

    # nginx staus
    location /nginx_status {
        stub_status;
    }

}

# panel-nginx-env.yaml
name: panel-nginx
channels:
  - conda-forge
dependencies:
  - python
  # Package managers
  - pip
  # Development
  - pip:
    - ipywidgets==8.0.6
    - ipywidgets_bokeh==1.4.0
    - panel==1.0.2

# app.py
import ipywidgets
import panel as pn

pn.Row(ipywidgets.HTML("asd")).servable()

#!/bin/bash
# panel-serve.sh

# Bokeh environment variables.
export BOKEH_MINIFIED=no # Do not minify JavaScript.
export BOKEH_DEV=true # Prevents hash values from being generated for static JavaScript.
export BOKEH_RESOURCES=server-dev # Use local resources, not CDN resources.
export BOKEH_ALLOW_WS_ORIGIN=localhost

# Activate the conda environment.
source activate panel-nginx; wait;

# Serve the panel apps.
panel serve app.py --port 5100 &
panel serve app.py --port 5101 &
panel serve app.py --port 5102 &

# Never stop.
wait -n