Open govinda18 opened 1 year ago
I've tried to reproduce this error, and I just cannot seem to get it give me the same errors you have gotten. My process is a bit terse, but this is the method I used to try and reproduce your error.
This is my Containerfile
as I am using podman, but I think the same will work with docker.
FROM ubuntu:latest
RUN apt-get update && \
apt-get install -y build-essential && \
apt-get install -y wget && \
apt-get install -y nginx && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
-O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda
ENV PATH=$CONDA_DIR/bin:$PATH
RUN rm /etc/nginx/sites-available/default
COPY default /etc/nginx/sites-available/
CMD ["nginx"]
RUN conda install -y pip
RUN pip install ipywidgets==7.7.0 ipywidgets_bokeh==1.3.0 panel==0.14.0
COPY app.py .
EXPOSE 5100
EXPOSE 5101
EXPOSE 5102
COPY panel-serve.sh .
RUN chmod +x panel-serve.sh
CMD ["/bin/bash"]
You will need the following files in order to make the above Containerfile
work.
The below python file called app.py
is the application we wish to serve.
# app.py
import ipywidgets
import panel as pn
pn.Row(ipywidgets.HTML("asd")).servable()
The nginx configuration that uses the documentation from Bokeh.
upstream app {
least_conn;
server 127.0.0.1:5100;
server 127.0.0.1:5101;
server 127.0.0.1:5102;
}
server {
location / {
proxy_pass http://app;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;
proxy_buffering off;
}
}
The final component is the bash script that we will run indefinitely while serving the app.
#!/bin/bash
panel serve app.py --port 5100 &
panel serve app.py --port 5101 &
panel serve app.py --port 5102 &
wait -n
We will use podman to build this image using the following command, assuming you are in the same directory with all the above files.
podman build -t my-container .
Once the image has been built, we can run a container (tagged as my-container
) with the following command.
podman run -td -p 5100:5100 -p 5101:5101 -p 5102:5102 localhost/my-container /panel-serve.sh
You can check to see the container is running with the command podman ps
. It should show a container running. The next thing to do is to go to your favorite browser and enter in the url localhost:5100
in one tab, and in another tab open localhost:5101
etc.
My results showed the widget display in all tabs. @govinda18 can you try the above procedure to see if this works for you?
I don't know if it is related, but it seems like you have a misspelling in your Containerfile
here: COPY default /etc/gninx/sites-available/
thanks @Hoxbro that was a typo, which has now been fixed. I'm curious if you tried the example out and if it worked
Hi Andy,
I think one of the issues with your setup is that you are directly exposing 5100, 5101 and 5102 from your container and when you hit them, nginx does not come into play. You might have to map some 8080:80 while running your image so that the request actually goes to nginx.
With that said, there is actually a simple way to reproduce this. Run the following:
panel serve app.py --port 5100
panel serve app.py --port 5101
Check the networks tab after opening and check the url for ipywidgets-bokeh. It will look something like: http://localhost:5100/static/extensions/ipywidgets_bokeh/ipywidgets_bokeh.js?v=6366d0c09db5d0dcc36e8cdc85c270e95e4cc21feb9987cc947fbb63c9dabee3
Try the same URL with the second port (make sure you do not open that app so that it is not initialized) - http://localhost:5101/static/extensions/ipywidgets_bokeh/ipywidgets_bokeh.js?v=6366d0c09db5d0dcc36e8cdc85c270e95e4cc21feb9987cc947fbb63c9dabee3
It should give 404. Now open the app at http://localhost:5101 and then open the above URL again and it will work. The hash argument seems to be machine dependent or something and is same across instances for a particular machine.
I did not try your setup yet but the above is very easy to reproduce. You can clearly see that nginx with least connection will send each request to a different server (including the get request at /static/ipywidgets_bokeh) and will run into the issue like above.
Let me know if this serves as a reproducer.
@govinda18 I was able to reproduce your steps above. I am outlining below what I did to ensure I understand it, and others can reproduce it easily.
conda
environment conda env create --file environment.yaml
using the environment.yaml
file below.conda activate panel-nginx
.app.py
module with panel
in both terminals choosing a different port for each one.
panel serve app.py --port 5100
panel serve app.py --port 5101
http://localhost:5100
in your favorite browser, and open the dev console. Navigate to the Network
tab and look for the ipywidets_bokeh
JavaScript. Click on this item and copy the URL shown.5100 -> 5101
. When you request this URL, you should get a 404.http://localhost:5101
.ipywidgets_bokeh
JavaScript gave you a 404. Reload the page and it will return the JavaScript as expected.The steps above do not use nginx
, but the result shows that panel/bokeh
is not resourcing the static
components correctly, as can be see in the Network tab of the dev console with this example.
I think I have modified the nginx/podman
workflow to reproduce bokeh/panel
error. See the section nginx below for the updated files. They now map all ports to 80, which is where nginx
is listening, and nothing will load in the browser when you navigate to http://localhost:5100
. The dev console does show panel
and ipywidgets_bokeh
not loading because the static
folder can not be found, which is what the above example shows.
You stated above that
For now, a workaround for me is to use something like extension_dirs['my_custom_ext'] = str(Path(file).parent / "dist"). Not sure what should be the idea fix here though - feels like it should be bokeh and not panel but logging it here as a starting point.
Are you building a custom bokeh extension that results in a dist
folder? If this is the case, then we can use nginx
to serve those static resources with another location
block in the config. I'll try to create a custom bokeh widget, but I'll have to admit I've never ever with much frustration been able to get past the dreaded error of
Javascript Error: Model '...' does not exist.
This could be due to a widget or a custom model not being registered before first usage.
and I've spent a lot of time attempting to get past it, see the discussion here https://github.com/bokeh/bokeh/discussions/12301. If I can't get past the registered model error, I will need your help with an example that does build so we can try to make nginx
serve static resources from the built model.
Alternatively, you can add this to your nginx
config and see what happens with your built models, where the ...
is your other config items.
server {
...
location /static/ {
alias path/to/static/dist/files;
}
}
# environment.yaml
name: panel-nginx
channels:
- conda-forge
dependencies:
- pip
- pip:
- ipywidgets==7.7.0
- ipywidgets_bokeh==1.3.0
- panel==0.14.0
# app.py
import ipywidgets
import panel as pn
pn.Row(ipywidgets.HTML("asd")).servable()
# Containerfile
FROM ubuntu:latest
RUN apt-get update && \
apt-get install -y build-essential && \
apt-get install -y wget && \
apt-get install -y nginx && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
-O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda
ENV PATH=$CONDA_DIR/bin:$PATH
RUN rm /etc/nginx/sites-available/default
COPY default /etc/nginx/sites-available/
CMD ["nginx"]
RUN conda install -y pip
RUN pip install ipywidgets==7.7.0 ipywidgets_bokeh==1.3.0 panel==0.14.0
COPY app.py .
EXPOSE 80
COPY panel-serve.sh .
RUN chmod +x panel-serve.sh
WORKDIR /
CMD ["/bin/bash", "panel-serve.sh"]
# app.py
import ipywidgets
import panel as pn
pn.Row(ipywidgets.HTML("asd")).servable()
# default
upstream app {
least_conn;
server 127.0.0.1:5100;
server 127.0.0.1:5101;
server 127.0.0.1:5102;
}
server {
listen 80 default_server;
server_name _;
access_log /tmp/bokeh.access.log;
error_log /tmp/bokeh.error.log debug;
location /app {
proxy_pass http://app;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;
proxy_buffering off;
}
}
Are you building a custom bokeh extension that results in a dist folder? If this is the case, then we can use nginx to serve those static resources with another location block in the config. I'll try to create a custom bokeh widget, but I'll have to admit I've never ever with much frustration been able to get past the dreaded error of
While I am building my own custom bokeh extension but the error is not limited to that. Any bokeh extension such as ipywidgets_bokeh will give you this error. There are a couple of issues with serving the static assets via nginx:
and I've spent a lot of time attempting to get past it, see the discussion here https://github.com/bokeh/bokeh/discussions/12301. If I can't get past the registered model error, I will need your help with an example that does build so we can try to make nginx serve static resources from the built model.
I would suggest here that we ignore custom extension and even if we can make ipywidgets bokeh work in a generic way, the solution should likely solve it for custom extensions too. If we still need a custom extension that builds, we can simply take any one from the the awesome-panel org. @MarcSkovMadsen also wrote a detailed guide on developing custom bokeh model here - this was the one I followed for the first time while building a custom extension.
Thanks for the further clarification.
@govinda18 I have found where the version hash for JavaScript extensions is created in Bokeh, and how to prevent it from being appended to resources being served by the bokeh/panel server. You can read more about why it was implemented in the below PR.
https://github.com/bokeh/bokeh/pull/11573
This works for bokeh 3, panel 1, and ipywidgets_bokeh 1.4.
I am going to outline the same container based solution above, with the changes needed to remove the hashed version, and how to point to and serve static resources. All the code is found below in the section labeled "code". The steps to get custom extensions without version hashing working are as follows.
podman build -t my-container .
podman run -td -p 5100:5100 -p 5101:5101 -p 5102:5102 localhost/my-container
The important changes to the files include:
panel-serve.sh
script. Without these, panel will append a version hash to the static JavaScript.nginx.conf
file include a new endpoint, called /static
. This enpoint maps to a folder in the container at /static
The Containerfile
creates this folder, and copies all the required static JavaScript from bokeh, panel and ipywidgets_bokeh to the places the panel server expects to find them. Without these, the app will fail and more information about why it is a good idea to move your static files to a directory can be found in the bokeh documentation https://docs.bokeh.org/en/latest/docs/user_guide/server/deploy.html#load-balancing.NOTE
that the steps outlined in the above https://github.com/holoviz/panel/issues/4074#issuecomment-1522266913 still hold. So if you replicate those steps, you will get a 404 on a different port for static resources if you have not access the app at that port yet. This is to be expected because whe you issue the command panel serve ...
, panel will start a server, but it will not serve anything until you access the document. This means that the 404 is to be expected, until you actually navigate to the the app at that port. Then the JS will get served, and you will no longer have a 404.
# Containerfile
# Load the official Debian nginx container.
FROM docker.io/library/nginx
# Update the container OS.
RUN apt-get update
# Copy our nginx configuration to the container.
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d/default.conf
# Download, install, set up conda, and create the virtual environment.
WORKDIR /
COPY panel-nginx-env.yaml .
RUN apt-get install -y wget \
&& wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /miniconda.sh \
&& /bin/bash /miniconda.sh -b -p /opt/conda
ENV CONDA_DIR /opt/conda
ENV PATH=$CONDA_DIR/bin:$PATH
ENV SHELL=/bin/bash
RUN conda env create --file panel-nginx-env.yaml
# Create a static folder and move all the needed JavaScript for Bokeh, Panel, and
# ipywidgets_bokeh in to it.
WORKDIR /
RUN mkdir -p /static/js \
&& mkdir -p /static/extensions/panel \
&& mkdir -p /static/extensions/ipywidgets_bokeh
WORKDIR /opt/conda/envs/panel-nginx/lib/python3.11/site-packages/bokeh/server/static/js
RUN cp bokeh.js /static/js \
&& cp bokeh-gl.js /static/js \
&& cp bokeh-tables.js /static/js \
&& cp bokeh-widgets.js /static/js
WORKDIR /opt/conda/envs/panel-nginx/lib/python3.11/site-packages/panel/dist
RUN cp panel.js /static/extensions/panel
WORKDIR /opt/conda/envs/panel-nginx/lib/python3.11/site-packages/ipywidgets_bokeh/dist
RUN cp ipywidgets_bokeh.js /static/extensions/ipywidgets_bokeh
# Copy the app to the container and make it executable.
WORKDIR /
COPY app.py .
COPY panel-serve.sh .
RUN chmod +x panel-serve.sh
# Clean up apt.
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
# Expose ports.
EXPOSE 80
# Start nginx.
RUN /usr/sbin/nginx
# Serve the panel app.
ENTRYPOINT ["/bin/bash"]
CMD ["panel-serve.sh"]
# nginx.conf
upstream panel_servers {
least_conn; # See https://nginx.org/en/docs/http/load_balancing.html
server 127.0.0.1:5100;
server 127.0.0.1:5101;
server 127.0.0.1:5102;
}
server {
# Catch all server name.
# See https://nginx.org/en/docs/http/server_names.html section "Miscellaneous names"
listen 80 default_server;
server_name _;
# Bokeh server logging.
# See https://docs.bokeh.org/en/latest/docs/user_guide/server/deploy.html#nginx
access_log /tmp/bokeh.access.log;
error_log /tmp/bokeh.error.log debug;
location / {
# WebSocket proxying
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;
proxy_buffering off;
proxy_pass http://panel_servers;
}
# Bokeh, Panel, ipywidgets_bokeh static files.
# See https://docs.bokeh.org/en/latest/docs/user_guide/server/deploy.html#nginx
# Also look at the Container file where static resources are copied to the /static
# folder.
location /static {
alias /static;
}
# nginx staus
location /nginx_status {
stub_status;
}
}
# panel-nginx-env.yaml
name: panel-nginx
channels:
- conda-forge
dependencies:
- python
# Package managers
- pip
# Development
- pip:
- ipywidgets==8.0.6
- ipywidgets_bokeh==1.4.0
- panel==1.0.2
# app.py
import ipywidgets
import panel as pn
pn.Row(ipywidgets.HTML("asd")).servable()
#!/bin/bash
# panel-serve.sh
# Bokeh environment variables.
export BOKEH_MINIFIED=no # Do not minify JavaScript.
export BOKEH_DEV=true # Prevents hash values from being generated for static JavaScript.
export BOKEH_RESOURCES=server-dev # Use local resources, not CDN resources.
export BOKEH_ALLOW_WS_ORIGIN=localhost
# Activate the conda environment.
source activate panel-nginx; wait;
# Serve the panel apps.
panel serve app.py --port 5100 &
panel serve app.py --port 5101 &
panel serve app.py --port 5102 &
# Never stop.
wait -n
ALL software version info
panel 0.14.0, nginx 1.22.0
Description of expected behavior and the observed behavior
I am using the architecture discussed in the bokeh docs for load balancing with panel server - Load balancing with Nginx . This does not work well with custom bokeh extensions such as ipywidgets_bokeh or any other extension for that matter.
The error here is that the javascript extensions are not loaded with a 404 error on the server. Here is a sample error:
and in the networks tab the js file at
/static/extensions/my_custom_ext/my_custom_ext.js?v=6b13789e43e5485634533de16a65d8ba9d34c4c9758588b665805435f80eb115
throws 404.As I debugged a bit, the issue here is that nginx with least connection strategy sends each request to a different server and hence a different server is asked to load the extensions and a different one is requested for the javascript file of the extension. I wondered how is panel loading and it seems like at this line, hardcoding the value to override extension_dirs.
For now, a workaround for me is to use something like
extension_dirs['my_custom_ext'] = str(Path(__file__).parent / "dist")
. Not sure what should be the idea fix here though - feels like it should be bokeh and not panel but logging it here as a starting point.Complete, minimal, self-contained example code that reproduces the issue
Follow https://docs.bokeh.org/en/test/docs/user_guide/server.html#load-balancing-with-nginx to setup and use a simple code like below: