search

plotly / Kaleido

Fast static image export for web-based visualization libraries with zero dependencies
MIT License
378 stars 38 forks source link

Kaleido:TNG failure using python:3.12-alpine and python:3.12-slim-bookworm #211

Closed gvwilson closed 2 weeks ago

gvwilson commented 4 weeks ago

A user reports, "I tried to run [the test program] in a container using python:3.12-alpine and python:3.12-slim-bookworm images. I installed chromium (apk add —no-cache chromium; apt update && apt install chromium). I get an error regarding "Close() was called before the browser finished opening”".

Test program:

import plotly.express as px
fig = px.scatter(px.data.iris(), x="sepal_length", y="sepal_width", color="species")
fig.write_image("figure.png", engine="kaleido")

Error message:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/choreographer/browser.py", line 225, in _open_async
    raise BrowserFailedError("The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.") from e
choreographer.browser.BrowserFailedError: The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.
Traceback (most recent call last):
  File "//test.py", line 3, in <module>
    fig.write_image("figure.png", engine="kaleido")
  File "/usr/local/lib/python3.12/site-packages/plotly/basedatatypes.py", line 3835, in write_image
    return pio.write_image(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 266, in write_image
    img_data = to_image(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 143, in to_image
    img_bytes = scope.transform(
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kaleido/scopes/plotly.py", line 197, in transform
    img = kaleido.to_image_block(spec, Path(self._tempfile.name).absolute(), self._topojson, self._mapbox_access_token)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kaleido/__init__.py", line 27, in to_image_block
    return asyncio.run(to_image(spec, f, topojson, mapbox_token))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kaleido/__init__.py", line 30, in to_image
    async with Browser(headless=True) as browser:
choreographer.browser.BrowserFailedError: Close() was called before the browser finished opening- maybe it crashed?

dtdoctor output attacheddtdoctor.txt

gvwilson commented 4 weeks ago

@mschoettle has provided a container for us to try: https://github.com/mschoettle/plotly-image-export-alpine - thank you

ayjayt commented 4 weeks ago

Interesting, probably this:

[1023/201538.372535:ERROR:zygote_host_impl_linux.cc(101)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.

causing chromium to shutdown immediately.

ill submit something before EOD

vfilter commented 3 weeks ago

Will this be included in the next RC?

ayjayt commented 3 weeks ago

I'm sorry I didn't see this earlier, the next RC should be released today or tomorrow and yes, I believe this issue was fixed.

In any case, all of the debugging output is being improved in this RC which will help us further support our our users.

Thanks for your patience.

ayjayt commented 2 weeks ago

Hi @vfilter

A new release candidate is available on pypi. The issue of running as root was solved, so if that was your issue and you run pip install --pre kaleido, it should be fixed.

The release candidate is currently going through a cycle of alpha testing, and includes options for debugging that will be documented to further help debug issues upon release.

Kaleido is under very activate development, I appreicate your patience.

vfilter commented 2 weeks ago

@ayjayt thank you for your kind response. What we thought is related to the original issue of this PR has actually not been solved in the latest prerelease for us. Happy to move this into a new PR if unrelated to this.

This works when running locally, but does not when running it on k8s in a container.

By the looks of it, Kaleido's init.py instantiates a Browser() class from the choreographer package, which is either looking for the BROWSER_PATH env variable, or trying to get it from this function in the system package.

If I understand this correctly, we need to have chrome installed in the docker image? If that's the case, I can simply add chrome to the Docker image and add the BROWSER_PATH env variable. I shall try that next.

Image

BrowserFailedError: Could not find an acceptable browser. Please set environmental variable BROWSER_PATH or pass path=/path/to/browser into the Browser() constructor.

File "/usr/local/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
    result = func()
             ^^^^^^
File "/usr/local/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec
    exec(code, module.__dict__)
File "/app/app.py", line 10, in <module>
    pg.run()
File "/usr/local/lib/python3.11/site-packages/streamlit/navigation/page.py", line 303, in run
    exec(code, module.__dict__)
File "/app/simulations.py", line 424, in <module>
    summary_svg = summary_chart.to_image(format="svg", width=896, height=432, scale=1)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/plotly/basedatatypes.py", line 3772, in to_image
    return pio.to_image(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/plotly/io/_kaleido.py", line 143, in to_image
    img_bytes = scope.transform(
                ^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kaleido/scopes/plotly.py", line 218, in transform
    img = kaleido.to_image_block(spec, Path(self._tempfile.name).absolute(), self._topojson, self._mapbox_access_token, debug=debug)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kaleido/__init__.py", line 43, in to_image_block
    return asyncio.run(to_image(spec, f, topojson, mapbox_token, debug=debug))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kaleido/__init__.py", line 53, in to_image
    Browser(headless=True, debug=debug, debug_browser=sys.stderr) as browser,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/choreographer/browser.py", line 106, in __init__
    raise BrowserFailedError(

Dockerfile

FROM python:3.11-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ARG MOTHERDUCK_TOKEN=${MOTHERDUCK_TOKEN}
ENV MOTHERDUCK_TOKEN=${MOTHERDUCK_TOKEN}

ARG APP_DIR="/app/"
WORKDIR ${APP_DIR}

COPY ./src/build.py ${APP_DIR}

RUN uv pip install --system duckdb==1.1.2
RUN uv run build.py

FROM python:3.11-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ARG APP_DIR="/app/"
WORKDIR ${APP_DIR}

COPY ./uv.lock /app/
COPY ./pyproject.toml /app/
COPY ./src/ /app/

RUN uv pip install --system -r pyproject.toml

COPY --from=build ${APP_DIR}/powerplants.duckdb ${APP_DIR}

EXPOSE 8501

HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health

ENTRYPOINT ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]
vfilter commented 2 weeks ago

Following up here. I've updated the Dockerfile in the following way:

FROM python:3.11-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ARG MOTHERDUCK_TOKEN=${MOTHERDUCK_TOKEN}
ENV MOTHERDUCK_TOKEN=${MOTHERDUCK_TOKEN}

ARG APP_DIR="/app/"
RUN mkdir -p ${APP_DIR}
WORKDIR ${APP_DIR}

COPY ./src/build.py ${APP_DIR}

RUN uv pip install --system duckdb==1.1.2
RUN uv run build.py

FROM python:3.11-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ARG APP_DIR="/app/"
ENV BROWSER_PATH=/opt/chrome/chrome-linux64/chrome
RUN mkdir -p ${APP_DIR}
WORKDIR ${APP_DIR}

# Install dependencies
RUN apt-get update -y && apt-get install -y wget xvfb unzip jq curl

# Install Google Chrome dependencies
RUN apt-get install -y libxss1 libappindicator1 libgconf-2-4 \
    fonts-liberation libasound2 libnspr4 libnss3 libx11-xcb1 libxtst6 lsb-release xdg-utils \
    libgbm1 libnss3 libatk-bridge2.0-0 libgtk-3-0 libx11-xcb1 libxcb-dri3-0

# Fetch the latest version numbers and URLs for Chrome and ChromeDriver
RUN curl -s https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json > /tmp/versions.json

RUN CHROME_URL=$(jq -r '.channels.Stable.downloads.chrome[] | select(.platform=="linux64") | .url' /tmp/versions.json) && \
    wget -q --continue -O /tmp/chrome-linux64.zip $CHROME_URL && \
    unzip /tmp/chrome-linux64.zip -d /opt/chrome

RUN chmod +x /opt/chrome/chrome-linux64/chrome

# Clean up
RUN rm /tmp/chrome-linux64.zip /tmp/versions.json

COPY ./uv.lock /app/
COPY ./pyproject.toml /app/
COPY ./src/ /app/

RUN uv pip install --system -r pyproject.toml

COPY --from=build ${APP_DIR}/powerplants.duckdb ${APP_DIR}

EXPOSE 8501

HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health

ENTRYPOINT ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

However, now I'm actually getting an error similar to what this thread was originally started about:

BrowserFailedError: Close() was called before the browser finished opening- maybe it crashed?

File "/usr/local/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
    result = func()
             ^^^^^^
File "/usr/local/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec
    exec(code, module.__dict__)
File "/app/app.py", line 10, in <module>
    pg.run()
File "/usr/local/lib/python3.11/site-packages/streamlit/navigation/page.py", line 303, in run
    exec(code, module.__dict__)
File "/app/simulations.py", line 424, in <module>
    summary_svg = summary_chart.to_image(format="svg", width=896, height=432, scale=1)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/plotly/basedatatypes.py", line 3772, in to_image
    return pio.to_image(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/plotly/io/_kaleido.py", line 143, in to_image
    img_bytes = scope.transform(
                ^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kaleido/scopes/plotly.py", line 218, in transform
    img = kaleido.to_image_block(spec, Path(self._tempfile.name).absolute(), self._topojson, self._mapbox_access_token, debug=debug)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kaleido/__init__.py", line 43, in to_image_block
    return asyncio.run(to_image(spec, f, topojson, mapbox_token, debug=debug))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kaleido/__init__.py", line 52, in to_image
    async with (
mschoettle commented 2 weeks ago

@vfilter Are you using kaleido==0.4.0rc3? Your code snippets are not showing the pyproject.toml where this could be seen.

ayjayt commented 2 weeks ago

If I understand this correctly, we need to have chrome installed in the docker image? If that's the case, I can simply add chrome to the Docker image and add the BROWSER_PATH env variable. I shall try that next.

This analysis was correct @vfilter, but you shouldn't necessarily need to add the BROWSER_PATH variable, as linux is pretty good about finding chrome by itself. Although I'm not sure given how you're pulling it in.

@mschoettle is also correct in that we cannot see if you are using the must updated version of kaleido, there have been a flurry of updates to deal with a wide range of bugs.

BrowserFailedErrors like the one you're receiving are very interesting to me because generally it's a configuration/platform match that we need to address- I will see later today if I can provoke it with the docker-file you supplied (give me two hours, actually).

That said, you can also also set the environmental variable KALEIDO-DEBUG to anything, and it will generate an ENORMOUS amount of logs that I can read.

ayjayt commented 2 weeks ago

Update: I was able to load a version of the above Dockerfile with the proprietary stuff removed and run all of our tests successfully.

I hope with a pip install --upgrade --pre kaleido you'll have a similiar experience.

Here was my Dockerfile (I realize some things don't make sense w/ the proprietary stuff removed lol):

FROM python:3.11-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

FROM python:3.11-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ENV BROWSER_PATH=/opt/chrome/chrome-linux64/chrome

# Install dependencies
RUN apt-get update -y && apt-get install -y wget xvfb unzip jq curl

# Install Google Chrome dependencies
RUN apt-get install -y libxss1 libappindicator1 libgconf-2-4 \
    fonts-liberation libasound2 libnspr4 libnss3 libx11-xcb1 libxtst6 lsb-release xdg-utils \
    libgbm1 libnss3 libatk-bridge2.0-0 libgtk-3-0 libx11-xcb1 libxcb-dri3-0

# Fetch the latest version numbers and URLs for Chrome and ChromeDriver
RUN curl -s https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json > /tmp/versions.json

RUN CHROME_URL=$(jq -r '.channels.Stable.downloads.chrome[] | select(.platform=="linux64") | .url' /tmp/versions.json) && \
    wget -q --continue -O /tmp/chrome-linux64.zip $CHROME_URL && \
    unzip /tmp/chrome-linux64.zip -d /opt/chrome

RUN chmod +x /opt/chrome/chrome-linux64/chrome

# Clean up
RUN rm /tmp/chrome-linux64.zip /tmp/versions.json
mschoettle commented 2 weeks ago

@ayjayt

The issue of running as root was solved

Does this imply that running as non-root worked already? That would be nice since we actually run as non-root (which is recommended in general).

Also, what requirements besides chromium itself are there?

I updated the above referenced repo (https://github.com/mschoettle/plotly-image-export-alpine) with the MRE to RC3 and get the following error now:

Old Headless mode will be removed from the Chrome binary soon. Please use the new Headless mode (https://developer.chrome.com/docs/chromium/new-headless) or the chrome-headless-shell which is a standalone implementation of the old Headless mode (https://developer.chrome.com/blog/chrome-headless-shell).

[1105/203530.086993:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1105/203530.097380:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1105/203530.097417:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1105/203530.164852:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1105/203530.165544:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1105/203530.165662:ERROR:angle_platform_impl.cc(44)] Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
ERR: Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.165798:ERROR:gl_display.cc(497)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.165954:ERROR:gl_display.cc(767)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1105/203530.166124:ERROR:gl_display.cc(801)] Initialization of all EGL display types failed.
[1105/203530.166373:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1105/203530.167130:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1105/203530.167203:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1105/203530.167315:ERROR:angle_platform_impl.cc(44)] Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
ERR: Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.167504:ERROR:gl_display.cc(497)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.167588:ERROR:gl_display.cc(767)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1105/203530.167840:ERROR:gl_display.cc(801)] Initialization of all EGL display types failed.
[1105/203530.167981:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1105/203530.171074:WARNING:viz_main_impl.cc(85)] VizNullHypothesis is disabled (not a warning)
[1105/203530.172819:ERROR:viz_main_impl.cc(181)] Exiting GPU process due to errors during initialization
[1105/203530.203224:WARNING:bluez_dbus_manager.cc(234)] Floss manager service not available, cannot set Floss enable/disable.
[1105/203530.213141:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1105/203530.213310:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1105/203530.213550:ERROR:angle_platform_impl.cc(44)] Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
ERR: Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.213678:ERROR:gl_display.cc(497)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.213851:ERROR:gl_display.cc(767)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1105/203530.214064:ERROR:gl_display.cc(801)] Initialization of all EGL display types failed.
[1105/203530.214206:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1105/203530.215079:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1105/203530.215129:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1105/203530.215190:ERROR:angle_platform_impl.cc(44)] Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
ERR: Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.215274:ERROR:gl_display.cc(497)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.215389:ERROR:gl_display.cc(767)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1105/203530.215563:ERROR:gl_display.cc(801)] Initialization of all EGL display types failed.
[1105/203530.215660:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1105/203530.216583:WARNING:viz_main_impl.cc(85)] VizNullHypothesis is disabled (not a warning)
[1105/203530.217075:ERROR:viz_main_impl.cc(181)] Exiting GPU process due to errors during initialization
[1105/203530.273613:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1105/203530.273712:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1105/203530.273767:ERROR:angle_platform_impl.cc(44)] Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
ERR: Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.273861:ERROR:gl_display.cc(497)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.273932:ERROR:gl_display.cc(767)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1105/203530.274030:ERROR:gl_display.cc(801)] Initialization of all EGL display types failed.
[1105/203530.274129:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1105/203530.274763:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1105/203530.274824:ERROR:angle_platform_impl.cc(44)] vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: vk_renderer.cpp:181 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1105/203530.274905:ERROR:angle_platform_impl.cc(44)] Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
ERR: Display.cpp:1083 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.275108:ERROR:gl_display.cc(497)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/vk_renderer.cpp, enableInstanceExtensions:1790.
[1105/203530.275168:ERROR:gl_display.cc(767)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1105/203530.275381:ERROR:gl_display.cc(801)] Initialization of all EGL display types failed.
[1105/203530.275456:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1105/203530.276654:WARNING:viz_main_impl.cc(85)] VizNullHypothesis is disabled (not a warning)
[1105/203530.276980:ERROR:viz_main_impl.cc(181)] Exiting GPU process due to errors during initialization
[1105/203530.282277:WARNING:viz_main_impl.cc(85)] VizNullHypothesis is disabled (not a warning)
ayjayt commented 2 weeks ago

Thanks @mschoettle for that output.

1.

Does this imply that running as non-root worked already?

Yes --> Chromium has to be asked to run as root w/ a second flag which we include by default now for users that run as root.

2. We also now disable GPU by default since it improves first-run success rates (but we will add a command switch this week to re-enable). The annoying and weird thing about chrome/ium is that even with --disable-gpu, it still attempts to find the GPU and throws out a bunch of errors like yours, except it will continue normally despite them with that flag.

Were there other errors? Did it fail to render? I will take a look at your reference repo if not this afternoon, tomorrow morning.

Thanks for your patience,

mschoettle commented 2 weeks ago

Thanks @ayjayt.

Good point. It actually renders and writes it to the file successfully. I updated the example repo so that the scripts writes it to a directory that is bind mounted. I also added the option to run as non-root (by default).

I noticed that it runs much quicker now than with rc1.

vfilter commented 2 weeks ago

@vfilter Are you using kaleido==0.4.0rc3? Your code snippets are not showing the pyproject.toml where this could be seen.

Here is my pyproject.toml. I'm using the rc4 actually.

[project]
name = "xxx"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
    "boto3>=1.35.49",
    "duckdb>=1.1.2",
    "kaleido==0.4.0rc4",
    "nbformat>=5.10.4",
    "plotly>=5.24.1",
    "polars>=1.11.0",
    "pydantic>=2.9.2",
    "pyxirr>=0.10.6",
    "streamlit>=1.39.0",
]

[tool.uv]
dev-dependencies = [
    "boto3-stubs[s3]>=1.35.49",
    "ipykernel>=6.29.5",
    "ruff>=0.7.1",
]

[tool.pytest.ini_options]
pythonpath = [".", "src"]
testpaths = ["tests"]
addopts = "--cov"

[tool.ruff]
extend-include = ["*.ipynb"]
fix = true
line-length = 120

[tool.ruff.lint]
fixable = ["ALL"]
extend-select = [
    "B",
    "Q",
    "I",
    "E",
    "PD",
    "PL",
    "TRY",
    "NPY",
    "N",
    "D",
    "UP",
    "S",
    "DTZ",
    "TD",
]

[tool.ruff.lint.flake8-type-checking]
strict = true

[tool.ruff.lint.isort]
force-single-line = true
lines-after-imports = 2
lines-between-types = 1
ayjayt commented 2 weeks ago

@vfilter thanks for the toml. Your environment isn't allowing chrome to run, for some reason. Your docker works fine on our machines, but sometimes security policies or unique configuration will cause chrome to crash. However, I can't see the specific reason, so please:

  1. 0.4.0rc5 is out, I recommend telling the toml to get the highest version: "kaleido>=0.4.0rc5".
  2. ENV KALEIDO_DEBUG="true" to your Dockerfile, it will generate a massive amount of debug output to stderr.

I believe you can view its output with docker logs CONTAINER_ID, please post it here so I can help further,

thanks so much for your patience.

vfilter commented 2 weeks ago

@ayjayt thanks a bunch for willing to take a look at this. Here are the logs I'm seeing. Running this in a Docker container locally BTW, not deployed to AWS.

2024-11-07 18:32:49 Tempdir: /root/.kaleido-n7p2cwrj
2024-11-07 18:32:49 ************Displaying generated HTML*************
2024-11-07 18:32:49 <!DOCTYPE html>
2024-11-07 18:32:49 <html>
2024-11-07 18:32:49   <head>
2024-11-07 18:32:49     <style id="head-style"></style>
2024-11-07 18:32:49     <title>Kaleido-fier</title>
2024-11-07 18:32:49     <script>
2024-11-07 18:32:49         window.KaleidoReport = ["start"];
2024-11-07 18:32:49         function logError(e) {
2024-11-07 18:32:49             window.KaleidoReport.push("error");
2024-11-07 18:32:49             window.KaleidoReport.push(e);
2024-11-07 18:32:49             if (!navigator.onLine) {
2024-11-07 18:32:49                 window.KaleidoReport.push("offline");
2024-11-07 18:32:49             }
2024-11-07 18:32:49         }
2024-11-07 18:32:49     </script>
2024-11-07 18:32:49     <script>
2024-11-07 18:32:49       window.PlotlyConfig = {MathJaxConfig: 'local'}
2024-11-07 18:32:49     </script>
2024-11-07 18:32:49  <script src="https://cdn.plot.ly/plotly-2.35.2.min.js" charset="utf-8" onerror="logError('plotly')"></script>
2024-11-07 18:32:49  <script type="text/javascript" id="MathJax-script" src="file:///usr/local/lib/python3.11/site-packages/kaleido/vendor/mathjax/MathJax.js?config=TeX-AMS-MML_SVG" onerror="logError('mathjax')"></script>
2024-11-07 18:32:49     <script src="file:///usr/local/lib/python3.11/site-packages/kaleido/vendor/kaleido_scopes.js" onerror="logError('scoper')"></script>  </head>
2024-11-07 18:32:49   <body style="{margin: 0; padding: 0;}"><img id="kaleido-image"><img></body>
2024-11-07 18:32:49 </html>
2024-11-07 18:32:49 ***********************end************************
2024-11-07 18:32:49 ************Displaying generated HTML*************
2024-11-07 18:32:49 <!DOCTYPE html>
2024-11-07 18:32:49 <html>
2024-11-07 18:32:49   <head>
2024-11-07 18:32:49     <style id="head-style"></style>
2024-11-07 18:32:49     <title>Kaleido-fier</title>
2024-11-07 18:32:49     <script>
2024-11-07 18:32:49         window.KaleidoReport = ["start"];
2024-11-07 18:32:49         function logError(e) {
2024-11-07 18:32:49             window.KaleidoReport.push("error");
2024-11-07 18:32:49             window.KaleidoReport.push(e);
2024-11-07 18:32:49             if (!navigator.onLine) {
2024-11-07 18:32:49                 window.KaleidoReport.push("offline");
2024-11-07 18:32:49             }
2024-11-07 18:32:49         }
2024-11-07 18:32:49     </script>
2024-11-07 18:32:49     <script>
2024-11-07 18:32:49       window.PlotlyConfig = {MathJaxConfig: 'local'}
2024-11-07 18:32:49     </script>
2024-11-07 18:32:49  <script src="file:///usr/local/lib/python3.11/site-packages/plotly/package_data/plotly.min.js" charset="utf-8" onerror="logError('plotly')"></script>
2024-11-07 18:32:49  <script type="text/javascript" id="MathJax-script" src="file:///usr/local/lib/python3.11/site-packages/kaleido/vendor/mathjax/MathJax.js?config=TeX-AMS-MML_SVG" onerror="logError('mathjax')"></script>
2024-11-07 18:32:49     <script src="file:///usr/local/lib/python3.11/site-packages/kaleido/vendor/kaleido_scopes.js" onerror="logError('scoper')"></script>  </head>
2024-11-07 18:32:49   <body style="{margin: 0; padding: 0;}"><img id="kaleido-image"><img></body>
2024-11-07 18:32:49 </html>
2024-11-07 18:32:49 ***********************end************************
2024-11-07 18:32:49 No loop, no thread
2024-11-07 18:32:49 STDERR: <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>
2024-11-07 18:32:49 DEBUG REPORT:
2024-11-07 18:32:49 BROWSER_PATH: /opt/chrome/chrome-linux64/chrome
2024-11-07 18:32:49 USER_DATA_DIR: /root/.choreographer-zbihu12l
2024-11-07 18:32:49 read_jsons (blocking):
2024-11-07 18:32:49 write_json:
2024-11-07 18:32:49 write_json: {"id": 0, "method": "Target.getTargets"}
2024-11-07 18:32:49 Starting watchdog
2024-11-07 18:32:49 wrote_json.
2024-11-07 18:32:49 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
2024-11-07 18:32:49  read_jsons pipe was closed, raising
2024-11-07 18:32:49 caught OSError in read() 
2024-11-07 18:32:49 PipeClosedError caught
2024-11-07 18:32:49 Browser is being closed because chrom* closed
2024-11-07 18:32:49 is_closed called with wait: 0
2024-11-07 18:32:49 Task exception was never retrieved
2024-11-07 18:32:49 future: <Task finished name='Task-195' coro=<Browser._open_async() done, defined at /usr/local/lib/python3.11/site-packages/choreographer/browser.py:213> exception=BrowserFailedError('The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.')>
2024-11-07 18:32:49 Traceback (most recent call last):
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/choreographer/browser.py", line 238, in _open_async
2024-11-07 18:32:49     await self.populate_targets()
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/choreographer/browser.py", line 563, in populate_targets
2024-11-07 18:32:49     response = await self.browser.send_command("Target.getTargets")
2024-11-07 18:32:49                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-07 18:32:49 choreographer.browser.BrowserClosedError: Command not completed because browser closed.
2024-11-07 18:32:49 
2024-11-07 18:32:49 The above exception was the direct cause of the following exception:
2024-11-07 18:32:49 
2024-11-07 18:32:49 Traceback (most recent call last):
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/choreographer/browser.py", line 241, in _open_async
2024-11-07 18:32:49     raise BrowserFailedError("The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.") from e
2024-11-07 18:32:49 choreographer.browser.BrowserFailedError: The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.
2024-11-07 18:32:49 Browser was already closed.
2024-11-07 18:32:49 Tempfile still exists?: False
2024-11-07 18:32:49 2024-11-07 09:32:49.090 Uncaught app exception
2024-11-07 18:32:49 Traceback (most recent call last):
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
2024-11-07 18:32:49     result = func()
2024-11-07 18:32:49              ^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec
2024-11-07 18:32:49     exec(code, module.__dict__)
2024-11-07 18:32:49   File "/app/app.py", line 10, in <module>
2024-11-07 18:32:49     pg.run()
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/streamlit/navigation/page.py", line 303, in run
2024-11-07 18:32:49     exec(code, module.__dict__)
2024-11-07 18:32:49   File "/app/simulations.py", line 424, in <module>
2024-11-07 18:32:49     summary_svg = summary_chart.to_image(format="svg", width=896, height=432, scale=1)
2024-11-07 18:32:49                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/plotly/basedatatypes.py", line 3772, in to_image
2024-11-07 18:32:49     return pio.to_image(self, *args, **kwargs)
2024-11-07 18:32:49            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/plotly/io/_kaleido.py", line 143, in to_image
2024-11-07 18:32:49     img_bytes = scope.transform(
2024-11-07 18:32:49                 ^^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/kaleido/scopes/plotly.py", line 218, in transform
2024-11-07 18:32:49     img = kaleido.to_image_block(spec, Path(self._tempfile.name).absolute(), self._topojson, self._mapbox_access_token, debug=debug)
2024-11-07 18:32:49           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/kaleido/__init__.py", line 43, in to_image_block
2024-11-07 18:32:49     return asyncio.run(to_image(spec, f, topojson, mapbox_token, debug=debug))
2024-11-07 18:32:49            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
2024-11-07 18:32:49     return runner.run(main)
2024-11-07 18:32:49            ^^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
2024-11-07 18:32:49     return self._loop.run_until_complete(task)
2024-11-07 18:32:49            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
2024-11-07 18:32:49     return future.result()
2024-11-07 18:32:49            ^^^^^^^^^^^^^^^
2024-11-07 18:32:49   File "/usr/local/lib/python3.11/site-packages/kaleido/__init__.py", line 52, in to_image
2024-11-07 18:32:49     async with (
2024-11-07 18:32:49 choreographer.browser.BrowserFailedError: Close() was called before the browser finished opening- maybe it crashed?
vfilter commented 2 weeks ago

Okay, I was able to solve this by using Chromium instead of Chrome in the Docker container. Here is the Dockerfile that ended up working. Thank you so much for taking a look at this 🙇

FROM python:3.11-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ARG MOTHERDUCK_TOKEN=${MOTHERDUCK_TOKEN}
ENV MOTHERDUCK_TOKEN=${MOTHERDUCK_TOKEN}

ARG APP_DIR="/app/"
RUN mkdir -p ${APP_DIR}
WORKDIR ${APP_DIR}

COPY ./src/build.py ${APP_DIR}

RUN uv pip install --system duckdb==1.1.2
RUN uv run build.py

FROM python:3.11-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

ARG APP_DIR="/app/"
RUN mkdir -p ${APP_DIR}
WORKDIR ${APP_DIR}

# Install dependencies
RUN apt-get update -y && apt-get install -y wget xvfb unzip jq curl

# Install Chromium and its dependencies
RUN apt-get update && apt-get install -y \
    chromium \
    libnss3 \
    libatk-bridge2.0-0 \
    libdrm2 \
    libxkbcommon0 \
    libxcomposite1 \
    libxdamage1 \
    libxfixes3 \
    libgbm1 \
    libasound2 \
    libpangocairo-1.0-0 \
    libgtk-3-0 \
    fonts-liberation \
    && rm -rf /var/lib/apt/lists/*

ENV PATH="$PATH:/usr/bin/chromium"

COPY ./uv.lock /app/
COPY ./pyproject.toml /app/
COPY ./src/ /app/

RUN uv pip install --system -r pyproject.toml

COPY --from=build ${APP_DIR}/data.duckdb ${APP_DIR}

EXPOSE 8501

HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health

ENTRYPOINT ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]
ayjayt commented 2 weeks ago

I'm glad that worked out:

2024-11-07 18:32:49 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2

That was the original problem, which seems to be because your docker image and that chrome download are compiled against different libc's- slim uses musl, and chrome uses glibc.

I'm going to close this! Thanks everyone.

mschoettle commented 2 weeks ago

@ayjayt Are there any plans to remove those GPU-related errors?

ayjayt commented 2 weeks ago

@mschoettle I'm going to take a look at your configuration because they should be hidden- normally chrome-based errors pass through if debug is set to the highest.

Unfortunately, these are chrome errors, coming from the chrome process. There is now a Browser(..., enable_gpu=True) option that can be used if you have a GPU... but the chromium team have an open issue as to why there are GPU errores when GPU is disabled so unless they resolve it, the best we can do is either enable_gpu or hide it. But it should already be hidden.

edit: found the issue

ayjayt commented 2 weeks ago

Okay, I can push a fix tonight @mschoettle , it'll make it into rc6 over the next few days, probably by Tuesday @gvwilson.

Thanks!

edit: done