search

apache / superset

Apache Superset is a Data Visualization and Data Exploration Platform
https://superset.apache.org/
Apache License 2.0
60.67k stars 13.12k forks source link

helm 0.12.6 fails on init db with #27420

Open AronsonDan opened 4 months ago

AronsonDan commented 4 months ago

Bug description

While trying to deploy on kubernetes:

Im getting the following error in finaloop-superset-init-db pod:

superset-init-db Upgrading DB schema...

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 568, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (cryptography 42.0.5 (/usr/local/lib/python3.9/site-packages), Requirement.parse('cryptography<41.1.0,>=41.0.2'), {
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
  File "/usr/local/bin/superset", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/superset/cli/main.py", line 28, in <module>
    from superset.cli.lib import normalize_token
  File "/app/superset/cli/lib.py", line 20, in <module>
    from superset import config
  File "/app/superset/config.py", line 39, in <module>
    import pkg_resources
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3243, in <module>
    def _initialize_master_working_set():
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 570, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 583, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'cryptography<41.1.0,>=41.0.2' distribution was not found and is required by apache-superset

Im using my own docker image. Docker file:

FROM apache/superset:3.1.0

USER root

RUN apt-get update && \
    apt-get install --no-install-recommends -y firefox-esr && \
    apt-get install wget

ENV GECKODRIVER_VERSION=0.33.0
RUN wget -q https://github.com/mozilla/geckodriver/releases/download/v${GECKODRIVER_VERSION}/geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz && \
    tar -x geckodriver -zf geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz -O > /usr/bin/geckodriver && \
    chmod 755 /usr/bin/geckodriver && \
    rm geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz

RUN pip install --no-cache gevent psycopg2 redis snowflake-sqlalchemy

USER superset

Would love to hear if you guys have any solution for it :-)

How to reproduce the bug

  1. build the docker file
  2. insert it into the k8s helm chart values yaml file
  3. deploy superset

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

16

Browser

Chrome

Additional context

No response

Checklist

AronsonDan commented 4 months ago

I have drilled down a bit into the installation and discovered that it is happening only when trying to install snowflake-sqlalchemy

craig-rueda commented 4 months ago

Looks like there's an issue with the cryptography lib. Try pip install -r requirements/docker.txt. That should get all the required libs.

AronsonDan commented 4 months ago

I have found an ugly solution to the issue: After installing snowflake-sqlalchemy i have installed the required version of cryptography

FROM apache/superset:3.1.0-py310

USER root

RUN apt-get update && \
    apt-get install --no-install-recommends -y firefox-esr && \
    apt-get install wget

ENV GECKODRIVER_VERSION=0.33.0
RUN wget -q https://github.com/mozilla/geckodriver/releases/download/v${GECKODRIVER_VERSION}/geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz && \
    tar -x geckodriver -zf geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz -O > /usr/bin/geckodriver && \
    chmod 755 /usr/bin/geckodriver && \
    rm geckodriver-v${GECKODRIVER_VERSION}-linux64.tar.gz

RUN pip install --no-cache gevent snowflake-sqlalchemy cryptography==41.0.2

USER superset

@craig-rueda isnt the apache/superset:3.1.0-py310 include these requirements in the first place? I guess one of the superset dependencies is conflicting with one of the snowflake-sqlalchemy dependencies which can prevent users from integrating snowflake with superset

AronsonDan commented 4 months ago