Cannot install mysql client onto 4.1.0rc2 (pkg-config not found)

ilsaloving commented 1 month ago

Bug description

I am trying to create a new docker container using 4.1.0rc2 as a base, and when I try to install the mysql client, I get pkg-config: not found. Stack trace is in Additional Content. Below is my docker file. Not sure if this should be pre-packaged with the superset container, or up to the user to install when extending the image?

FROM apache/superset:4.1.0rc2

USER root
RUN <<EOF
    apt-get update
    apt-get install -y curl links zip unzip basez
    apt-get purge -y --auto-remove
EOF
COPY superset_config.py superset-init.sh /app/
COPY --from=common get-secret.sh discover-roleid.sh /usr/local/bin/

USER superset

RUN <<EOF
    pip install --upgrade pip 
    pip install  \
    shillelagh[gsheetsapi] \
    gunicorn[gevent] \
    pymssql \
    elasticsearch-dbapi \
    elasticsearch-dbapi[opendistro] \
    mysqlclient \
    psycopg2 \
    trino \
    trino[sqlalchemy] \
    playwright \
    python-ldap \
    pip-system-certs \

EOF

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.10

Node version

Not applicable

Browser

Not applicable

Additional context

[superset stage-0 3/5] RUN <<EOF (pip install --upgrade pip...): 1.139 Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (24.2) 2.100 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. 3.571 Collecting pymssql 3.786 Downloading pymssql-2.3.1-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (7.1 kB) 4.044 Collecting elasticsearch-dbapi 4.085 Downloading elasticsearch_dbapi-0.2.11-py3-none-any.whl.metadata (8.4 kB) 4.506 Collecting mysqlclient 4.546 Downloading mysqlclient-2.2.4.tar.gz (90 kB) 4.754 Installing build dependencies: started 8.710 Installing build dependencies: finished with status 'done' 8.713 Getting requirements to build wheel: started 8.932 Getting requirements to build wheel: finished with status 'error' 8.947 error: subprocess-exited-with-error 8.947 8.947 × Getting requirements to build wheel did not run successfully. 8.947 │ exit code: 1 8.947 ╰─> [27 lines of output] 8.947 /bin/sh: 1: pkg-config: not found 8.947 /bin/sh: 1: pkg-config: not found 8.947 /bin/sh: 1: pkg-config: not found 8.947 Trying pkg-config --exists mysqlclient 8.947 Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127. 8.947 Trying pkg-config --exists mariadb 8.947 Command 'pkg-config --exists mariadb' returned non-zero exit status 127. 8.947 Trying pkg-config --exists libmariadb 8.947 Command 'pkg-config --exists libmariadb' returned non-zero exit status 127. 8.947 Traceback (most recent call last): 8.947 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in 8.947 main() 8.947 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main 8.947 json_out['return_val'] = hook(**hook_input['kwargs']) 8.947 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel 8.947 return hook(config_settings) 8.947 File "/tmp/pip-build-env-sncefshn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel 8.947 return self._get_build_requires(config_settings, requirements=[]) 8.947 File "/tmp/pip-build-env-sncefshn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires 8.947 self.run_setup() 8.947 File "/tmp/pip-build-env-sncefshn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup 8.947 exec(code, locals()) 8.947 File "", line 155, in 8.947 File "", line 49, in get_config_posix 8.947 File "", line 28, in find_package_name 8.947 Exception: Can not find valid pkg-config name. 8.947 Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually 8.947 [end of output] 8.947 8.947 note: This error originates from a subprocess, and is likely not a problem with pip. 9.005 error: subprocess-exited-with-error 9.005 9.005 × Getting requirements to build wheel did not run successfully. 9.005 │ exit code: 1 9.005 ╰─> See above for output. 9.005 9.005 note: This error originates from a subprocess, and is likely not a problem with pip.

failed to solve: process "/bin/sh -c \tpip install --upgrade pip\n\tpip install \\n\tshillelagh[gsheetsapi] \\n\tgunicorn[gevent] \\n\tpymssql \\n\telasticsearch-dbapi \\n\telasticsearch-dbapi[opendistro] \\n\tmysqlclient \\n\tpsycopg2 \\n\ttrino \\n\ttrino[sqlalchemy] \\n\tplaywright \\n\tpython-ldap \\n\tpip-system-certs \\n\n" did not complete successfully: exit code: 1

Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

dosubot[bot] commented 1 month ago

The error you're encountering is due to the absence of pkg-config in your Docker image. To resolve this, you need to install pkg-config before attempting to install mysqlclient. You can modify your Dockerfile as follows:

FROM apache/superset:4.1.0rc2

USER root
RUN apt-get update && apt-get install -y \
    pkg-config \
    default-libmysqlclient-dev \
    curl links zip unzip basez && \
    apt-get purge -y --auto-remove

RUN pip install --upgrade pip --root-user-action && \
    pip install --root-user-action \
    shillelagh[gsheetsapi] \
    gunicorn[gevent] \
    pymssql \
    elasticsearch-dbapi \
    elasticsearch-dbapi[opendistro] \
    mysqlclient \
    psycopg2 \
    trino \
    trino[sqlalchemy] \
    playwright \
    python-ldap \
    pip-system-certs

# Custom init
COPY superset_config.py superset-init.sh /app/

USER superset

This should resolve the pkg-config: not found error and allow you to install mysqlclient successfully.

To continue talking to Dosu, mention @dosu.

ilsaloving commented 1 month ago

I was able to solve it by installing pkg-config, but now I'm getting complaints about missing gcc. I thought this was a different issue from #29727 but apparently it's the same thing.

I'm wondering if there needs to be an easier mechanism or guidance for people to install needed database drivers. I can see this being a problem for less experienced people.

ilsaloving commented 1 month ago

Since the base image is running off Debian 12, I was able to make the following adjustments to my Dockerfile so everything would work without requiring a full compilation environment. I'm still fiddling with it, but this gets past the build errors.

Note: I also had to install the distro packages for ldap and psycopg2 as well, and then install psycopg2-binary via pip or superset wouldn't find it.

FROM apache/superset:4.1.0rc2

USER root
RUN <<EOF
    apt-get update
    apt-get upgrade -y
    apt-get install -y curl links zip unzip basez python3-mysqldb python3-ldap python3-psycopg2 python3-psycopg2cffi
    apt-get purge -y --auto-remove
EOF

COPY superset_config.py superset-init.sh /app/

RUN <<EOF
    pip --root-user-action install --upgrade pip 
    pip install  \
    psycopg2-binary \
    shillelagh[gsheetsapi] \
    gunicorn[gevent] \
    pymssql \
    elasticsearch-dbapi \
    elasticsearch-dbapi[opendistro] \
    trino \
    trino[sqlalchemy] \
    playwright \
    pip-system-certs \
    Pillow \

EOF

USER superset

ilsaloving commented 1 month ago

Final update... I will create a separate ticket for this but the distro-provided python3-ldap does not work, because it is installed for python 3.11, but superset uses 3.10. While I could alter the system to point to 3.11 instead, I'm hesitant to do that since it's not a supported configuration.

rusackas commented 3 weeks ago

Sounds like we're all set to close this then? Let me know if you think there's anything left to tackle. If you want to add any docs, that'd be encouraged. Indeed we don't yet support 3.11, but we will in time, no doubt.

ilsaloving commented 3 weeks ago

I guess, but only because this is a symptom of a bigger problem.

apache / superset