`dbt==0.18.1` Package failing to install starting on March 10, 2021.

jspreddy commented 3 years ago

Describe the bug

Package failed to install dbt==0.18.1 starting on March 10, 2021.

Steps To Reproduce

Inside docker ubuntu:bionic with python 3. (see Dockerfile below for details)
Installing package dbt==0.18.1 started failing on March 10, 2021.
This seems to be due to a failure while installing a dependency of dbt-core==0.18.1 namely agate<2,>=1.6 which resolves to agate==1.6.2.
agate 1.6.2 was released on March 10, 2021. agate release history

Expected behavior

It should install properly.

Summary log

Failure while -> Collecting PyICU>=2.4.2 (from agate<2,>=1.6 -> dbt-core==0.18.1 -> dbt==0.18.1 -> -r ./requirements.txt (line 1))

Detailed logs

Collecting dbt==0.18.1 (from -r ./requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/57/06/b9a5ccb8c5e89d7295ef6a46d3c1cfd58fa48a2e705d7f03284590eeafa2/dbt-0.18.1-py3-none-any.whl
.
.
.
Collecting PyICU>=2.4.2 (from agate<2,>=1.6->dbt-core==0.18.1->dbt==0.18.1->-r ./requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/31/46/fa08c8efae2951e67681ec24319f789fc1a74e2096dd74373e34c79319de/PyICU-2.6.tar.gz (233kB)
    Complete output from command python setup.py egg_info:
    (running 'icu-config --version')
    (running 'pkg-config --modversion icu-i18n')
    Traceback (most recent call last):
      File "/tmp/pip-build-bwz5wo1k/PyICU/setup.py", line 63, in <module>
        ICU_VERSION = os.environ['ICU_VERSION']
      File "/usr/lib/python3.6/os.py", line 669, in __getitem__
        raise KeyError(key) from None
    KeyError: 'ICU_VERSION'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/tmp/pip-build-bwz5wo1k/PyICU/setup.py", line 66, in <module>
        ICU_VERSION = check_output(('icu-config', '--version')).strip()
      File "/tmp/pip-build-bwz5wo1k/PyICU/setup.py", line 19, in check_output
        return subprocess_check_output(popenargs)
      File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
        **kwargs).stdout
      File "/usr/lib/python3.6/subprocess.py", line 423, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
        restore_signals, start_new_session)
      File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'icu-config': 'icu-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/tmp/pip-build-bwz5wo1k/PyICU/setup.py", line 69, in <module>
        ICU_VERSION = check_output(('pkg-config', '--modversion', 'icu-i18n')).strip()
      File "/tmp/pip-build-bwz5wo1k/PyICU/setup.py", line 19, in check_output
        return subprocess_check_output(popenargs)
      File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
        **kwargs).stdout
      File "/usr/lib/python3.6/subprocess.py", line 423, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
        restore_signals, start_new_session)
      File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'pkg-config': 'pkg-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-bwz5wo1k/PyICU/setup.py", line 74, in <module>
        ''')
    RuntimeError:
    Please install pkg-config on your system or set the ICU_VERSION environment
    variable to the version of ICU you have installed.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-bwz5wo1k/PyICU/

The output of dbt --version:

N/A. It would be 0.18.1

The operating system you're using: ubuntu:bionic

The output of python --version: Python 3.6.1

Additional context

Installing dbt==0.18.1 in a Dockerfile during build process via requirements.txt

Dockerfile:

FROM ubuntu:bionic

RUN apt-get update \
    && apt-get install -y  \
    gcc \
    groff \
    jq \
    libevent-dev \
    libffi-dev \
    libpq-dev \
    make \
    musl-dev \
    netcat-openbsd \
    python-dev \
    python3-pip \
    python3-venv \
    && rm -Rf /var/lib/apt/lists/*

COPY ./requirements.txt ./requirements.txt
RUN pip3 install -r ./requirements.txt   # <---fails here
.
.
.

Requirements.txt

dbt==0.18.*
awscli==1.*
dsnparse==0.*
boto3==1.*
jmespath==0.*

jspreddy commented 3 years ago

For anyone experiencing the same issue, here is a workaround.

In your requirements.txt lock the agate version to agate==1.6.1.

requirements.txt

dbt==0.18.*
# dbt's 3rd level dependency broken, so locking said dependency "agate" to working version 1.6.1
agate==1.6.1
.
.
.

jtcohen6 commented 3 years ago

Two other workarounds (thanks to @sean-rose in Slack):

Upgrading pip + setuptools to the latest version does allow the dbt installation to complete, despite the PyICU error—but the package is not installed, and it's likely to have a subtle and negative effect on agate/dbt operation.
Install PyICU via the instructions here. On MacOS, it looks like the recommendation is brew install icu4c, with some follow-on steps to ensure that icu-config utility is accessible via the PATH. After doing this, dbt installation can complete with the latest version of agate, and its new dependency PyICU successfully installed.

However, both of those workarounds seem far from ideal! In the meantime, I agree it makes most sense to pin dbt's dependency to agate>=1.6,<1.6.2. This shouldn't be a breaking change for any users, so we'll try to include the fix in v0.19.1.

dbt-labs / dbt-core