Closed wendytang closed 1 year ago
@wendytang can you please share a Python snippet where you see this error along with the Python version that you're using so that we can replicate this error?
Hi @desertaxle, I'm using python 3.9.5, and this error occurs at poetry install
- I've attached the backports-zoneinfo portion of the lock file:
[[package]]
name = "backports-zoneinfo"
version = "0.2.1"
description = "Backport of the standard library zoneinfo module"
category = "main"
optional = false
python-versions = ">=3.6"
files = [
{file = "backports.zoneinfo-0.2.1-cp36-cp36m-macosx_10_14_x86_64.whl", hash = "sha256:da6013fd84a690242c310d77ddb8441a559e9cb3d3d59ebac9aca1a57b2e18bc"},
{file = "backports.zoneinfo-0.2.1-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:89a48c0d158a3cc3f654da4c2de1ceba85263fafb861b98b59040a5086259722"},
{file = "backports.zoneinfo-0.2.1-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:1c5742112073a563c81f786e77514969acb58649bcdf6cdf0b4ed31a348d4546"},
{file = "backports.zoneinfo-0.2.1-cp36-cp36m-win32.whl", hash = "sha256:e8236383a20872c0cdf5a62b554b27538db7fa1bbec52429d8d106effbaeca08"},
{file = "backports.zoneinfo-0.2.1-cp36-cp36m-win_amd64.whl", hash = "sha256:8439c030a11780786a2002261569bdf362264f605dfa4d65090b64b05c9f79a7"},
{file = "backports.zoneinfo-0.2.1-cp37-cp37m-macosx_10_14_x86_64.whl", hash = "sha256:f04e857b59d9d1ccc39ce2da1021d196e47234873820cbeaad210724b1ee28ac"},
{file = "backports.zoneinfo-0.2.1-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:17746bd546106fa389c51dbea67c8b7c8f0d14b5526a579ca6ccf5ed72c526cf"},
{file = "backports.zoneinfo-0.2.1-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:5c144945a7752ca544b4b78c8c41544cdfaf9786f25fe5ffb10e838e19a27570"},
{file = "backports.zoneinfo-0.2.1-cp37-cp37m-win32.whl", hash = "sha256:e55b384612d93be96506932a786bbcde5a2db7a9e6a4bb4bffe8b733f5b9036b"},
{file = "backports.zoneinfo-0.2.1-cp37-cp37m-win_amd64.whl", hash = "sha256:a76b38c52400b762e48131494ba26be363491ac4f9a04c1b7e92483d169f6582"},
{file = "backports.zoneinfo-0.2.1-cp38-cp38-macosx_10_14_x86_64.whl", hash = "sha256:8961c0f32cd0336fb8e8ead11a1f8cd99ec07145ec2931122faaac1c8f7fd987"},
{file = "backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_i686.whl", hash = "sha256:e81b76cace8eda1fca50e345242ba977f9be6ae3945af8d46326d776b4cf78d1"},
{file = "backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:7b0a64cda4145548fed9efc10322770f929b944ce5cee6c0dfe0c87bf4c0c8c9"},
{file = "backports.zoneinfo-0.2.1-cp38-cp38-win32.whl", hash = "sha256:1b13e654a55cd45672cb54ed12148cd33628f672548f373963b0bff67b217328"},
{file = "backports.zoneinfo-0.2.1-cp38-cp38-win_amd64.whl", hash = "sha256:4a0f800587060bf8880f954dbef70de6c11bbe59c673c3d818921f042f9954a6"},
{file = "backports.zoneinfo-0.2.1.tar.gz", hash = "sha256:fadbfe37f74051d024037f223b8e001611eac868b5c5b06144ef4d8b799862f2"},
]
[package.extras]
tzdata = ["tzdata"]
Any pyproject.toml file with Prefect as a dependency.
[tool.poetry]
name = "example_flow"
version = "0.1.0"
description = "Demonstrates a flow"
authors = ["Wendy Tang <wendytang@squareup.com>"]
[tool.poetry.dependencies]
python = ">=3.8,<3.11"
prefect = ">=2.10.0"
Hey there!
I cannot seem to reproduce this - can you provide any additional detail that might help us figure out what's happening here?
$ conda --version
conda 22.9.0
$ python --version
Python 3.9.16
$ poetry --version
Poetry (version 1.5.1)
$ poetry install
Installing dependencies from lock file
No dependencies to install or update
I ran poetry init
and created a simple placeholder project -- this is the pyproject.toml
that it gave me:
[tool.poetry]
name = "prefect9943"
version = "0.1.0"
description = ""
authors = ["Jonathan Yu <jonathan@prefect.io>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.9"
prefect = "^2.10.17"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
I don't know dateparser
well enough to know whether a migration would be straightforward, though we only have a handful of references to it, so it certainly seems practical. I've just been unable to confirm the original issue as reported, and I don't know if there's some other compatibility issue introduced by Ubuntu.
Looking at your report more closely, the problem seems to be that Python is falling back to compiling the package from source instead of installing the wheel:
clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/builds/default-i-0c87c589b05e37c77-2/cash/cash-ml-prefect2/cash_ml_prefect2/examples/example_snowflake_flow/.venv/include -I/root/.cache/hermit/pkg/python3-3.9.5/install/include/python3.9 -c lib/zoneinfo_module.c -o build/temp.linux-x86_64-cpython-39/lib/zoneinfo_module.o -std=c99
error: command 'clang' failed: No such file or directory
I think you can resolve this by installing build-essentials
or clang
packages using apt
, but I'm not sure why the published wheels are insufficient here 🤔
I also tried the supplied image and everything seemed to work for me (I've abbreviated this output to remove some output):
$ podman run --rm -it public.ecr.aws/lts/ubuntu:20.04_stable
Trying to pull public.ecr.aws/lts/ubuntu:20.04_stable...
Getting image source signatures
Copying blob sha256:fd7ea62b7af53fe93cb837560acd7903ee12e26127affb518c2b9efc6500d503
Copying config sha256:bf6a7b6c94869fe72f4c96fd84e37eebad67cc4d521b4cfdb9434d498925a49a
Writing manifest to image destination
Storing signatures
root@1092d61079ff:/# apt update
root@1092d61079ff:/# apt install python3 python3-pip python3.9
root@1092d61079ff:/# python3.9 -m pip install dateparser
Collecting dateparser
Downloading dateparser-1.1.8-py2.py3-none-any.whl (293 kB)
|████████████████████████████████| 293 kB 2.4 MB/s
Collecting tzlocal
Downloading tzlocal-5.0.1-py3-none-any.whl (20 kB)
Collecting regex!=2019.02.19,!=2021.8.27
Downloading regex-2023.6.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (768 kB)
|████████████████████████████████| 768 kB 7.2 MB/s
Collecting python-dateutil
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
|████████████████████████████████| 247 kB 13.3 MB/s
Collecting pytz
Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB)
|████████████████████████████████| 502 kB 13.3 MB/s
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: tzlocal, regex, six, python-dateutil, pytz, dateparser
Successfully installed dateparser-1.1.8 python-dateutil-2.8.2 pytz-2023.3 regex-2023.6.3 six-1.16.0 tzlocal-5.0.1
Hey @jawnsy - we communicated offline earlier regarding the fix. Would you mind summarizing your findings here for my understanding - why we need the dev versions of python3?
RUN apt-get -qq install -y --no-install-recommends build-essential python3-dev python3.9-dev
In this instance, the problem is that installing dateparser
requires backports.zoneinfo
, and for some reason, there were no precompiled wheels available.
When installing Python with apt-get --no-install-recommends
, the build toolchain and Python development headers will be missing from the system, resulting in a failure to build the wheel. The build-essential
virtual package includes the default build toolchain on Debian-based systems . The python3-dev
and python3.9-dev
packages include files used for building/compiling C language programs that integrate with Python.
Most importantly, the build-essential package depends on libc-dev (C Library headers), gcc, g++, make - this is a minimal compiler toolchain used to build C-language programs:
# apt show build-essential
Package: build-essential
Version: 12.8ubuntu1.1
Priority: optional
Build-Essential: yes
Section: devel
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Matthias Klose <doko@debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 21.5 kB
Depends: libc6-dev | libc-dev, gcc (>= 4:9.2), g++ (>= 4:9.2), make, dpkg-dev (>= 1.17.11)
Task: ubuntu-mate-core, ubuntu-mate-desktop
Download-Size: 4664 B
APT-Sources: http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 Packages
Description: Informational list of build-essential packages
The python-dev headers include Python.h
, which is part of the C interface to CPython. This is used by any applications or libraries that write C code that need to work with Python, such as Python libraries implemented in C, or when embedding a Python runtime in a C program (i.e. this supports cross-language access of Python-to-C or C-to-Python).
To demonstrate the problem and solution, I've prepared the following Dockerfile
:
FROM public.ecr.aws/lts/ubuntu:20.04_stable
RUN apt-get -qq update
RUN apt-get -qq install -y --no-install-recommends python3.9 python3-pip
# these dependencies are necessary to build wheels (e.g. backports.zoneinfo)
RUN apt-get -qq install -y --no-install-recommends build-essential python3-dev python3.9-dev
RUN python3.9 -m pip -q install prefect
If you do not install build-essential, python3-dev, python3.9-dev, then pip
will not be able to compile Python libraries with a C language portion. This is okay if your system can use precompiled binaries (wheels), but for whatever reason, that does not seem to be the case here - it may have to do with an incompatibility with this base image and available wheels published by the maintainers of backports.zoneinfo
.
This results in an error similar to the one you shared in the "Error" section of your issue:
clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -fPIC -I/builds/default-i-0c87c589b05e37c77-2/cash/cash-ml-prefect2/cash_ml_prefect2/examples/example_snowflake_flow/.venv/include -I/root/.cache/hermit/pkg/python3-3.9.5/install/include/python3.9 -c lib/zoneinfo_module.c -o build/temp.linux-x86_64-cpython-39/lib/zoneinfo_module.o -std=c99
error: command 'clang' failed: No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for backports.zoneinfo
Failed to build backports.zoneinfo
ERROR: Could not build wheels for backports.zoneinfo, which is required to install pyproject.toml-based projects
This is trying to compile an extension (zoneinfo_module.c) but there are no system compilers installed (presumably the Python build toolchain looks for either gcc
or clang
), which results in an error message and thus a failure to build the package.
If you add a compiler but do not have python3-dev, then you'll get an error similar to the following:
Building wheels for collected packages: backports.zoneinfo
Building wheel for backports.zoneinfo (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3.9 /tmp/tmpmuocyhk0_in_process.py build_wheel /tmp/tmp10jkrvjc
cwd: /tmp/pip-install-hrs0avny/backports.zoneinfo
Complete output (53 lines):
/tmp/pip-build-env-x6o3nn2_/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
!!
********************************************************************************
The license_file parameter is deprecated, use license_files instead.
By 2023-Oct-30, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
********************************************************************************
!!
parsed = self.parsers.get(option_name, lambda x: x)(value)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-aarch64-cpython-39
creating build/lib.linux-aarch64-cpython-39/backports
copying src/backports/__init__.py -> build/lib.linux-aarch64-cpython-39/backports
creating build/lib.linux-aarch64-cpython-39/backports/zoneinfo
copying src/backports/zoneinfo/__init__.py -> build/lib.linux-aarch64-cpython-39/backports/zoneinfo
copying src/backports/zoneinfo/_common.py -> build/lib.linux-aarch64-cpython-39/backports/zoneinfo
copying src/backports/zoneinfo/_tzpath.py -> build/lib.linux-aarch64-cpython-39/backports/zoneinfo
copying src/backports/zoneinfo/_version.py -> build/lib.linux-aarch64-cpython-39/backports/zoneinfo
copying src/backports/zoneinfo/_zoneinfo.py -> build/lib.linux-aarch64-cpython-39/backports/zoneinfo
running egg_info
writing src/backports.zoneinfo.egg-info/PKG-INFO
writing dependency_links to src/backports.zoneinfo.egg-info/dependency_links.txt
writing requirements to src/backports.zoneinfo.egg-info/requires.txt
writing top-level names to src/backports.zoneinfo.egg-info/top_level.txt
reading manifest file 'src/backports.zoneinfo.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.png' under directory 'docs'
warning: no files found matching '*.svg' under directory 'docs'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_output'
adding license file 'LICENSE'
adding license file 'licenses/LICENSE_APACHE'
writing manifest file 'src/backports.zoneinfo.egg-info/SOURCES.txt'
copying src/backports/zoneinfo/__init__.pyi -> build/lib.linux-aarch64-cpython-39/backports/zoneinfo
copying src/backports/zoneinfo/py.typed -> build/lib.linux-aarch64-cpython-39/backports/zoneinfo
running build_ext
building 'backports.zoneinfo._czoneinfo' extension
creating build/temp.linux-aarch64-cpython-39
creating build/temp.linux-aarch64-cpython-39/lib
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.9 -c lib/zoneinfo_module.c -o build/temp.linux-aarch64-cpython-39/lib/zoneinfo_module.o -std=c99
lib/zoneinfo_module.c:1:10: fatal error: Python.h: No such file or directory
1 | #include "Python.h"
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for backports.zoneinfo
Failed to build backports.zoneinfo
ERROR: Could not build wheels for backports.zoneinfo which use PEP 517 and cannot be installed directly
This error is because zoneinfo_module.c
tries to load Python APIs and those are not installed with the python package by default (they are included in python3-dev
).
There is additionally an issue somewhere in dependency resolution, because this package should not be required for Python 3.9 or later, due to the inclusion of zoneinfo
in the Python standard library.
I think this may be because of how packages are being installed - the python3 package is actually Python 3.8, which requires the backport (since zoneinfo was introduced to the standard library in 3.9), so if you run pip install dateparser
then the output will look like this:
# pip install dateparser
Collecting dateparser
Downloading dateparser-1.1.8-py2.py3-none-any.whl (293 kB)
|████████████████████████████████| 293 kB 2.9 MB/s
Collecting regex!=2019.02.19,!=2021.8.27
Downloading regex-2023.6.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (771 kB)
|████████████████████████████████| 771 kB 7.0 MB/s
Collecting pytz
Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB)
|████████████████████████████████| 502 kB 7.6 MB/s
Collecting python-dateutil
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
|████████████████████████████████| 247 kB 8.8 MB/s
Collecting tzlocal
Downloading tzlocal-5.0.1-py3-none-any.whl (20 kB)
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting backports.zoneinfo; python_version < "3.9"
Using cached backports.zoneinfo-0.2.1.tar.gz (74 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Building wheels for collected packages: backports.zoneinfo
Building wheel for backports.zoneinfo (PEP 517) ... done
Created wheel for backports.zoneinfo: filename=backports.zoneinfo-0.2.1-cp38-cp38-linux_aarch64.whl size=82062 sha256=64191a3aba1b45c3eaeeda011757584902676234cbe733b32b5ad42f3a3988ca
Stored in directory: /root/.cache/pip/wheels/c7/de/cc/c405827ed64f81b56142f1e0075a970b2731b00d21983d54fb
Successfully built backports.zoneinfo
Installing collected packages: regex, pytz, six, python-dateutil, backports.zoneinfo, tzlocal, dateparser
Successfully installed backports.zoneinfo-0.2.1 dateparser-1.1.8 python-dateutil-2.8.2 pytz-2023.3 regex-2023.6.3 six-1.16.0 tzlocal-5.0.1
If you instead run python3.9 pip install dateparser
then it will correctly detect the zoneinfo included in the corelib:
# python3.9 -m pip install dateparser
Collecting dateparser
Using cached dateparser-1.1.8-py2.py3-none-any.whl (293 kB)
Collecting regex!=2019.02.19,!=2021.8.27
Downloading regex-2023.6.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (768 kB)
|████████████████████████████████| 768 kB 3.0 MB/s
Collecting pytz
Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting tzlocal
Using cached tzlocal-5.0.1-py3-none-any.whl (20 kB)
Collecting python-dateutil
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting six>=1.5
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: regex, pytz, tzlocal, six, python-dateutil, dateparser
Successfully installed dateparser-1.1.8 python-dateutil-2.8.2 pytz-2023.3 regex-2023.6.3 six-1.16.0 tzlocal-5.0.1
To sum up, there are a few issues here:
pip install
(or, I suppose, poetry install
) will resolve standard library dependencies using the system Python, and thus try to install backports.zoneinfo
. If you instead run python3.9 -m pip install
, then it will resolve correctly using the Python 3.9 dependenciesWe are considering removal of dateparser
as a dependency, but that would require a gradual deprecation process as it would be a backward-incompatible change. I've created a related issue here: https://github.com/PrefectHQ/prefect/issues/10096
Closing this issue as complete, given the above. There is a follow-up item regarding possible removal of dateparser, but I think this issue should be resolved now. Thanks for using Prefect!
First check
Bug summary
Hi team,
Prefect currently has a dependency on dateparser which depends on tzdata which depends on backports.zoneinfo. However, according to backports.zoneinfo documentation:
I've been running into this error trying to import backports.zoneinfo, so just wanted to flag here, if there's any way to move off of dateparser and hence backports.zoneinfo, it would be much appreciated.
Reproduction
Error
Versions
Additional context
No response