search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.39k stars 14.35k forks source link

unicodecsv dependency makes apache-airflow Python package un-installable #42892

Closed Chais closed 1 month ago

Chais commented 1 month ago

Apache Airflow version

2.10.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Airflow cannot be installed in a venv with any system that properly solves dependencies and version requirements.

This happens because the lowest supported Python version of apache-airflow is 3.8, while the highest supported version of unicodecsv is 3.5. So there is no Python version that would be supported by apache-airflow and all of its dependencies, making it un-installable.

What you think should happen instead?

I know unicodecsv has been dropped as a dependency very recently, but no new version has been released since then.

How to reproduce

I encountered the issue with Poetry, but any tool that ensures all requirements for dependency packages are met will encounter the same issue.
I encountered the issue targeting Python 3.8, but since there is no overlap between supported Python versions between unicecsv and apache-airflow it doesn't really matter.

Paste the following pyproject.toml into an empty directory:

[tool.poetry]
name = "airflow-dependency-test"
version = "0.1.0"
description = ""
authos = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "~3.8"
apache-airflow = "2.10.2"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core >= 1.0.0"]
build-backend = "poetry-core.masonry.api"

In a terminal navigate to the directory and run poetry install

Operating System

Ubuntu 22.04

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 1 month ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

potiuk commented 1 month ago

This is invalid.

Agree - we missed that we have an old and not really used package before, but it should not be prevented from being installed on higher airflow version.

While unicodecsv had indeed only specified classifiers for up to Python 3.5 it had no requirements set to be only used for Python < 3.5 and since it had no dependencies on its own and it did not have python_requires that prevented it from being installed.

What you are likely referring to when you mane are Trove Classifiers:

Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Programming Language :: Python :: Implementation :: CPython

The trove classifiers are defined in https://peps.python.org/pep-0301/ and they are only there to classify projects when you search for them, but they should not be used by packaging tools to decide if they are installed for specific python version (`python_requires is the way to do that).

I am not sure what errors you had an which tools you used besides poetry - you missed to explain what kind of error you had when you tried to install airflow. But I am quite sure it is installable for higher python versio. In fact I just did it and it works (python 3.11 in this case):

[jarek:~/code/airflow] fix-dependencies-removal+ 1 ± python --version
Python 3.11.6
[jarek:~/code/airflow] fix-dependencies-removal+ ± pip install unicodecsv
DEPRECATION: Loading egg at /Users/jarek/.pyenv/versions/3.11.6/lib/python3.11/site-packages/Flask_OpenID-1.3.1-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
DEPRECATION: Loading egg at /Users/jarek/.pyenv/versions/3.11.6/lib/python3.11/site-packages/defusedxml-0.8.0rc2-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
DEPRECATION: Loading egg at /Users/jarek/.pyenv/versions/3.11.6/lib/python3.11/site-packages/python3_openid-3.2.0-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
Collecting unicodecsv
  Using cached unicodecsv-0.14.1-py3-none-any.whl
Installing collected packages: unicodecsv
Successfully installed unicodecsv-0.14.1

The setup.py of unicodecsv does not contain python_requires so generally no tool should be limited to installing it on higher python version than 3.5 (and since it is very simple code, it generally installs and works fine on higher versions).

This is setup.py of latest unicode.csv - there is no limitation of Python version:

setup(
    name='unicodecsv',
    version=version,
    description="Python2's stdlib csv module is nice, but it doesn't support unicode. This module is a drop-in replacement which *does*.",
    long_description=open(os.path.join(os.path.dirname(__file__), 'README.rst'), 'rb').read().decode('utf-8'),
    author='Jeremy Dunck',
    author_email='jdunck@gmail.com',
    url='https://github.com/jdunck/python-unicodecsv',
    packages=find_packages(),
    tests_require=['unittest2>=0.5.1'],
    test_suite='runtests.get_suite',
    license='BSD License',
    classifiers=['Development Status :: 5 - Production/Stable',
                'Intended Audience :: Developers',
                'License :: OSI Approved :: BSD License',
                'Natural Language :: English',
                'Programming Language :: Python :: 2.6',
                'Programming Language :: Python :: 2.7',
                'Programming Language :: Python :: 3.3',
                'Programming Language :: Python :: 3.4',
                'Programming Language :: Python :: 3.5',
                'Programming Language :: Python :: Implementation :: PyPy',
                'Programming Language :: Python :: Implementation :: CPython',],

I am not sure how you came to the conclusion that unicodecsv is not installable for Python 3.8+ - maybe you can explain what exactly error you experienced and what exactly you did in a follow-up discussion but I can assure you Airflow is instalable so this issue is invalid.

Converting to discussion if you need more. I will also make sure to check if unicodecsv removal is cherry-picked to 2-10 branch - so also we will remove it in the next release, but this is more of a cleanup than problem with installation.