Sqlalchemy breaks projects that need to support Pandas < 2.0

[aschreiber1]

https://github.com/apache/iceberg-python/blob/e2ed2eb7966210203631ac8ece61056479e59a23/pyproject.toml#L79

Ever since 0.7.0 the project no longer installs into an environment that has a depedency on sqlalchemy < 2.0. For example this yaml: dependencies:

• python=3.10
• sqlalchemy >=1.3.24,<2.0
• pyiceberg>=0.7.1,<1

Does not install via conda.

Encountered problems while solving:

• package pyiceberg-0.7.1-pyhd8ed1ab_0 has constraint sqlalchemy >=2.0.18,<3.0.0 conflicting with sqlalchemy-1.3.24-py310h5764c6d_1

Sqlalchemy less than 2.0 is needed for projects that support Pandas < 2.0 so this limits were we can use pyiceberg

[sungwy]

Hi @aschreiber1 - sorry that you are running into this issue. Unfortunately with Python projects, dealing with the dependency hell and making decisions on what remains the minimum version is a tough decision the community will have to continue making as the project evolves.

Could you explain a bit more about your setup?

sqlalchemy is an optional dependency that's only installed when using sql-sqlite or sql-postgres extra packages. And as seen in this PR, the sqlalchemy dependency and these extra packages were introduced together already pinned to this version: https://github.com/apache/iceberg-python/commit/be510c677a8e9ea99b2d7b10eb69260cf6f74a0c

So using sql-sqlite or sql-postgres extra packages even in lower versions of PyIceberg would have had the same version constraint on sqlalchemy

[aschreiber1]

Yeah totally appreciate depedency hell... So if i just do a basic conda install (conda env create) with the above envrionment.yml file, it fails for me that error. If I use pyiceberg 0.6.1 earlier it works fine. I agree its odd, it seems like those dependencies have been there, so i'm not quire sure what changed between the two that now makes this required.

Also I think because we are doing conda installs and not pip here conda can't handle the optional dependencies so it does everything?

[aschreiber1]

I realized this an issue specifically to the conda recipe for pyiceberg and not pyiceberg itself, you can see the dependency here: https://github.com/conda-forge/pyiceberg-feedstock/pull/22/files, I'll raise the issue there

[aschreiber1]

I realized this an issue specifically to the conda recipe for pyiceberg and not pyiceberg itself, you can see the dependency here: https://github.com/conda-forge/pyiceberg-feedstock/pull/22/files, I'll raise the issue there