setuptools-70.0.0 doesn't seem to work with ParallelCluster

[rmarable-flaretx]

AWS ParallelCluster 3.9.0 and beyond don't seem to work with the most recently released version of setuptools (70.0.0) on OSX.

 ❯ pcluster version
Traceback (most recent call last):
  File "/opt/homebrew/bin/pcluster", line 5, in <module>
    from pcluster.cli.entrypoint import main
  File "/opt/homebrew/lib/python3.11/site-packages/pcluster/cli/entrypoint.py", line 32, in <module>
    import pcluster.api.controllers.cluster_compute_fleet_controller  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/pcluster/api/controllers/cluster_compute_fleet_controller.py", line 11, in <module>
    from pcluster.api.controllers.common import configure_aws_region, convert_errors, validate_cluster
  File "/opt/homebrew/lib/python3.11/site-packages/pcluster/api/controllers/common.py", line 20, in <module>
    from pkg_resources import packaging
ImportError: cannot import name 'packaging' from 'pkg_resources' (/opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py)

Revering back to setuptools-69.5.1 seems to fix the issue.

❯ pip install setuptools==69.5.1
Collecting setuptools==69.5.1
  Using cached setuptools-69.5.1-py3-none-any.whl.metadata (6.2 kB)
Using cached setuptools-69.5.1-py3-none-any.whl (894 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 70.0.0
    Uninstalling setuptools-70.0.0:
      Successfully uninstalled setuptools-70.0.0
Successfully installed setuptools-69.5.1
❯ pcluster version
{
  "version": "3.9.0"
}

I didn't test this, but I suspect the same is true for Linux as well.

[enrico-usai]

Hi @rmarable-flaretx ,

It's recommended to install pcluster in a Python virtual env, to avoid conflicts with other Python dependencies. See instructions here: https://docs.aws.amazon.com/parallelcluster/latest/ug/install-v3-virtual-environment.html

This will solve any kind of library incompatibility issues.

[rmarable-flaretx]

I am using a virtual Python environment. The issue is that the most recent version of setuptools doesn't seem to work; downgrading to 69.5.1 did.

[richard-vevo]

@rmarable-flaretx thanks for the workaround, reverting to setuptools==69.5.1 did the trick for me. Looks like setuptools==70.0.0 deprecated pkg_resources which is causing the error. https://setuptools.pypa.io/en/latest/pkg_resources.html

[enrico-usai]

I just found that we were aware of this issue, so we pinned setuptools a month ago: https://github.com/aws/aws-parallelcluster/pull/6251 the fix has been included starting from ParallelCluster 3.9.1

[rmarable-flaretx]

Thank you @enrico-usai!