Import fails with `protobuf>3.20.x`

[pat-alt]

Hi there,

Not sure if this is expected, but importing CARLA fails when protobuf>3.20.x is installed (see below). I've tried various default pyenv version (3.7.x) and they all shipped with protobuf>3.20.x, that's how I came across this.

Many thanks!

TypeError                                 Traceback (most recent call last)
/var/folders/zn/yz12tx_128ggf8_ln3nn69p00000gn/T/ipykernel_93359/2127982349.py in <module>
      3 sys.path.append('..')
      4 
----> 5 from carla.recourse_methods import Dice, Wachter
      6 from copy import deepcopy
      7 # from model_shifts import (DynamicCsvCatalog,

~/.pyenv/versions/3.7.10/lib/python3.7/site-packages/carla/__init__.py in <module>
     16 from ._version import __version__
     17 from .data import Data, DataCatalog
---> 18 from .evaluation import Benchmark
     19 from .models import MLModel, MLModelCatalog
     20 from .recourse_methods import RecourseMethod

~/.pyenv/versions/3.7.10/lib/python3.7/site-packages/carla/evaluation/__init__.py in <module>
      1 # flake8: noqa
      2 
----> 3 from .benchmark import Benchmark
      4 from .distances import get_distances
      5 from .nearest_neighbours import yNN

~/.pyenv/versions/3.7.10/lib/python3.7/site-packages/carla/evaluation/benchmark.py in <module>
      6 
...
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

[JohanvandenHeuvel]

Hey, yeah I don't really know what is going on here, has to do with this and see this. So as far as I know this is not because of any issue on our end, but rather because we rely on other libraries which need to fix this.

Doing one of the workarounds should fix it.

[pat-alt]

Thanks, just downgrading that library worked. Related to this, are there any plans to bump up compatibility to more recent Python versions?

[JohanvandenHeuvel]

As far as I know, any Python 3.x version is backwards compatible with other Python 3.x version. So you can use CARLA with more recent Python versions if you want.

[pat-alt]

Hmmm I run into trouble when trying to do that. I've just tried it with Python 3.8 as follows:

pyenv install 3.8.1
pyenv local 3.8.1

Then running pip install carla-recourse yields the following error:

Collecting carla-recourse
  Using cached carla_recourse-0.0.5-py3-none-any.whl (138 kB)
Collecting pandas==1.1.4
  Downloading pandas-1.1.4-cp38-cp38-macosx_10_9_x86_64.whl (10.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.1/10.1 MB 9.0 MB/s eta 0:00:00
Collecting ipython
  Using cached ipython-8.4.0-py3-none-any.whl (750 kB)
Collecting causalgraphicalmodels==0.0.4
  Using cached causalgraphicalmodels-0.0.4-py3-none-any.whl (11 kB)
Collecting dice-ml==0.5
  Using cached dice_ml-0.5-py3-none-any.whl (224 kB)
Collecting scikit-learn==0.23.2
  Downloading scikit_learn-0.23.2-cp38-cp38-macosx_10_9_x86_64.whl (7.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 9.0 MB/s eta 0:00:00
Collecting mip==1.12.0
  Using cached mip-1.12.0-py3-none-any.whl (47.1 MB)
Collecting torch==1.7.0
  Downloading torch-1.7.0-cp38-none-macosx_10_9_x86_64.whl (108.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.1/108.1 MB 5.8 MB/s eta 0:00:00
Collecting xgboost==1.4.2
  Using cached xgboost-1.4.2-py3-none-macosx_10_14_x86_64.macosx_10_15_x86_64.macosx_11_0_x86_64.whl (1.2 MB)
Collecting lime==0.2.0.1
  Using cached lime-0.2.0.1.tar.gz (275 kB)
  Preparing metadata (setup.py) ... done
Collecting recourse==1.0.0
  Using cached recourse-1.0.0-py3-none-any.whl (45 kB)
Collecting torchvision==0.8.1
  Downloading torchvision-0.8.1-cp38-cp38-macosx_10_9_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 7.6 MB/s eta 0:00:00
Collecting h5py==2.10.0
  Downloading h5py-2.10.0-cp38-cp38-macosx_10_9_x86_64.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 9.1 MB/s eta 0:00:00
Collecting carla-recourse
  Using cached carla_recourse-0.0.4-py3-none-any.whl (97 kB)
ERROR: Cannot install carla-recourse==0.0.4 and carla-recourse==0.0.5 because these package versions have conflicting dependencies.

The conflict is caused by:
    carla-recourse 0.0.5 depends on tensorflow==1.14.0
    carla-recourse 0.0.4 depends on tensorflow==1.14.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[pat-alt]

Looks like Python 3.8 requires TensorFlow 2.2 or later (see here).

[JohanvandenHeuvel]

I wasn't aware of that, thanks for bringing that up. At some point we would like to remove the dependence on tensorflow==1.14.0, however it is low priority for now. Is there a particular reason you want to use CARLA with a more recent Python version?

[pat-alt]

No worries, perhaps I can have a look myself. I'm using CARLA for a project that depends on another package which itself needs Python >=3.8, so the two are not compatible at the moment.

[kushan-gunasekera]

For anyone who is looking for a proper solution,

answer:- https://stackoverflow.com/a/72493690/6194097