diambra / arena

DIAMBRA Arena: a New Reinforcement Learning Platform for Research and Experimentation
https://docs.diambra.ai
Other
313 stars 22 forks source link

Use latest compatible grpcio-tools version and unlock support for Python 3.11 #57

Open discordianfish opened 1 year ago

discordianfish commented 1 year ago

Currently blocked to 1.48.2 because of TF 2.x limitation of protobuf < 3.20 (and TF is currently used in Ray RL Lib). This also limit python version to 3.11 as it requires a newer grpcio version (above 1.48)

Files affected:

Other relevant info

Precondition on protobuf changed between: grpcio-tools 1.48.2 https://github.com/grpc/grpc/blob/5e97a6003d0e44bdf3c4234d37b9509d02bf55ea/tools/distrib/python/grpcio_tools/setup.py#L293 and 1.49: https://github.com/grpc/grpc/blob/8f8edfd04b46ee67f90454b3f6a70aa58ff82c2d/tools/distrib/python/grpcio_tools/setup.py#L293

Compatibility problem of protobuf https://stackoverflow.com/questions/71759248/importerror-cannot-import-name-builder-from-google-protobuf-internal

discordianfish commented 1 year ago

To sum this up:

Therefor we need to:

  1. Generate diambra-arena-engine using a grpcio-tools version that generate code for a version matching protobuf (<3.20,>=3.9.2), e.g 1.48.2
  2. Use that version's dependency (protobuf (<4.0dev,>=3.12.0) in our setup.py

That will allow users to install ray-rllib where pip dep resolver should install protobuf <3.20,>=3.9.2 while allowing users not depending on ray/tf2 to also install <=3.12.0 and >=3.20.