search

SeldonIO / MLServer

An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
https://mlserver.readthedocs.io/en/latest/
Apache License 2.0
685 stars 177 forks source link

Unable to use `tritonclient.grpc` #1730

Open tokoko opened 4 months ago

tokoko commented 4 months ago

I'm trying to use tritonclient.grpc inside an mlserver model, but looks like they don't really go well with one another. python -c "import mlserver;import tritonclient.grpc" results in a Couldn't build proto file into descriptor pool: duplicate symbol 'inference.ServerLiveRequest' . I'm not a protobuf expert, but seems like both mlserver and tritonclient have protobuf-generated classes from identical (or maybe just similar?) OIP protos and they conflict with one another. Is there any way to work around this? For example, is it realistic for mlserver to use proto classes from tritonclient directly since it's a required dependency anyway?

I think another simpler solution might be changing package declaration in dataplane.proto from inference to something like inference_mlserver.

tokoko commented 4 months ago

As far as I could tell, the one thing that package name change would affect is Prometheus metrics, package name is part of a key in metrics, I think.

lc525 commented 4 months ago

Both mlserver and tritonclient.grpc declare a submodule called inference. This leads to the duplicate symbol, which should be solvable through typical means (more specific imports, import ... as, etc).

tokoko commented 4 months ago

Hey, thanks for the response. Unfortunately, I don't think it's that simple. The error happens even with top-level imports, I'm not trying to import anything message related specifically. For example running python -c "import tritonclient.grpc; import mlserver" leads to an error with the following stack trace:

File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/mlserver/__init__.py", line 2, in <module>
    from .server import MLServer
  File "/usr/local/lib/python3.9/site-packages/mlserver/server.py", line 17, in <module>
    from .grpc import GRPCServer
  File "/usr/local/lib/python3.9/site-packages/mlserver/grpc/__init__.py", line 1, in <module>
    from .server import GRPCServer
  File "/usr/local/lib/python3.9/site-packages/mlserver/grpc/server.py", line 9, in <module>
    from .servicers import InferenceServicer
  File "/usr/local/lib/python3.9/site-packages/mlserver/grpc/servicers.py", line 3, in <module>
    from . import dataplane_pb2 as pb
  File "/usr/local/lib/python3.9/site-packages/mlserver/grpc/dataplane_pb2.py", line 16, in <module>
    DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
TypeError: Couldn't build proto file into descriptor pool: duplicate symbol 'inference.ServerLiveRequest'

I'm not exactly sure what this method (_descriptor_pool.Default().AddSerializedFile) does, but looks like it's registering symbols in some global namespace, which can't really be remedied with a simple import ... as ... statement, symbols are hardcoded in both libraries.

tokoko commented 3 months ago

@lc525 hey, have you had an opportunity to look into this?