bug: HTTPClient dies with segfault

H-Simpson123 commented 1 year ago

Describe the bug

I've tried to execute your toy example from the docs with different host and client machines:

import openllm
client = openllm.client.HTTPClient('http://some_other_machine.local:3000')
print(client.framework, client.model_id, client.configuration)
print('Query:')
print(client.query('Explain to me the difference between "further" and "farther"'))

Output:
pt llama {'use_llama2_prompt': False, 'generation_config': {'max_new_tokens': 128, 'min_length': 0, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'use_cache': True, 'temperature': 0.6, 'top_k': 12, 'top_p': 0.9, 'typical_p': 1.0, 'epsilon_cutoff': 0.0, 'eta_cutoff': 0.0, 'diversity_penalty': 0.0, 'repetition_penalty': 1.0, 'encoder_repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'renormalize_logits': False, 'remove_invalid_values': False, 'num_return_sequences': 1, 'output_attentions': False, 'output_hidden_states': False, 'output_scores': False, 'encoder_no_repeat_ngram_size': 0}, 'n': 1, 'best_of': 1, 'presence_penalty': 0.5, 'frequency_penalty': 0.0, 'use_beam_search': False, 'ignore_eos': False}
Query:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

On the host logs I can see the requests the /v1/metadata endpoint

If i execute the same example on the host machine (with localhost instead) it works without problems

To reproduce

No response

Logs

No response

Environment

accelerate 0.21.0 aiohttp 3.8.5 aiosignal 1.3.1 anyio 3.7.1 appdirs 1.4.4 asgiref 3.7.2 async-timeout 4.0.3 attrs 23.1.0 bentoml 1.1.1 bitsandbytes 0.41.1 blessed 1.20.0 build 0.10.0 cattrs 23.1.2 certifi 2023.7.22 charset-normalizer 3.2.0 circus 0.18.0 click 8.1.7 click-option-group 0.5.6 cloudpickle 2.2.1 cmake 3.27.2 coloredlogs 15.0.1 contextlib2 21.6.0 cuda-python 12.2.0 Cython 3.0.0 datasets 2.14.4 deepmerge 1.1.0 Deprecated 1.2.14 dill 0.3.7 exceptiongroup 1.1.3 fairscale 0.4.13 fastapi 0.101.1 fastcore 1.5.29 filelock 3.12.2 filetype 1.2.0 frozenlist 1.4.0 fs 2.4.16 fsspec 2023.6.0 ghapi 1.0.4 gpustat 1.1 grpcio 1.57.0 grpcio-health-checking 1.57.0 h11 0.14.0 httpcore 0.17.3 httpx 0.24.1 huggingface-hub 0.16.4 humanfriendly 10.0 idna 3.4 importlib-metadata 6.0.1 importlib-resources 6.0.1 inflection 0.5.1 Jinja2 3.1.2 jsonschema 4.19.0 jsonschema-specifications 2023.7.1 lit 16.0.6 markdown-it-py 3.0.0 MarkupSafe 2.1.3 mdurl 0.1.2 mpmath 1.3.0 msgpack 1.0.5 multidict 6.0.4 multiprocess 0.70.15 mypy-extensions 1.0.0 networkx 3.1 ninja 1.11.1 numpy 1.24.4 nvidia-cublas-cu11 11.10.3.66 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cudnn-cu11 8.5.0.96 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.2.10.91 nvidia-cusolver-cu11 11.4.0.1 nvidia-cusparse-cu11 11.7.4.91 nvidia-ml-py 12.535.77 nvidia-nccl-cu11 2.14.3 nvidia-nvtx-cu11 11.7.91 openllm 0.2.26 opentelemetry-api 1.18.0 opentelemetry-instrumentation 0.39b0 opentelemetry-instrumentation-aiohttp-client 0.39b0 opentelemetry-instrumentation-asgi 0.39b0 opentelemetry-instrumentation-grpc 0.39b0 opentelemetry-sdk 1.18.0 opentelemetry-semantic-conventions 0.39b0 opentelemetry-util-http 0.39b0 optimum 1.11.2 orjson 3.9.5 packaging 23.1 pandas 2.0.3 pathspec 0.11.2 Pillow 10.0.0 pip 23.2.1 pip-requirements-parser 32.0.1 pip-tools 7.3.0 pkg_resources 0.0.0 pkgutil_resolve_name 1.3.10 prometheus-client 0.17.1 protobuf 4.24.0 psutil 5.9.5 pyarrow 12.0.1 pydantic 1.10.12 Pygments 2.16.1 pynvml 11.5.0 pyparsing 3.1.1 pyproject_hooks 1.0.0 python-dateutil 2.8.2 python-json-logger 2.0.7 python-multipart 0.0.6 pytz 2023.3 PyYAML 6.0.1 pyzmq 25.1.1 ray 2.6.3 referencing 0.30.2 regex 2023.8.8 requests 2.31.0 rich 13.5.2 rpds-py 0.9.2 safetensors 0.3.2 schema 0.7.5 scipy 1.10.1 sentencepiece 0.1.99 setuptools 44.0.0 simple-di 0.1.5 six 1.16.0 sniffio 1.3.0 starlette 0.27.0 sympy 1.12 tabulate 0.9.0 tokenizers 0.13.3 tomli 2.0.1 torch 2.0.1 tornado 6.3.3 tqdm 4.66.1 transformers 4.31.0 triton 2.0.0 typing_extensions 4.7.1 tzdata 2023.3 urllib3 2.0.4 uvicorn 0.23.2 vllm 0.1.3 watchfiles 0.19.0 wcwidth 0.2.6 wheel 0.34.2 wrapt 1.15.0 xformers 0.0.21 xxhash 3.3.0 yarl 1.9.2 zipp 3.16.2

System information (Optional)

No response

aarnphm commented 1 year ago

What version of client on your local machine?

H-Simpson123 commented 1 year ago

Host:

Package                                      Version
-------------------------------------------- ----------
accelerate                                   0.21.0
aiohttp                                      3.8.5
aiosignal                                    1.3.1
anyio                                        3.7.1
appdirs                                      1.4.4
asgiref                                      3.7.2
async-timeout                                4.0.3
attrs                                        23.1.0
bentoml                                      1.1.1
bitsandbytes                                 0.41.1
blessed                                      1.20.0
build                                        0.10.0
cattrs                                       23.1.2
certifi                                      2023.7.22
charset-normalizer                           3.2.0
circus                                       0.18.0
click                                        8.1.7
click-option-group                           0.5.6
cloudpickle                                  2.2.1
cmake                                        3.27.2
coloredlogs                                  15.0.1
contextlib2                                  21.6.0
cuda-python                                  12.2.0
Cython                                       3.0.0
datasets                                     2.14.4
deepmerge                                    1.1.0
Deprecated                                   1.2.14
dill                                         0.3.7
exceptiongroup                               1.1.3
fairscale                                    0.4.13
fastapi                                      0.101.1
fastcore                                     1.5.29
filelock                                     3.12.2
filetype                                     1.2.0
frozenlist                                   1.4.0
fs                                           2.4.16
fsspec                                       2023.6.0
ghapi                                        1.0.4
gpustat                                      1.1
grpcio                                       1.57.0
grpcio-health-checking                       1.57.0
h11                                          0.14.0
httpcore                                     0.17.3
httpx                                        0.24.1
huggingface-hub                              0.16.4
humanfriendly                                10.0
idna                                         3.4
importlib-metadata                           6.0.1
importlib-resources                          6.0.1
inflection                                   0.5.1
Jinja2                                       3.1.2
jsonschema                                   4.19.0
jsonschema-specifications                    2023.7.1
lit                                          16.0.6
markdown-it-py                               3.0.0
MarkupSafe                                   2.1.3
mdurl                                        0.1.2
mpmath                                       1.3.0
msgpack                                      1.0.5
multidict                                    6.0.4
multiprocess                                 0.70.15
mypy-extensions                              1.0.0
networkx                                     3.1
ninja                                        1.11.1
numpy                                        1.24.4
nvidia-cublas-cu11                           11.10.3.66
nvidia-cuda-cupti-cu11                       11.7.101
nvidia-cuda-nvrtc-cu11                       11.7.99
nvidia-cuda-runtime-cu11                     11.7.99
nvidia-cudnn-cu11                            8.5.0.96
nvidia-cufft-cu11                            10.9.0.58
nvidia-curand-cu11                           10.2.10.91
nvidia-cusolver-cu11                         11.4.0.1
nvidia-cusparse-cu11                         11.7.4.91
nvidia-ml-py                                 12.535.77
nvidia-nccl-cu11                             2.14.3
nvidia-nvtx-cu11                             11.7.91
openllm                                      0.2.26
opentelemetry-api                            1.18.0
opentelemetry-instrumentation                0.39b0
opentelemetry-instrumentation-aiohttp-client 0.39b0
opentelemetry-instrumentation-asgi           0.39b0
opentelemetry-instrumentation-grpc           0.39b0
opentelemetry-sdk                            1.18.0
opentelemetry-semantic-conventions           0.39b0
opentelemetry-util-http                      0.39b0
optimum                                      1.11.2
orjson                                       3.9.5
packaging                                    23.1
pandas                                       2.0.3
pathspec                                     0.11.2
Pillow                                       10.0.0
pip                                          23.2.1
pip-requirements-parser                      32.0.1
pip-tools                                    7.3.0
pkg_resources                                0.0.0
pkgutil_resolve_name                         1.3.10
prometheus-client                            0.17.1
protobuf                                     4.24.0
psutil                                       5.9.5
pyarrow                                      12.0.1
pydantic                                     1.10.12
Pygments                                     2.16.1
pynvml                                       11.5.0
pyparsing                                    3.1.1
pyproject_hooks                              1.0.0
python-dateutil                              2.8.2
python-json-logger                           2.0.7
python-multipart                             0.0.6
pytz                                         2023.3
PyYAML                                       6.0.1
pyzmq                                        25.1.1
ray                                          2.6.3
referencing                                  0.30.2
regex                                        2023.8.8
requests                                     2.31.0
rich                                         13.5.2
rpds-py                                      0.9.2
safetensors                                  0.3.2
schema                                       0.7.5
scipy                                        1.10.1
sentencepiece                                0.1.99
setuptools                                   44.0.0
simple-di                                    0.1.5
six                                          1.16.0
sniffio                                      1.3.0
starlette                                    0.27.0
sympy                                        1.12
tabulate                                     0.9.0
tokenizers                                   0.13.3
tomli                                        2.0.1
torch                                        2.0.1
tornado                                      6.3.3
tqdm                                         4.66.1
transformers                                 4.31.0
triton                                       2.0.0
typing_extensions                            4.7.1
tzdata                                       2023.3
urllib3                                      2.0.4
uvicorn                                      0.23.2
vllm                                         0.1.3
watchfiles                                   0.19.0
wcwidth                                      0.2.6
wheel                                        0.34.2
wrapt                                        1.15.0
xformers                                     0.0.21
xxhash                                       3.3.0
yarl                                         1.9.2
zipp                                         3.16.2

Client:


Package                                      Version
-------------------------------------------- ----------
accelerate                                   0.21.0
aiohttp                                      3.8.5
aiosignal                                    1.3.1
anyio                                        3.7.1
appdirs                                      1.4.4
asgiref                                      3.7.2
async-timeout                                4.0.3
atlassian-python-api                         3.41.0
attrs                                        23.1.0
beautifulsoup4                               4.12.2
bentoml                                      1.1.1
bitsandbytes                                 0.41.1
blinker                                      1.6.2
build                                        0.10.0
cattrs                                       23.1.2
certifi                                      2023.7.22
charset-normalizer                           3.2.0
circus                                       0.18.0
click                                        8.1.7
click-option-group                           0.5.6
cloudpickle                                  2.2.1
cmake                                        3.27.2
coloredlogs                                  15.0.1
contextlib2                                  21.6.0
cuda-python                                  12.2.0
Cython                                       3.0.0
dataclasses-json                             0.5.14
datasets                                     2.14.4
deepmerge                                    1.1.0
Deprecated                                   1.2.14
dill                                         0.3.7
exceptiongroup                               1.1.3
faiss-gpu                                    1.7.2
fastcore                                     1.5.29
filelock                                     3.12.2
filetype                                     1.2.0
Flask                                        2.3.2
frozenlist                                   1.4.0
fs                                           2.4.16
fsspec                                       2023.6.0
ghapi                                        1.0.4
greenlet                                     2.0.2
grpcio                                       1.57.0
grpcio-health-checking                       1.57.0
h11                                          0.14.0
httpcore                                     0.17.3
httpx                                        0.24.1
huggingface-hub                              0.16.4
humanfriendly                                10.0
idna                                         3.4
importlib-metadata                           6.0.1
inflection                                   0.5.1
itsdangerous                                 2.1.2
Jinja2                                       3.1.2
langchain                                    0.0.268
langsmith                                    0.0.24
lit                                          16.0.6
lxml                                         4.9.3
markdown-it-py                               3.0.0
MarkupSafe                                   2.1.3
marshmallow                                  3.20.1
mdurl                                        0.1.2
mpmath                                       1.3.0
multidict                                    6.0.4
multiprocess                                 0.70.15
mypy-extensions                              1.0.0
networkx                                     3.1
numexpr                                      2.8.5
numpy                                        1.24.4
nvidia-cublas-cu11                           11.10.3.66
nvidia-cuda-cupti-cu11                       11.7.101
nvidia-cuda-nvrtc-cu11                       11.7.99
nvidia-cuda-runtime-cu11                     11.7.99
nvidia-cudnn-cu11                            8.5.0.96
nvidia-cufft-cu11                            10.9.0.58
nvidia-curand-cu11                           10.2.10.91
nvidia-cusolver-cu11                         11.4.0.1
nvidia-cusparse-cu11                         11.7.4.91
nvidia-nccl-cu11                             2.14.3
nvidia-nvtx-cu11                             11.7.91
oauthlib                                     3.2.2
openllm                                      0.2.26
opentelemetry-api                            1.18.0
opentelemetry-instrumentation                0.39b0
opentelemetry-instrumentation-aiohttp-client 0.39b0
opentelemetry-instrumentation-asgi           0.39b0
opentelemetry-instrumentation-grpc           0.39b0
opentelemetry-sdk                            1.18.0
opentelemetry-semantic-conventions           0.39b0
opentelemetry-util-http                      0.39b0
optimum                                      1.11.2
orjson                                       3.9.5
packaging                                    23.1
pandas                                       2.0.3
pathspec                                     0.11.2
Pillow                                       10.0.0
pip                                          22.3.1
pip-requirements-parser                      32.0.1
pip-tools                                    7.3.0
prometheus-client                            0.17.1
protobuf                                     4.24.0
psutil                                       5.9.5
pyarrow                                      12.0.1
pydantic                                     1.10.12
Pygments                                     2.16.1
pynvml                                       11.5.0
pyparsing                                    3.1.1
pyproject_hooks                              1.0.0
python-dateutil                              2.8.2
python-json-logger                           2.0.7
python-multipart                             0.0.6
pytz                                         2023.3
PyYAML                                       6.0.1
pyzmq                                        25.1.1
regex                                        2023.8.8
requests                                     2.31.0
requests-oauthlib                            1.3.1
rich                                         13.5.2
safetensors                                  0.3.2
schema                                       0.7.5
sentencepiece                                0.1.99
setuptools                                   65.5.1
simple-di                                    0.1.5
six                                          1.16.0
sniffio                                      1.3.0
soupsieve                                    2.4.1
SQLAlchemy                                   2.0.20
starlette                                    0.28.0
sympy                                        1.12
tabulate                                     0.9.0
tenacity                                     8.2.3
tokenizers                                   0.13.3
tomli                                        2.0.1
torch                                        2.0.1
tornado                                      6.3.3
tqdm                                         4.66.1
transformers                                 4.31.0
triton                                       2.0.0
typing_extensions                            4.7.1
typing-inspect                               0.9.0
tzdata                                       2023.3
urllib3                                      2.0.4
uvicorn                                      0.23.2
watchfiles                                   0.19.0
wcwidth                                      0.2.6
Werkzeug                                     2.3.7
wheel                                        0.38.4
wrapt                                        1.15.0
xxhash                                       3.3.0
yarl                                         1.9.2
zipp                                         3.16.2

H-Simpson123 commented 1 year ago

gdb tells that the segfault happens in PyDict_SetItem ()

aarnphm commented 1 year ago

Hmm can you try again to see you still run into this issue with 0.3.0?

aarnphm commented 1 year ago

This should now be fixed with 0.3.2. I have disabled compiled wheels for now.

bentoml / OpenLLM