search

triton-inference-server / fastertransformer_backend

BSD 3-Clause "New" or "Revised" License
411 stars 133 forks source link

Streaming throwing queue.get() error #44

Open rtalaricw opened 2 years ago

rtalaricw commented 2 years ago

Description

Dockerfile: faster_transformer(v1.2)
Model: GPT-J

Reproduced Steps

The streaming example in issue_requests.py throws the following error when passing in a request:

Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rtalari/coreweave/kubernetes-cloud/online-inference/fastertransformer/client/example_http.py", line 94, in stream_consumer
    result = queue.get()
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
TypeError: __init__() missing 1 required positional argument: 'msg'

A simple call to a sample request is passed which works with HTTP with output for "She is a good girl":

Ouput: She is a good girl. She's a good girl. She's a good girl...'

'But you were in bed when we came in,' she says, and I
byshiue commented 2 years ago

I don't understand your question clearly, but the problem may be because issue_request.py needs to receive a request like requests/sample_request.json.

If this is not your problem, please provide more information and details. Please provide your end-to-end workflow step by step to help us understand your questions, thank you.

shanekong commented 1 year ago

i meet the same promblem, by running python3 tools/issue_request.py tools/requests/sample_request_stream.json

Process Process-1: Traceback (most recent call last): File "/data/kongxiangxing/env/miniconda3/envs/python37/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/data/kongxiangxing/env/miniconda3/envs/python37/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "tools/issue_request.py", line 110, in stream_consumer result = queue.get() File "/data/kongxiangxing/env/miniconda3/envs/python37/lib/python3.7/multiprocessing/queues.py", line 113, in get return _ForkingPickler.loads(res) TypeError: init() missing 1 required positional argument: 'msg'

can anyone give any suggestions?