[bentoml 1.0.0a2] bentoml serve production mode fails

[alexdivet]

Describe the bug I'm testing bentoml 1.0.0a2 following your quickstart but I'm hitting an issue with the production mode (it works fine without --production)

To Reproduce Follow the same steps as in the quickstart and run bentoml serve iris_classifier:latest --production.

That's the definition of the service from the quickstart

import bentoml
import bentoml.sklearn
import numpy as np

from bentoml.io import NumpyNdarray

# Load the runner for the latest ScikitLearn model we just saved
iris_clf_runner = bentoml.sklearn.load_runner("iris_clf:latest")

# Create the iris_classifier service with the ScikitLearn runner
svc = bentoml.Service("iris_classifier", runners=[iris_clf_runner])

# Create API function with pre- and post- processing logic
@svc.api(input=NumpyNdarray(), output=NumpyNdarray())
def predict(input_ndarray: np.ndarray) -> np.ndarray:
    # Define pre-processing logic
    result = iris_clf_runner.run(input_ndarray)
    # Define post-processing logic
    return result

Test a request

curl \
-X POST \
-H "content-type: application/json" \
--data "[5, 4, 3, 2]" \
http://127.0.0.1:5000/predict

Screenshots/Logs

[17:24:57] ERROR    Exception on /predict [POST]                                                                                                                                                                                                                                                                               
                    ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/bentoml/_internal/server/service_app.py:356 in api_func                                                                                                                                                              │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   353 │   │   │   │   │   if isinstance(api.input, Multipart):                                                                                                                                                                                                                                          │
                    │   354 │   │   │   │   │   │   output: t.Any = await run_in_threadpool(api.func, **input_data)                                                                                                                                                                                                           │
                    │   355 │   │   │   │   │   else:                                                                                                                                                                                                                                                                         │
                    │ ❱ 356 │   │   │   │   │   │   output: t.Any = await run_in_threadpool(api.func, input_data)                                                                                                                                                                                                             │
                    │   357 │   │   │   │   response = await api.output.to_http_response(output)                                                                                                                                                                                                                              │
                    │   358 │   │   │   except BentoMLException as e:                                                                                                                                                                                                                                                         │
                    │   359 │   │   │   │   log_exception(request, sys.exc_info())                                                                                                                                                                                                                                            │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/starlette/concurrency.py:39 in run_in_threadpool                                                                                                                                                                     │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   36 │   elif kwargs:  # pragma: no cover                                                                                                                                                                                                                                                               │
                    │   37 │   │   # run_sync doesn't accept 'kwargs', so bind them in here                                                                                                                                                                                                                                   │
                    │   38 │   │   func = functools.partial(func, **kwargs)                                                                                                                                                                                                                                                   │
                    │ ❱ 39 │   return await anyio.to_thread.run_sync(func, *args)                                                                                                                                                                                                                                             │
                    │   40                                                                                                                                                                                                                                                                                                    │
                    │   41                                                                                                                                                                                                                                                                                                    │
                    │   42 class _StopIteration(Exception):                                                                                                                                                                                                                                                                   │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/anyio/to_thread.py:28 in run_sync                                                                                                                                                                                    │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   25 │   :return: an awaitable that yields the return value of the function.                                                                                                                                                                                                                            │
                    │   26 │                                                                                                                                                                                                                                                                                                  │
                    │   27 │   """                                                                                                                                                                                                                                                                                            │
                    │ ❱ 28 │   return await get_asynclib().run_sync_in_worker_thread(func, *args,                                                                                                                                                                                                                             │
                    │      cancellable=cancellable,                                                                                                                                                                                                                                                                           │
                    │   29 │   │   │   │   │   │   │   │   │   │   │   │   │   │     limiter=limiter)                                                                                                                                                                                                                         │
                    │   30                                                                                                                                                                                                                                                                                                    │
                    │   31                                                                                                                                                                                                                                                                                                    │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py:818 in run_sync_in_worker_thread                                                                                                                                                         │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    815 │   │   │   context = copy_context()                                                                                                                                                                                                                                                             │
                    │    816 │   │   │   context.run(sniffio.current_async_library_cvar.set, None)                                                                                                                                                                                                                            │
                    │    817 │   │   │   worker.queue.put_nowait((context, func, args, future))                                                                                                                                                                                                                               │
                    │ ❱  818 │   │   │   return await future                                                                                                                                                                                                                                                                  │
                    │    819                                                                                                                                                                                                                                                                                                  │
                    │    820                                                                                                                                                                                                                                                                                                  │
                    │    821 def run_sync_from_thread(func: Callable[..., T_Retval], *args: object,                                                                                                                                                                                                                           │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py:754 in run                                                                                                                                                                               │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    751 │   │   │   │   │   result = None                                                                                                                                                                                                                                                                │
                    │    752 │   │   │   │   │   exception: Optional[BaseException] = None                                                                                                                                                                                                                                    │
                    │    753 │   │   │   │   │   try:                                                                                                                                                                                                                                                                         │
                    │ ❱  754 │   │   │   │   │   │   result = context.run(func, *args)                                                                                                                                                                                                                                        │
                    │    755 │   │   │   │   │   except BaseException as exc:                                                                                                                                                                                                                                                 │
                    │    756 │   │   │   │   │   │   exception = exc                                                                                                                                                                                                                                                          │
                    │    757                                                                                                                                                                                                                                                                                                  │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/bentoml/bentos/iris_classifier/gr7lcmt5gcrejlg6/src/bento.py:17 in predict                                                                                                                                                                                                               │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   14 @svc.api(input=NumpyNdarray(), output=NumpyNdarray())                                                                                                                                                                                                                                              │
                    │   15 def predict(input_ndarray: np.ndarray) -> np.ndarray:                                                                                                                                                                                                                                              │
                    │   16 │   # Define pre-processing logic                                                                                                                                                                                                                                                                  │
                    │ ❱ 17 │   result = iris_clf_runner.run(input_ndarray)                                                                                                                                                                                                                                                    │
                    │   18 │   # Define post-processing logic                                                                                                                                                                                                                                                                 │
                    │   19 │   return result                                                                                                                                                                                                                                                                                  │
                    │   20                                                                                                                                                                                                                                                                                                    │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/bentoml/_internal/runner/runner.py:141 in run                                                                                                                                                                        │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   138 │   │   return await self._impl.async_run_batch(*args, **kwargs)                                                                                                                                                                                                                                  │
                    │   139 │                                                                                                                                                                                                                                                                                                 │
                    │   140 │   def run(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                                                                                                                                                                                                                                        │
                    │ ❱ 141 │   │   return self._impl.run(*args, **kwargs)                                                                                                                                                                                                                                                    │
                    │   142 │                                                                                                                                                                                                                                                                                                 │
                    │   143 │   def run_batch(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                                                                                                                                                                                                                                  │
                    │   144 │   │   return self._impl.run_batch(*args, **kwargs)                                                                                                                                                                                                                                              │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/bentoml/_internal/runner/remote.py:102 in run                                                                                                                                                                        │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    99 │   def run(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                                                                                                                                                                                                                                        │
                    │   100 │   │   import anyio                                                                                                                                                                                                                                                                              │
                    │   101 │   │                                                                                                                                                                                                                                                                                             │
                    │ ❱ 102 │   │   return anyio.run(self.async_run, *args, **kwargs)                                                                                                                                                                                                                                         │
                    │   103 │                                                                                                                                                                                                                                                                                                 │
                    │   104 │   def run_batch(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                                                                                                                                                                                                                                  │
                    │   105 │   │   import anyio                                                                                                                                                                                                                                                                              │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/anyio/_core/_eventloop.py:56 in run                                                                                                                                                                                  │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    53 │                                                                                                                                                                                                                                                                                                 │
                    │    54 │   try:                                                                                                                                                                                                                                                                                          │
                    │    55 │   │   backend_options = backend_options or {}                                                                                                                                                                                                                                                   │
                    │ ❱  56 │   │   return asynclib.run(func, *args, **backend_options)                                                                                                                                                                                                                                       │
                    │    57 │   finally:                                                                                                                                                                                                                                                                                      │
                    │    58 │   │   if token:                                                                                                                                                                                                                                                                                 │
                    │    59 │   │   │   sniffio.current_async_library_cvar.reset(token)                                                                                                                                                                                                                                       │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py:233 in run                                                                                                                                                                               │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    230 │   │   │   del _task_states[task]                                                                                                                                                                                                                                                               │
                    │    231 │                                                                                                                                                                                                                                                                                                │
                    │    232 │   _maybe_set_event_loop_policy(policy, use_uvloop)                                                                                                                                                                                                                                             │
                    │ ❱  233 │   return native_run(wrapper(), debug=debug)                                                                                                                                                                                                                                                    │
                    │    234                                                                                                                                                                                                                                                                                                  │
                    │    235                                                                                                                                                                                                                                                                                                  │
                    │    236 #                                                                                                                                                                                                                                                                                                │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/lib/python3.8/asyncio/runners.py:44 in run                                                                                                                                                                                                                        │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   41 │   │   events.set_event_loop(loop)                                                                                                                                                                                                                                                                │
                    │   42 │   │   if debug is not None:                                                                                                                                                                                                                                                                      │
                    │   43 │   │   │   loop.set_debug(debug)                                                                                                                                                                                                                                                                  │
                    │ ❱ 44 │   │   return loop.run_until_complete(main)                                                                                                                                                                                                                                                       │
                    │   45 │   finally:                                                                                                                                                                                                                                                                                       │
                    │   46 │   │   try:                                                                                                                                                                                                                                                                                       │
                    │   47 │   │   │   _cancel_all_tasks(loop)                                                                                                                                                                                                                                                                │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/lib/python3.8/asyncio/base_events.py:616 in run_until_complete                                                                                                                                                                                                    │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    613 │   │   if not future.done():                                                                                                                                                                                                                                                                    │
                    │    614 │   │   │   raise RuntimeError('Event loop stopped before Future completed.')                                                                                                                                                                                                                    │
                    │    615 │   │                                                                                                                                                                                                                                                                                            │
                    │ ❱  616 │   │   return future.result()                                                                                                                                                                                                                                                                   │
                    │    617 │                                                                                                                                                                                                                                                                                                │
                    │    618 │   def stop(self):                                                                                                                                                                                                                                                                              │
                    │    619 │   │   """Stop running the event loop.                                                                                                                                                                                                                                                          │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py:228 in wrapper                                                                                                                                                                           │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    225 │   │   │   task.set_name(task_state.name)                                                                                                                                                                                                                                                       │
                    │    226 │   │                                                                                                                                                                                                                                                                                            │
                    │    227 │   │   try:                                                                                                                                                                                                                                                                                     │
                    │ ❱  228 │   │   │   return await func(*args)                                                                                                                                                                                                                                                             │
                    │    229 │   │   finally:                                                                                                                                                                                                                                                                                 │
                    │    230 │   │   │   del _task_states[task]                                                                                                                                                                                                                                                               │
                    │    231                                                                                                                                                                                                                                                                                                  │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/bentoml/_internal/runner/remote.py:93 in async_run                                                                                                                                                                   │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    90 │                                                                                                                                                                                                                                                                                                 │
                    │    91 │   async def async_run(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                                                                                                                                                                                                                            │
                    │    92 │   │   url = "http://127.0.0.1:8000/run"                                                                                                                                                                                                                                                         │
                    │ ❱  93 │   │   return await self._async_req(url, *args, **kwargs)                                                                                                                                                                                                                                        │
                    │    94 │                                                                                                                                                                                                                                                                                                 │
                    │    95 │   async def async_run_batch(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                                                                                                                                                                                                                      │
                    │    96 │   │   url = "http://127.0.0.1:8000/run_batch"                                                                                                                                                                                                                                                   │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/bentoml/_internal/runner/remote.py:81 in _async_req                                                                                                                                                                  │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    78 │   │                                                                                                                                                                                                                                                                                             │
                    │    79 │   │   params = Params(*args, **kwargs).map(AutoContainer.single_to_payload)                                                                                                                                                                                                                     │
                    │    80 │   │   multipart = payload_params_to_multipart(params)                                                                                                                                                                                                                                           │
                    │ ❱  81 │   │   client = self._get_client()                                                                                                                                                                                                                                                               │
                    │    82 │   │   async with client.post(url, data=multipart) as resp:                                                                                                                                                                                                                                      │
                    │    83 │   │   │   body = await resp.read()                                                                                                                                                                                                                                                              │
                    │    84 │   │   │   meta_header = resp.headers[PAYLOAD_META_HEADER]                                                                                                                                                                                                                                       │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/simple_di/__init__.py:124 in _                                                                                                                                                                                       │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   121 │   │   bind = sig.bind_partial(*filtered_args, **filtered_kwargs)                                                                                                                                                                                                                                │
                    │   122 │   │   bind.apply_defaults()                                                                                                                                                                                                                                                                     │
                    │   123 │   │                                                                                                                                                                                                                                                                                             │
                    │ ❱ 124 │   │   return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))                                                                                                                                                                                                                      │
                    │   125 │                                                                                                                                                                                                                                                                                                 │
                    │   126 │   setattr(_, "_is_injected", True)                                                                                                                                                                                                                                                              │
                    │   127 │   return cast(WrappedCallable, _)                                                                                                                                                                                                                                                               │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/bentoml/_internal/runner/remote.py:67 in _get_client                                                                                                                                                                 │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    64 │   │   │   │   DEFAULT_TIMEOUT = aiohttp.ClientTimeout(total=5 * 60)                                                                                                                                                                                                                             │
                    │    65 │   │   │   │   timeout = DEFAULT_TIMEOUT                                                                                                                                                                                                                                                         │
                    │    66 │   │   │   self._client = aiohttp.ClientSession(                                                                                                                                                                                                                                                 │
                    │ ❱  67 │   │   │   │   connector=self._get_conn(),                                                                                                                                                                                                                                                       │
                    │    68 │   │   │   │   auto_decompress=False,                                                                                                                                                                                                                                                            │
                    │    69 │   │   │   │   cookie_jar=jar,                                                                                                                                                                                                                                                                   │
                    │    70 │   │   │   │   connector_owner=False,                                                                                                                                                                                                                                                            │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/simple_di/__init__.py:124 in _                                                                                                                                                                                       │
                    │                                                                                                                                                                                                                                                                                                         │
                    │   121 │   │   bind = sig.bind_partial(*filtered_args, **filtered_kwargs)                                                                                                                                                                                                                                │
                    │   122 │   │   bind.apply_defaults()                                                                                                                                                                                                                                                                     │
                    │   123 │   │                                                                                                                                                                                                                                                                                             │
                    │ ❱ 124 │   │   return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))                                                                                                                                                                                                                      │
                    │   125 │                                                                                                                                                                                                                                                                                                 │
                    │   126 │   setattr(_, "_is_injected", True)                                                                                                                                                                                                                                                              │
                    │   127 │   return cast(WrappedCallable, _)                                                                                                                                                                                                                                                               │
                    │                                                                                                                                                                                                                                                                                                         │
                    │ /Users/e056232/.pyenv/versions/3.8.12/envs/bentoml-env/lib/python3.8/site-packages/bentoml/_internal/runner/remote.py:44 in _get_conn                                                                                                                                                                   │
                    │                                                                                                                                                                                                                                                                                                         │
                    │    41 │   │   │   or self._loop.is_closed()                                                                                                                                                                                                                                                             │
                    │    42 │   │   ):                                                                                                                                                                                                                                                                                        │
                    │    43 │   │   │   self._loop = asyncio.get_event_loop()                                                                                                                                                                                                                                                 │
                    │ ❱  44 │   │   │   self._conn = aiohttp.UnixConnector(path=uds, loop=self._loop)                                                                                                                                                                                                                         │
                    │    45 │   │   return self._conn                                                                                                                                                                                                                                                                         │
                    │    46 │                                                                                                                                                                                                                                                                                                 │
                    │    47 │   @inject                                                                                                                                                                                                                                                                                       │
                    ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
                    TypeError: __init__() got an unexpected keyword argument 'loop'                                                                                                                                                                                                                                            
[17:24:58] INFO     127.0.0.1:51131 - "POST /predict HTTP/1.1" 500 Internal Server Error

Environment: OS: MacOS 11.6 Python Version Python 3.8.12 BentoML Version BentoML-1.0.0a2

[parano]

hi @alexdivet, what is your aiohttp version?

[alexdivet]

my bad, it's aiohttp==4.0.0a1 and not the stable aiohttp==3.8.1

I've just realised I'm using pip install bentoml --pre to install the preview version, but it also installs all pre-release / development versions of dependent libraries. Safer to go with pip install bentoml==1.0.0a2

[parano]

@alexdivet that's indeed an important issue for preview release users, we will fix that in the next preview release to set a max version for those dependencies.