search

PaddlePaddle / PaddleNLP

👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
https://paddlenlp.readthedocs.io
Apache License 2.0
12.09k stars 2.93k forks source link

[Bug]: BiGRU+CRF训练快递序列标注后导出静态模型后部署问题： #5588

Open suntao2015005848 opened 1 year ago

suntao2015005848 commented 1 year ago

软件环境

- paddlepaddle:
- paddlepaddle-gpu: 2.3.2
- paddlenlp: 2.5.2
- ernie-3.0-medium-zh

重复问题

错误描述

1、训练后成功导出静态模型脚本：https://github.com/PaddlePaddle/PaddleNLP/blob/develop/examples/information_extraction/waybill_ie/export_bigru_crf_model.py
2、利用simple_serving服务部署：
https://github.com/PaddlePaddle/PaddleNLP/blob/release/2.5/model_zoo/ernie-3.0/deploy/simple_serving/server_token_cls.py
3、利用脚本服务启动后预测报错：
https://github.com/PaddlePaddle/PaddleNLP/blob/release/2.5/model_zoo/ernie-3.0/deploy/simple_serving/client_token_cls.py

稳定复现步骤 & 代码

[2023-04-10 15:30:53,767] [    INFO] - The predictor id: 0 is selected by running the model.
INFO:     127.0.0.1:55380 - "POST /models/esmod HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 430, in run_asgi
    self.scope, self.receive, self.send
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/fastapi/routing.py", line 238, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/fastapi/routing.py", line 165, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/anyio/to_thread.py", line 32, in run_sync
    func, *args, cancellable=cancellable, limiter=limiter
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlenlp/server/http_router/router.py", line 61, in predict
    result = self._app._model_manager.predict(inference_request.data, inference_request.parameters)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlenlp/server/model_manager.py", line 94, in predict
    model_output = self._model_handler(self._predictor_list[predictor_id], self._tokenizer, data, parameters)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddlenlp/server/handlers/token_model_handler.py", line 97, in process
    predictor._predictor.run()
ValueError: In user code:

    File "export_bigru_crf_model.py", line 60, in <module>
      paddle.jit.save(model, save_path)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py", line 631, in wrapper
      func(layer, path, input_spec, **configs)
    File "<decorator-gen-106>", line 2, in save

    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py", line 861, in save
      inner_input_spec, with_hook=with_hook)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 528, in concrete_program_specify_input_spec
      *desired_input_spec, with_hook=with_hook)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 436, in get_concrete_program
      concrete_program, partial_program_layer = self._program_cache[cache_key]
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 801, in __getitem__
      self._caches[item_id] = self._build_once(item)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 790, in _build_once
      **cache_key.kwargs)
    File "<decorator-gen-104>", line 2, in from_func_spec

    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
      return func(*args, **kwargs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 733, in from_func_spec
      outputs = static_func(*inputs)
    File "/tmp/tmpekqd9lql.py", line 32, in forward
      __return_value_0))
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
      out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
      return true_fn(*true_args) if pred else false_fn(*false_args)
    File "/home/aistudio/train_crf/model.py", line 51, in forward
      _, prediction = self.viterbi_decoder(emission, lengths)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/text/viterbi_decode.py", line 136, in forward
      self.include_bos_eos_tag, self.name)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/text/viterbi_decode.py", line 87, in viterbi_decode
      attrs=attrs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 44, in append_op
      return self.main_program.current_block().append_op(*args, **kwargs)
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3621, in append_op
      attrs=kwargs.get("attrs", None))
    File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2635, in __init__
      for frame in traceback.extract_stack():

    InvalidArgumentError: The rank of Length in ViterbiDecode must be 1. But received Length's rank is 2.
      [Hint: Expected length_dims.size() == 1, but received length_dims.size():2 != 1:1.] (at /paddle/paddle/phi/infermeta/ternary.cc:681)
      [operator < viterbi_decode > error]
w5688414 commented 5 months ago

可以升级paddle版本后试试，另外可以提供最小复现代码，方便我们快速定位