paddaleserving预测模型terminate called after throwing an instance of 'phi::enforce::EnforceNotMet'

bug描述 Describe the Bug

λ 2bb8d1c612d1 /paddle/PaddleDetection/output_inference/faster_rcnn_r50_1x_coco python -m paddle_serving_server.serve --model serving_server --port 9393 /usr/lib/python3.7/runpy.py:125: RuntimeWarning: 'paddle_serving_server.serve' found in sys.modules after import of package 'paddle_serving_server', but prior to execution of 'paddle_serving_server.serve'; this may result in unpredictable behaviour warn(RuntimeWarning(msg)) Going to Run Comand /usr/local/lib/python3.7/dist-packages/paddle_serving_server/serving-gpu-112-0.9.0/serving -enable_model_toolkit -inferservice_path workdir_9393 -inferservice_file infer_service.prototxt -max_concurrency 0 -num_threads 4 -port 9393 -precision fp32 -use_calib=False -reload_interval_s 10 -resource_path workdir_9393 -resource_file resource.prototxt -workflow_path workdir_9393 -workflow_file workflow.prototxt -bthread_concurrency 4 -max_body_size 536870912 I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralDetectionOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralDistKVInferOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralDistKVQuantInferOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralFeatureExtractOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralInferOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralPicodetOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralReaderOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralRecOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralRemoteOp I0100 00:00:00.000000 1624 op_repository.h:68] RAW: Succ regist op: GeneralResponseOp I0100 00:00:00.000000 1624 service_manager.h:79] RAW: Service[LoadGeneralModelService] insert successfully! I0100 00:00:00.000000 1624 load_general_model_service.pb.h:333] RAW: Success regist service[LoadGeneralModelService][PN5baidu14paddle_serving9predictor26load_general_model_service27LoadGeneralModelServiceImplE] I0100 00:00:00.000000 1624 service_manager.h:79] RAW: Service[GeneralModelService] insert successfully! I0100 00:00:00.000000 1624 general_model_service.pb.h:1650] RAW: Success regist service[GeneralModelService][PN5baidu14paddle_serving9predictor13general_model23GeneralModelServiceImplE] I0100 00:00:00.000000 1624 factory.h:155] RAW: Succ insert one factory, tag: PADDLE_INFER, base type N5baidu14paddle_serving9predictor11InferEngineE W0100 00:00:00.000000 1624 paddle_engine.cpp:34] RAW: Succ regist factory: ::baidu::paddle_serving::predictor::FluidInferEngine->::baidu::paddle_serving::predictor::InferEngine, tag: PADDLE_INFER in macro! --- Running analysis [ir_graph_build_pass] --- Running analysis [ir_graph_clean_pass] --- Running analysis [ir_analysis_pass] --- Running analysis [ir_params_sync_among_devices_pass] --- Running analysis [adjust_cudnn_workspace_size_pass] --- Running analysis [inference_op_replace_pass] --- Running analysis [memory_optimize_pass] --- Running analysis [ir_graph_to_program_pass] C++ Serving service started successfully! terminate called after throwing an instance of 'phi::enforce::EnforceNotMet' what():

Compile Traceback (most recent call last): File "tools/export_model.py", line 110, in main() File "tools/export_model.py", line 106, in main run(FLAGS, cfg) File "tools/export_model.py", line 73, in run trainer.export(FLAGS.output_dir) File "/paddle/PaddleDetection/ppdet/engine/trainer.py", line 1171, in export save_dir) File "/paddle/PaddleDetection/ppdet/engine/trainer.py", line 1130, in _get_infer_cfg_and_input_spec input_spec, static_model.forward.main_program, File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/program_translator.py", line 894, in main_program concrete_program = self.concrete_program File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/program_translator.py", line 663, in concrete_program return self.concrete_program_specify_input_spec(input_spec=None) File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/program_translator.py", line 713, in concrete_program_specify_input_spec is_prim_infer=is_prim_infer, File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/program_translator.py", line 590, in get_concrete_program cache_key File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/program_translator.py", line 1249, in getitem self._caches[item_id] = self._build_once(item) File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/program_translator.py", line 1198, in _build_once cache_key.kwargs, File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), *kw) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl return wrapped_func(args, kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 75, in impl return func(*args, kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/program_translator.py", line 1063, in from_func_spec outputs = static_func(inputs) File "/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 59, in forward if self.training: File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 353, in convert_ifelse pred, true_fn, false_fn, get_args, set_args, return_name_ids File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 429, in _run_py_ifelse py_outs = true_fn() if pred else false_fn() File "/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 69, in forward for inp in inputs_list: File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 121, in convert_while_loop _run_py_while(cond, body, getter, setter) File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 185, in _run_py_while body() File "/paddle/PaddleDetection/ppdet/modeling/architectures/meta_arch.py", line 76, in forward outs.append(self.get_pred()) File "/paddle/PaddleDetection/ppdet/modeling/architectures/faster_rcnn.py", line 121, in get_pred if self.use_extra_data: File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 353, in convert_ifelse pred, true_fn, false_fn, get_args, set_args, return_name_ids File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 429, in _run_py_ifelse py_outs = true_fn() if pred else false_fn() File "/paddle/PaddleDetection/ppdet/modeling/architectures/faster_rcnn.py", line 125, in get_pred bbox_pred, bbox_num = self._forward() File "/paddle/PaddleDetection/ppdet/modeling/architectures/faster_rcnn.py", line 79, in _forward if self.training: File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 353, in convert_ifelse pred, true_fn, false_fn, get_args, set_args, return_name_ids File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 429, in _run_py_ifelse py_outs = true_fn() if pred else false_fn() File "/paddle/PaddleDetection/ppdet/modeling/architectures/faster_rcnn.py", line 85, in _forward rois, roisnum, = self.rpn_head(body_feats, self.inputs) File "/usr/local/lib/python3.7/dist-packages/paddle/nn/layer/layers.py", line 1256, in call return self._dygraph_call_func(inputs, kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/nn/layer/layers.py", line 1235, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "/paddle/PaddleDetection/ppdet/modeling/proposal_generator/rpn_head.py", line 141, in forward rois, rois_num = self._gen_proposal(scores, deltas, anchors, inputs) File "/paddle/PaddleDetection/ppdet/modeling/proposal_generator/rpn_head.py", line 161, in _gen_proposal if self.export_onnx: File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 353, in convert_ifelse pred, true_fn, false_fn, get_args, set_args, return_name_ids File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 429, in _run_py_ifelse py_outs = true_fn() if pred else false_fn() File "/paddle/PaddleDetection/ppdet/modeling/proposal_generator/rpn_head.py", line 199, in _gen_proposal for i in range(batch_size): File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 118, in convert_while_loop cond, body, getter, setter, return_name_ids, push_pop_names File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 171, in _run_paddle_while loop_vars = control_flow.while_loop(new_cond_fn, new_body_fn, loop_vars) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layers/control_flow.py", line 1045, in while_loop pre_cond = cond(loop_vars) File "/usr/local/lib/python3.7/dist-packages/paddle/jit/dy2static/convert_operators.py", line 160, in new_cond_fn return cond() File "/root/.cache/paddle/to_static_tmp/1039/RPNHeadgen_proposaldgklyz0z.py", line 114, in for_loop_condition_9 return _jst.Ld(__for_loop_var_index_9) < _jst.Ld(for_loop_var_len_9) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layers/math_op_patch.py", line 449, in impl attrs={'axis': axis}, File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 4019, in append_op attrs=kwargs.get("attrs", None), File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 2781, in init for frame in traceback.extract_stack():

C++ Traceback (most recent call last):

Error Message Summary:

InvalidArgumentError: Axis should be less than 1, but received axis is 1. [Hint: Expected axis < max_dim, but received axis:1 >= max_dim:1.] (at /paddle/paddle/phi/kernels/funcs/common_shape.h:53) [operator < less_than > error] Aborted

其他补充信息 Additional Supplementary Information

No response

PaddlePaddle / Paddle