wangxinxin08
commented
3 years ago 请问是在导出模型时报错吗？看你的报错信息都是报在导出模型时报错
Open hymanzhu1983 opened 3 years ago
wangxinxin08
commented
3 years ago 请问是在导出模型时报错吗？看你的报错信息都是报在导出模型时报错
hymanzhu1983
commented
3 years ago 不是的，导出模型没有报错，只是有warning，是运行的时候出错的。导出模型的log如下： python tools/export_serving_model.py -c configs/mask_rcnn_r50_vd_fpn_2x_bill.yml -o use_gpu=true weights=/home/sy/work/ocr_car/PaddleDetection-release-2.0-beta/output/mask_rcnn_r50_vd_fpn_2x_bill/model_final --output_dir=mask_test_model
/home/sy/.conda/envs/solov2-paddle-2.0rc1/lib/python3.7/site-packages/paddle/fluid/layers/math_op_patch.py:297: UserWarning: /home/sy/work/ocr_car/PaddleDetection-release-2.0-beta/ppdet/modeling/roi_heads/bbox_head.py:315 The behavior of expression A / B has been unified with elementwise_div(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_div(X, Y, axis=0) instead of A / B. This transitional warning will be dropped in the future. op_type, op_type, EXPRESSION_MAP[method_name])) /home/sy/.conda/envs/solov2-paddle-2.0rc1/lib/python3.7/site-packages/paddle/fluid/layers/math_op_patch.py:297: UserWarning: /home/sy/work/ocr_car/PaddleDetection-release-2.0-beta/ppdet/modeling/architectures/mask_rcnn.py:253 The behavior of expression A B has been unified with elementwise_mul(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_mul(X, Y, axis=0) instead of A B. This transitional warning will be dropped in the future. op_type, op_type, EXPRESSION_MAP[method_name])) 2021-03-19 14:06:24,105-INFO: save_inference_model pruned unused feed variables im_id 2021-03-19 14:06:24,105-INFO: Export serving model to mask_test_model, client side: mask_test_model/serving_client, server side: mask_test_model/serving_server. input: ['image', 'im_info', 'im_shape'], output: ['multiclass_nms_0.tmp_0', 'mask_pred']... 2021-03-19 14:06:24,420-INFO: Not found annotation file annotations/instance_val2017.json, load coco17 categories. 2021-03-19 14:06:24,423-INFO: Export inference config file to mask_test_model/mask_rcnn_r50_vd_fpn_2x_bill/infer_cfg.yml
liuhuiCNN
commented
3 years ago 请问使用的是静态图还是动态图？ 建议您尝试一下 paddleserving 0.5 版本
hymanzhu1983
commented
3 years ago 在paddleserving0.4环境下调用tools/export_serving_model.py，生成的模型有很多文件，诸如bbox_pred_b/fpn_inner_res2_sum_lateral_w/fpn_inner_res3_sum_lateral_b之类，升级paddleserving 0.5后，调用tools/export_serving_model.py，生成的模型是params 和model两个文件。请问模型的使用方式有什么不同吗？
我跑原来的程序，报错如下：
python web_service.py /home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle_serving_server_gpu/pipeline/pipeline_server.py:339: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
conf = yaml.load(f.read())
/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:26: DeprecationWarning: np.int is a deprecated alias for the builtin int. To silence this warning, use int by itself. Doing this will not modify any behavior and is safe. When replacing np.int, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
def convert_to_list(value, n, name, dtype=np.int):
W0322 16:15:37.033921 7428 analysis_predictor.cc:1145] Deprecated. Please use CreatePredictor instead.
W0322 16:15:38.345609 7428 analysis_predictor.cc:677] The one-time configuration of analysis predictor failed, which may be due to native predictor called first and its configurations taken effect.
I0322 16:15:38.345649 7428 analysis_predictor.cc:155] Profiler is deactivated, and no profiling report will be generated.
I0322 16:15:38.357976 7428 analysis_predictor.cc:574] ir_optim is turned off, no IR pass will be executed
--- Running analysis [ir_graph_build_pass]
Traceback (most recent call last):
File "web_service.py", line 639, in
hymanzhu1983
commented
3 years ago 请问paddleserving 0.5和paddledetection 2.0-rc是兼容的吗？
hymanzhu1983
commented
3 years ago 用官方的Mask_rcnn模型也存在同样的问题。复现： 1.用serving方式部署这个https://paddlemodels.bj.bcebos.com/object_detection/mask_rcnn_r50_vd_fpn_2x.tar模型 2.准备十张图片，其中8张人物图片，2张全白色图片，发送请求到serving服务器，服务器报如下错误： ERROR 2021-03-23 15:33:32,417 [operator.py:769] (data_id=4 log_id=4) [bill_det|0] Failed to process(batch: [4]): In user code:
File "tools/export_serving_model.py", line 107, in <module>
main()
File "tools/export_serving_model.py", line 93, in main
save_serving_model(FLAGS, exe, feed_vars, test_fetches, infer_prog)
File "tools/export_serving_model.py", line 63, in save_serving_model
main_program=infer_prog)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle_serving_client/io/__init__.py", line 147, in save_model
main_program=main_program)
File "<decorator-gen-79>", line 2, in save_inference_model
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
return wrapped_func(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 234, in __impl__
return func(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/io.py", line 1336, in save_inference_model
var, 1., name="save_infer_model/scale_{}".format(i))
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 11449, in scale
type='scale', inputs=inputs, outputs={'Out': out}, attrs=attrs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3023, in append_op
attrs=kwargs.get("attrs", None))
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2107, in __init__
for frame in traceback.extract_stack():
ExternalError: Cuda error(77), an illegal memory access was encountered.
[Advise: The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at /paddle/paddle/fluid/platform/stream/cuda_stream.cc:65)
[operator < scale > error]Traceback (most recent call last): File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle_serving_server_gpu/pipeline/operator.py", line 764, in _run_process midped_batch = self.process(feed_batch, typical_logid) File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle_serving_server_gpu/pipeline/operator.py", line 443, in process log_id=typical_logid) File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle_serving_app/local_predict.py", line 217, in predict self.predictor.zero_copy_run() OSError: In user code:
File "tools/export_serving_model.py", line 107, in <module>
main()
File "tools/export_serving_model.py", line 93, in main
save_serving_model(FLAGS, exe, feed_vars, test_fetches, infer_prog)
File "tools/export_serving_model.py", line 63, in save_serving_model
main_program=infer_prog)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle_serving_client/io/__init__.py", line 147, in save_model
main_program=main_program)
File "<decorator-gen-79>", line 2, in save_inference_model
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
return wrapped_func(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 234, in __impl__
return func(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/io.py", line 1336, in save_inference_model
var, 1., name="save_infer_model/scale_{}".format(i))
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 11449, in scale
type='scale', inputs=inputs, outputs={'Out': out}, attrs=attrs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3023, in append_op
attrs=kwargs.get("attrs", None))
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2107, in __init__
for frame in traceback.extract_stack():
ExternalError: Cuda error(77), an illegal memory access was encountered.
[Advise: The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at /paddle/paddle/fluid/platform/stream/cuda_stream.cc:65)
[operator < scale > error]ERROR 2021-03-23 15:33:32,418 [dag.py:409] (data_id=4 log_id=0) Failed to predict: (data_id=4 log_id=4) [bill_det|0] Failed to process(batch: [4]): In user code:
File "tools/export_serving_model.py", line 107, in <module>
main()
File "tools/export_serving_model.py", line 93, in main
save_serving_model(FLAGS, exe, feed_vars, test_fetches, infer_prog)
File "tools/export_serving_model.py", line 63, in save_serving_model
main_program=infer_prog)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle_serving_client/io/__init__.py", line 147, in save_model
main_program=main_program)
File "<decorator-gen-79>", line 2, in save_inference_model
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
return wrapped_func(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 234, in __impl__
return func(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/io.py", line 1336, in save_inference_model
var, 1., name="save_infer_model/scale_{}".format(i))
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 11449, in scale
type='scale', inputs=inputs, outputs={'Out': out}, attrs=attrs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(*args, **kwargs)
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3023, in append_op
attrs=kwargs.get("attrs", None))
File "/home/sy/.conda/envs/maskrcnn-paddle-2.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2107, in __init__
for frame in traceback.extract_stack():
ExternalError: Cuda error(77), an illegal memory access was encountered.
[Advise: The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at /paddle/paddle/fluid/platform/stream/cuda_stream.cc:65)
[operator < scale > error]
PaddleDetection版本：2.0 beta serving 版本：0.4 paddlepaddle-gpu : 2.0.0rc1 用serving方式部署了一个基于私有数据集的MaskRcnn模型，测试如下（采用多线程同时发送http请求方式）： 1）全部用只有背景的图片测试，正常工作。 2）全部用带有目标的图片测试，正常工作，检测结果正常，目标都能检出。 3）混合有目标的图片和没有目标的背景图片测试，出现如下错误 ERROR 2021-03-19 12:09:25,110 [operator.py:733] (data_id=19 log_id=19) [bill_det|0] Failed to process(batch: [19]): In user code:
Traceback (most recent call last): File "/home/sy/.conda/envs/solov2-paddle-2.0rc1/lib/python3.7/site-packages/paddle_serving_server_gpu/pipeline/operator.py", line 728, in _run_process midped_batch = self.process(feed_batch, typical_logid) File "/home/sy/.conda/envs/solov2-paddle-2.0rc1/lib/python3.7/site-packages/paddle_serving_server_gpu/pipeline/operator.py", line 437, in process log_id=typical_logid) File "/home/sy/.conda/envs/solov2-paddle-2.0rc1/lib/python3.7/site-packages/paddle_serving_app/local_predict.py", line 201, in predict self.predictor.zero_copy_run() OSError: In user code:
ERROR 2021-03-19 12:09:25,141 [dag.py:409] (data_id=19 log_id=0) Failed to predict: (data_id=19 log_id=19) [bill_det|0] Failed to process(batch: [19]): In user code:
请问这个问题如何解决？