search

huawei-noah / vega

AutoML tools chain
http://www.noahlab.com.hk/opensource/vega/
Other
841 stars 177 forks source link

adelaide-ea训练失败 #273

Open ultraWeiger opened 1 year ago

ultraWeiger commented 1 year ago

详情见此

[ST][MS/modelzoo][NET][ascend][adelaide_ea] train fail https://e.gitee.com/mind_spore/dashboard?issue=I5XW50

anzq001 commented 1 year ago

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填) cd solution_test/cases/02network/00cv/dnetnas/train pytest -s test_ms_adelaide_ea_ilsvrc_ascend_check_fps_0001.py Describe the expected behavior / 预期结果 (Mandatory / 必填) 训练成功

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

epoch: 1 step: 1, loss is 2.782377243041992
[ERROR] OP(174721,python3.7):2022-10-25-21:49:07.766.417 [nn_calculation_ops.cc:4212][OP_PROTO] CheckNegativePadConv2d:4212 OpName:[Conv2D] "pads should be positive,  actual is [-1,-1,-1,-1]."
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.538 [op_desc.cc:1295]174721 CallInferFunc: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Call][InferFunc] for Conv2D failed. ret:4294967295
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.561 [analyzer.cc:160]174721 GetJsonObject: ErrorNo: 1343225857(Parameter's invalid!) [COMP][PRE_OPT][Check][SessionId]session_id:0 does not exist! graph_id:88
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.634 [analyzer.cc:260]174721 DoAnalyze: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Check][Param:graph_info]null is invalid
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.643 [analyzer.cc:160]174721 GetJsonObject: ErrorNo: 1343225857(Parameter's invalid!) [COMP][PRE_OPT][Check][SessionId]session_id:0 does not exist! graph_id:88
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.660 [analyzer.cc:217]174721 SaveAnalyzerDataToFile: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Check][Param:graph_info]null is invalid
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.673 [infershape_pass.cc:120]174721 Infer: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][InferShapeAndType] for node:Conv2D(Conv2D) failed
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.681 [infer_base_pass.cc:114]174721 InferAndUpdate: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][Infer] failed for node Conv2D, ret: 1343242270
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.688 [infer_base_pass.cc:71]174721 Run: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][InferAndUpdate] for node Conv2D failed! ret: 1343242270
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.701 [base_pass.cc:532]174721 RunPassesOnNode: ErrorNo: 1343225860(Internal errors) [COMP][PRE_OPT][Process][Pass] InferShapePass on node Conv2D failed, result 4294967295, the passes will be terminated immediately.
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.709 [base_pass.cc:483]174721 RunPassesNodeOnce: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Process][Passes] on node Conv2D type Conv2D failed, error code:4294967295
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.717 [base_pass.cc:439]174721 RunPassesGraphRepass: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Process][Passes] on node Conv2D type Conv2D failed, error code:4294967295
[ERROR] GE(174721,python3.7):2022-10-25-21:49:07.766.728 [graph_prepare.cc:2197]174721 InferShapeForPreprocess: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Run][GePasses] infershape for preprocess failed, ret:4294967295.