search

WHULuoJiaTeam / luojianet

http://58.48.42.237/luojiaNet/
Apache License 2.0
187 stars 36 forks source link

[BUG]model zoo中deeplabv3训练中出现RuntimeError: luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:199 ImportBpropFromMindIR] The bprop mindir files are not up to da te. #17

Closed 123pu closed 2 years ago

123pu commented 2 years ago

deeplabv3模型训练时出现以下错误： [WARNING] ME(2199:140703017723712,MainProcess):2022-10-28-16:29:18.609.789 [luojianet_ms/common/_decorator.py:38] 'TensorAdd' is d eprecated from version 1.1 and will be removed in a future version, use 'Add' instead. Total Epoch:200 Training num:10000 Validation num:1000 INFO:log:Total Epoch:200 Training num:10000 Validation num:1000 [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.388 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:172] C heckBpropHash] The bprop mindir files are not up to date. Please run the /usr/local/python3.7.5/lib/python3.7/site-packages/luojia net_ms/ops/_grad/../bprop_mindir/generate_mindir.py to generate new mindir files. bprop_fg hash: 3d4ca3af3054d32fe54a557e457674558c4179705eccb4c3dae775993ba1a76a bprop hash list:

[ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.420 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] 5dbd9e9d72c7b3227bfe7cc41cc4311526259c3297cb24ab2d4d7aa122c901e6 [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.430 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] d15b0d6de0f996dcacce657aaec10a3a526e8567314332c8628d9cbc7bc26a03 [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.438 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] c252daaf204cd7d19bad3054c4734cd874f00b9a0ad6d8b3b01ffe56bf1f0b2f [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.448 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] 15cf11818e324bb31ab387f78edcce13902e54eb561b8406f4aac743e626cc2a [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.455 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] da9b8c2c77b9335db3c701e0a27fc1b3514c7abf3614b2b477cf94d2420d770e [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.461 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] dcbcbe9c73bbeb292f97cefef13d1e66d4df58dc4d0f4d61b3b3c0c48e61f014 [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.466 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] 300e3a12184504bf922c740d3c92af822c2517aef91dd32c009b58266169f93d [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.473 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] 8a2f1d24f72f3e2821cdd873682fda4b8322905ab83ebe12885e2c9aec2957d0 [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.478 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] 9c4e884cbc35fd59fd028a19bc7f9c9510d0b05d90ad3de2939e2498a66a6ade [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.484 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] fd36050ed51ca754e75ba8ab0e59cf7aefc9aaf4425e607aa1d6115a69d64933 [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.488 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] 55e08f68aa4e972c4ebb7fc191119b682d6347830d383020968203382a8b59cc [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.493 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] ec3beb004dc8c56197a40e8b91c092539a103b8e7e7be62edaf0e83671e1966e [ERROR] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.499 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:177] C heckBpropHash] 2930c5739d67414f42123aec886e845ed9e11e82b7a792124f3a04f57e17b5c4 [CRITICAL] OPTIMIZER(2199,7ff7f963e740,python3):2022-10-28-16:29:25.897.507 [luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:199 ] ImportBpropFromMindIR] The bprop mindir files are not up to date. [WARNING] VM(2199,7ff768fb9700,python3):2022-10-28-16:29:25.897.759 [luojianet_ms/ccsrc/runtime/pynative/op_task.h:106] Run] Op bu ild failed, no need to launch. Traceback (most recent call last): File "train.py", line 67, in train_net() File "train.py", line 63, in train_net train(param=param, model=model, train_dataset=train_dataset, valid_dataset=val_dataset) File "/code/utils/deeplearning_dp.py", line 121, in train train_net_step(data, label) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 613, in call raise err File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 610, in call output = self._run_construct(cast_inputs, kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 430, in _run_construct output = self.forward(*cast_inputs, kwargs) File "/code/utils/deeplearning_dp.py", line 50, in forward loss = self.network(data, label) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 613, in call raise err File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 610, in call output = self._run_construct(cast_inputs, kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 430, in _run_construct output = self.forward(*cast_inputs, *kwargs) File "/code/utils/deeplearning_dp.py", line 25, in forward out = self.backbone(data) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 613, in call raise err File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 610, in call output = self._run_construct(cast_inputs, kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 430, in _run_construct output = self.forward(cast_inputs, kwargs) File "/code/nets/init.py", line 17, in forward x =self.model(x) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 613, in call raise err File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 610, in call output = self._run_construct(cast_inputs, kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 430, in _run_construct output = self.forward(*cast_inputs, kwargs) File "/code/nets/deeplabv3.py", line 204, in forward out = self.resnet(x) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 613, in call raise err File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 610, in call output = self._run_construct(cast_inputs, kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 430, in _run_construct output = self.forward(*cast_inputs, *kwargs) File "/code/nets/deeplabv3.py", line 64, in forward out = self.relu(out) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 613, in call raise err File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 610, in call output = self._run_construct(cast_inputs, kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/cell.py", line 430, in _run_construct output = self.forward(cast_inputs, kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/nn/layer/activation.py", line 299, in forward return self.relu(x) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/ops/primitive.py", line 295, in call return _run_op(self, self.name, args) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/common/api.py", line 91, in wrapper results = fn(*arg, **kwargs) File "/usr/local/python3.7.5/lib/python3.7/site-packages/luojianet_ms/ops/primitive.py", line 755, in _run_op output = real_run_op(obj, op_name, args) RuntimeError: luojianet_ms/ccsrc/frontend/optimizer/ad/kprim.cc:199 ImportBpropFromMindIR] The bprop mindir files are not up to date. 以上出现的错误，是什么原因造成的呢，希望给予帮助，非常感谢！！！

Expected Behavior

Current Behavior

Context

Steps to Reproduce

Your Environment

MiZhangWhuer commented 2 years ago

/usr/local/python3.7.5/lib/python3.7/site-packages/luojia net_ms/ops/_grad/../bprop_mindir/generate_mindir.py

运行如下命令： python /usr/local/python3.7.5/lib/python3.7/site-packages/luojia net_ms/ops/_grad/../bprop_mindir/generate_mindir.py 然后再执行程序