total images num: 9
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
[ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.540 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/1464347597.py]
[WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.629 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/1464347597.py' may not exists.
[ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.705 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/1464347597.py]
[WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.725 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/1464347597.py' may not exists.
[ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.442 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/1464347597.py]
[WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.470 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/1464347597.py' may not exists.
[ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.529 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/130566329.py]
[WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.547 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/130566329.py' may not exists.
[ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.738.671 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/130566329.py]
[WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.738.696 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/130566329.py' may not exists.
/tmp/ipykernel_50128/1464347597.py in apply_eval(eval_param_dict)
20 image_shape = data['image_shape']
21
---> 22 output = net(Tensor(img_np))
23
24 for batch_idx in range(img_np.shape[0]):
~/.local/lib/python3.9/site-packages/mindspore/nn/cell.py in call(self, *args, *kwargs)
728 f"function, please use context.set_context to set pynative mode.")
729 self._self_check()
--> 730 out = self.compile_and_run(args, **kwargs)
731 return out
732
Start Eval! Load Checkpoint!
========================================
total images num: 9 loading annotations into memory... Done (t=0.01s) creating index... index created! [ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.540 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/1464347597.py] [WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.629 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/1464347597.py' may not exists. [ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.705 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/1464347597.py] [WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.735.725 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/1464347597.py' may not exists. [ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.442 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/1464347597.py] [WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.470 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/1464347597.py' may not exists. [ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.529 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/130566329.py] [WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.737.547 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/130566329.py' may not exists. [ERROR] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.738.671 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/tmp/ipykernel_50128/130566329.py] [WARNING] CORE(50128,e7ffe003e020,python):2024-10-15-18:07:28.738.696 [mindspore/core/utils/info.cc:121] ToString] The file '/tmp/ipykernel_50128/130566329.py' may not exists.
[ERROR] DEVICE(50128,e7fead72f120,python):2024-10-15-18:08:02.895.892 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:233] TaskExceptionCallback] Run Task failed, task_id: 11, stream_id: 1, tid: 50128, device_id: 0, retcode: 507018 (aicpu exception) [ERROR] DEVICE(50128,e7fead72f120,python):2024-10-15-18:08:02.896.541 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:274] SyncStream] Call runtime aclrtSynchronizeStreamWithTimeout error. [ERROR] DEVICE(50128,e7fead72f120,python):2024-10-15-18:08:02.896.578 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_stream_manager.cc:287] SyncAllStreams] SyncStream for stream id 0 failed.
RuntimeError Traceback (most recent call last) /tmp/ipykernel_50128/3533520499.py in <cell line: 26>() 24 ssd_eval(mindrecord_file, "./ssd-60_9.ckpt", anno_json) 25 ---> 26 eval_net()
/tmp/ipykernel_50128/3533520499.py in eval_net() 22 def eval_net(): 23 print("Start Eval!") ---> 24 ssd_eval(mindrecord_file, "./ssd-60_9.ckpt", anno_json) 25 26 eval_net()
/tmp/ipykernel_50128/3533520499.py in ssd_eval(dataset_path, ckpt_path, anno_json) 16 print("total images num: ", total) 17 eval_param_dict = {"net": net, "dataset": ds, "anno_json": anno_json} ---> 18 mAP = apply_eval(eval_param_dict) 19 print("\n========================================\n") 20 print(f"mAP: {mAP}")
/tmp/ipykernel_50128/1464347597.py in apply_eval(eval_param_dict) 20 image_shape = data['image_shape'] 21 ---> 22 output = net(Tensor(img_np)) 23 24 for batch_idx in range(img_np.shape[0]):
~/.local/lib/python3.9/site-packages/mindspore/nn/cell.py in call(self, *args, *kwargs) 728 f"function, please use context.set_context to set pynative mode.") 729 self._self_check() --> 730 out = self.compile_and_run(args, **kwargs) 731 return out 732
~/.local/lib/python3.9/site-packages/mindspore/nn/cell.py in compile_and_run(self, *args, *kwargs) 1149 self.add_flags(ge_sync_data=False) 1150 new_args = _get_args_for_run(self, args, kwargs, self._compile_args) -> 1151 return _cell_graph_executor(self, new_args, phase=self.phase) 1152 1153 def auto_parallel_compile_and_run(self):
~/.local/lib/python3.9/site-packages/mindspore/common/api.py in call(self, obj, phase, args) 1879 if context.get_context("precompile_only") or _is_role_sched(): 1880 return None -> 1881 return self.run(obj, args, phase=phase) 1882 1883 def has_compiled(self, phase='predict'):
~/.local/lib/python3.9/site-packages/mindspore/common/api.py in run(self, obj, phase, args) 1930 phase_real = phase + '.' + str(obj.create_time) + '.' + str(id(obj)) + '.' + obj.arguments_key 1931 if self.has_compiled(phase_real): -> 1932 return self._exec_pip(obj, args, phase=phase_real) 1933 raise KeyError('{} graph is not exist.'.format(phase_real)) 1934
~/.local/lib/python3.9/site-packages/mindspore/common/api.py in wrapper(*arg, kwargs) 186 @wraps(fn) 187 def wrapper(*arg, *kwargs): --> 188 results = fn(arg, kwargs) 189 return _convert_python_data(results) 190
~/.local/lib/python3.9/site-packages/mindspore/common/api.py in _exec_pip(self, obj, phase, args) 1910 fn = obj.construct 1911 obj.__parse_method = fn.name__ -> 1912 return self._graph_executor(args, phase) 1913 1914 def run(self, obj, args, phase='predict'):
RuntimeError: Sync stream failed:Ascend_0
Ascend Error Message:
E39999: Inner Error! E39999: [PID: 50128] 2024-10-15-18:08:02.895.102 The error from device(chipId:0, dieId:0), serial number is 2, an exception occurred during AICPU execution, stream_id:1, task_id:11, errcode:5, msg:aicpu execute failed..[FUNC:ProcessStarsAicpuErrorInfo][FILE:device_error_proc.cc][LINE:1280][THREAD:52634] TraceBack (most recent call last): Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1221][THREAD:52634] AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1062][THREAD:52634] Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=11, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinic_kernel_task.cc][LINE:1024][THREAD:52634] Aicpu kernel execute failed, device_id=0, stream_id=1, task_id=11, flip_num=0, fault so_name=, fault kernel_name=, fault op_name=, extend_info=.[FUNC:GetError][FILE:stream.cc][LINE:1062][THREAD:52634] rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53][THREAD:52634] synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161][THREAD:52634]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
C++ Call Stack: (For framework developers)
mindspore/ccsrc/runtime/graph_scheduler/graph_scheduler.cc:998 Run