huawei-noah / vega

AutoML tools chain
http://www.noahlab.com.hk/opensource/vega/
Other
841 stars 177 forks source link

vega-noah:esr-ea 训练失败 #274

Open ultraWeiger opened 1 year ago

ultraWeiger commented 1 year ago

[ST][MS/modelzoo][NET][ascend][esr_ea] train fail https://e.gitee.com/mind_spore/issues/table?issue=I5XW7F

anzq001 commented 1 year ago

Related testcase / 关联用例 (Mandatory / 必填) test_ms_esr_ea_ilsvrc_ascend_check_fps_0001.py

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填) cd solution_test/cases/02network/00cv/dnetnas/train pytest -s test_ms_esr_ea_ilsvrc_ascend_check_fps_0001.py Describe the expected behavior / 预期结果 (Mandatory / 必填) 训练成功

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

[TRACE] TDT(93189,python3.7):2022-10-26-10:07:19.520.164 [status:Running] [log.cpp:154]Channel "e67241b8-54d2-11ed-be98-78b46a368ae4": Send Sample Files,[tensor_data_deliver.cpp:279:Send]94052
[CRITICAL] ANALYZER(93189,ffffafba6010,python3.7):2022-10-26-10:07:22.753.424 [mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1455] GetEvaluatedValueForNameSpaceString] External object has no attribute _null
2022-10-26 10:07:23.554 ERROR Traceback (most recent call last):
  File "/home/jenkins/.local/lib/python3.7/site-packages/vega/core/scheduler/local_master.py", line 58, in run
    worker.train_process()
  File "/home/jenkins/.local/lib/python3.7/site-packages/vega/common/wrappers.py", line 66, in wrapper
    r = func(self, *args, **kwargs)
  File "/home/jenkins/.local/lib/python3.7/site-packages/vega/trainer/trainer_base.py", line 130, in train_process
    self._train_loop()
  File "/home/jenkins/.local/lib/python3.7/site-packages/vega/trainer/trainer_base.py", line 266, in _train_loop
    self._train_epoch()
  File "/home/jenkins/.local/lib/python3.7/site-packages/vega/trainer/trainer_ms.py", line 106, in _train_epoch
    dataset_sink_mode=self.dataset_sink_mode)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 1062, in train
    initial_epoch=initial_epoch)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 98, in wrapper
    func(self, *args, **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 624, in _train
    cb_params, sink_size, initial_epoch, valid_infos)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 702, in _train_dataset_sink_process
    outputs = train_network(*inputs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/nn/cell.py", line 619, in __call__
    out = self.compile_and_run(*args)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/nn/cell.py", line 1004, in compile_and_run
    self.compile(*inputs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/nn/cell.py", line 976, in compile
    jit_config_dict=self._jit_config_dict)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/common/api.py", line 1150, in compile
    result = self._graph_executor.compile(obj, args_list, phase, self._use_vm_mode())
AttributeError: External object has no attribute _null