[coat_tiny] [Ascend910] [GRAPH] Unable to reproduce precision

If this is your first time, please read our contributor guidelines: https://github.com/mindspore-lab/mindcv/blob/main/CONTRIBUTING.md

Describe the bug/ 问题描述 (Mandatory / 必填) coat_tiny边训边推过程中精度异常

Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端: /device ascend
Software Environment / 软件环境 (Mandatory / 必填): -- MindSpore version (e.g., 1.7.0.Bxxx) :MindSpore_v2.2.10.B180 mindcv-test/ms2.2 -- Python version (e.g., Python 3.7.5) :3.8 -- OS platform and distribution (e.g., Linux Ubuntu 16.04):5.4.0-139-generic 156-Ubuntu x86_64 GNU/Linux -- GCC/Compiler version (if compiled from source):7.3.0
Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式: /mode graph

To Reproduce / 重现步骤 (Mandatory / 必填) Steps to reproduce the behavior:

mpirun --allow-run-as-root -n 8 python train.py --config configs/coat/coat_tiny_ascend.yaml --distribute True --data_dir /ImageNet_Origin/

Expected behavior / 预期结果 (Mandatory / 必填) 复现达标精度

Screenshots/ 日志 / 截图 (Mandatory / 必填) [2024-01-19 07:00:55] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4100%, time: 270.009686s [2024-01-19 07:26:12] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4100%, time: 29.201576s [2024-01-19 07:51:29] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.353800s [2024-01-19 08:16:47] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.382253s [2024-01-19 08:42:05] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.319528s [2024-01-19 09:07:23] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.244623s [2024-01-19 09:32:42] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.391960s [2024-01-19 09:57:59] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.351547s [2024-01-19 10:23:16] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.447677s [2024-01-19 10:48:33] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.377890s [2024-01-19 11:13:49] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.384644s [2024-01-19 11:39:06] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.427650s [2024-01-19 12:04:23] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.483277s [2024-01-19 12:29:42] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.500403s [2024-01-19 12:55:01] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.434532s [2024-01-19 13:20:20] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.438494s [2024-01-19 13:45:38] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.420137s [2024-01-19 14:10:55] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.318372s [2024-01-19 14:36:13] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.397343s [2024-01-19 15:01:31] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.418429s [2024-01-19 15:26:50] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.317333s [2024-01-19 15:52:08] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.506110s [2024-01-19 16:17:26] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.385936s [2024-01-19 16:42:44] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.498171s [2024-01-19 17:08:02] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.447979s [2024-01-19 17:33:19] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.538416s [2024-01-19 17:58:37] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.527485s [2024-01-19 18:23:54] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.531306s [2024-01-19 18:49:12] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.356908s [2024-01-19 19:14:30] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.494307s [2024-01-19 19:39:47] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.289113s [2024-01-19 20:05:04] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.457028s [2024-01-19 20:30:22] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.357243s [2024-01-19 20:55:39] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.254013s [2024-01-19 21:20:55] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.580259s [2024-01-19 21:46:12] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.281834s [2024-01-19 22:11:29] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.381883s [2024-01-19 22:36:46] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.446761s [2024-01-19 23:02:03] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.229984s [2024-01-19 23:27:21] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.501944s [2024-01-19 23:52:38] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.437370s [2024-01-20 00:17:55] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.322477s [2024-01-20 00:43:14] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.501396s [2024-01-20 01:08:32] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.268045s [2024-01-20 01:33:49] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.343203s [2024-01-20 01:59:08] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.462093s [2024-01-20 02:24:27] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.427761s [2024-01-20 02:49:46] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.0720%, Top_5_Accuracy: 0.4120%, time: 29.616131s

Additional context / 备注 (Optional / 选填) Add any other context about the problem here.

mindspore-lab / mindcv

[coat_tiny] [Ascend910] [GRAPH] Unable to reproduce precision #758