mindspore-lab / mindyolo

A toolbox of yolo models and algorithms based on MindSpore
Apache License 2.0
84 stars 36 forks source link

YOLOv3 To train COCO datasets #245

Open Chen414 opened 8 months ago

Chen414 commented 8 months ago

loading annotations into memory... Done (t=2.84s) creating index... index created! 2023-11-22 21:21:50,302 [INFO] Dataset Cache file hash/version check success. 2023-11-22 21:21:50,302 [INFO] Load dataset cache from [small_coco\train2017.cache.npy] success. Scanning 'small_coco\train2017.cache.npy' images and labels... 32520 found, 0 missing, 0 empty, 0 corrupted: 100%|█ 2023-11-22 21:21:50,451 [INFO] Dataloader num parallel workers: [4] [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:21:50.486.711 [mindspore\dataset\engine\datasets_user_defined.py:784] Python multiprocessing is not supported on Windows platform. 2023-11-22 21:21:50,905 [WARNING] Parse Model, args: nearest, keep str type 2023-11-22 21:21:50,945 [WARNING] Parse Model, args: nearest, keep str type 2023-11-22 21:21:51,003 [INFO] number of network params, total: 62.001796M, trainable: 61.949149M [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:21:52.178.27 [mindspore\train\serialization.py:1408] For 'load_param_into_net', remove parameter prefix name: ema., continue to load. 2023-11-22 21:21:52,048 [INFO] Load checkpoint from [./yolov3_backbone.ckpt] success. 2023-11-22 21:21:52,394 [INFO] Registry(name=callback, total=4) 2023-11-22 21:21:52,394 [INFO] (0): YoloxSwitchTrain in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] (1): EvalWhileTrain in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] (2): SummaryCallback in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] (3): ProfilerCallback in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] 2023-11-22 21:21:52,404 [INFO] got 1 active callback as follows: 2023-11-22 21:21:52,404 [INFO] SummaryCallback() 2023-11-22 21:21:52,405 [WARNING] The first epoch will be compiled for the graph, which may take a long time; You can come back later :). albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:06.238.868 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:07.905.046 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:08.742.447 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:09.479.567 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. Traceback (most recent call last): File "d:/晶晶宝贝/mindyolo-train/mindyolo-train/main.py", line 394, in main() File "d:/晶晶宝贝/mindyolo-train/mindyolo-train/main.py", line 361, in main trainer.train( File "d:\晶晶宝贝\mindyolo-train\mindyolo-train\mindyolo\utils\trainer_factory.py", line 170, in train run_context.loss, run_context.lr = self.train_step(imgs, labels, segments, File "d:\晶晶宝贝\mindyolo-train\mindyolo-train\mindyolo\utils\trainer_factory.py", line 366, in train_step
loss, lossitem, , grads_finite = self.train_step_fn(imgs, labels, True) File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 718, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args, kwargs)
File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 121, in wrapper
results = fn(*arg, *kwargs) File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 350, in call
raise err File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 347, in call
phase = self.compile(self.fn.name,
args_list,
kwargs) File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 435, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, kwargs, phase, True) RuntimeError: bad allocation

zhanghuiyao commented 7 months ago

这个看起来是编译的时候报错了 可以确认下mindspore版本和内存占用情况,版本依赖可以参考 installation