loading annotations into memory...
Done (t=2.84s)
creating index...
index created!
2023-11-22 21:21:50,302 [INFO] Dataset Cache file hash/version check success.
2023-11-22 21:21:50,302 [INFO] Load dataset cache from [small_coco\train2017.cache.npy] success.
Scanning 'small_coco\train2017.cache.npy' images and labels... 32520 found, 0 missing, 0 empty, 0 corrupted: 100%|█
2023-11-22 21:21:50,451 [INFO] Dataloader num parallel workers: [4]
[WARNING] ME(22588:29712,MainProcess):2023-11-22-21:21:50.486.711 [mindspore\dataset\engine\datasets_user_defined.py:784] Python multiprocessing is not supported on Windows platform.
2023-11-22 21:21:50,905 [WARNING] Parse Model, args: nearest, keep str type
2023-11-22 21:21:50,945 [WARNING] Parse Model, args: nearest, keep str type
2023-11-22 21:21:51,003 [INFO] number of network params, total: 62.001796M, trainable: 61.949149M
[WARNING] ME(22588:29712,MainProcess):2023-11-22-21:21:52.178.27 [mindspore\train\serialization.py:1408] For 'load_param_into_net', remove parameter prefix name: ema., continue to load.
2023-11-22 21:21:52,048 [INFO] Load checkpoint from [./yolov3_backbone.ckpt] success.
2023-11-22 21:21:52,394 [INFO] Registry(name=callback, total=4)
2023-11-22 21:21:52,394 [INFO] (0): YoloxSwitchTrain in mindyolo\utils\callback.py
2023-11-22 21:21:52,394 [INFO] (1): EvalWhileTrain in mindyolo\utils\callback.py
2023-11-22 21:21:52,394 [INFO] (2): SummaryCallback in mindyolo\utils\callback.py
2023-11-22 21:21:52,394 [INFO] (3): ProfilerCallback in mindyolo\utils\callback.py
2023-11-22 21:21:52,394 [INFO]
2023-11-22 21:21:52,404 [INFO] got 1 active callback as follows:
2023-11-22 21:21:52,404 [INFO] SummaryCallback()
2023-11-22 21:21:52,405 [WARNING] The first epoch will be compiled for the graph, which may take a long time; You can come back later :).
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
[INFO] albumentations load success
[INFO] albumentations load success
[INFO] albumentations load success
[INFO] albumentations load success
[WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:06.238.868 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value.
[WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:07.905.046 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value.
[WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:08.742.447 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value.
[WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:09.479.567 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value.
Traceback (most recent call last):
File "d:/晶晶宝贝/mindyolo-train/mindyolo-train/main.py", line 394, in
main()
File "d:/晶晶宝贝/mindyolo-train/mindyolo-train/main.py", line 361, in main
trainer.train(
File "d:\晶晶宝贝\mindyolo-train\mindyolo-train\mindyolo\utils\trainer_factory.py", line 170, in train
run_context.loss, run_context.lr = self.train_step(imgs, labels, segments,
File "d:\晶晶宝贝\mindyolo-train\mindyolo-train\mindyolo\utils\trainer_factory.py", line 366, in train_step
loss, lossitem, , grads_finite = self.train_step_fn(imgs, labels, True)
File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 718, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args, kwargs)
File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 121, in wrapper
results = fn(*arg, *kwargs)
File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 350, in call
raise err
File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 347, in call
phase = self.compile(self.fn.name, args_list, kwargs)
File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 435, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, kwargs, phase, True)
RuntimeError: bad allocation
loading annotations into memory... Done (t=2.84s) creating index... index created! 2023-11-22 21:21:50,302 [INFO] Dataset Cache file hash/version check success. 2023-11-22 21:21:50,302 [INFO] Load dataset cache from [small_coco\train2017.cache.npy] success. Scanning 'small_coco\train2017.cache.npy' images and labels... 32520 found, 0 missing, 0 empty, 0 corrupted: 100%|█ 2023-11-22 21:21:50,451 [INFO] Dataloader num parallel workers: [4] [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:21:50.486.711 [mindspore\dataset\engine\datasets_user_defined.py:784] Python multiprocessing is not supported on Windows platform. 2023-11-22 21:21:50,905 [WARNING] Parse Model, args: nearest, keep str type 2023-11-22 21:21:50,945 [WARNING] Parse Model, args: nearest, keep str type 2023-11-22 21:21:51,003 [INFO] number of network params, total: 62.001796M, trainable: 61.949149M [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:21:52.178.27 [mindspore\train\serialization.py:1408] For 'load_param_into_net', remove parameter prefix name: ema., continue to load. 2023-11-22 21:21:52,048 [INFO] Load checkpoint from [./yolov3_backbone.ckpt] success. 2023-11-22 21:21:52,394 [INFO] Registry(name=callback, total=4) 2023-11-22 21:21:52,394 [INFO] (0): YoloxSwitchTrain in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] (1): EvalWhileTrain in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] (2): SummaryCallback in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] (3): ProfilerCallback in mindyolo\utils\callback.py 2023-11-22 21:21:52,394 [INFO] 2023-11-22 21:21:52,404 [INFO] got 1 active callback as follows: 2023-11-22 21:21:52,404 [INFO] SummaryCallback() 2023-11-22 21:21:52,405 [WARNING] The first epoch will be compiled for the graph, which may take a long time; You can come back later :). albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:06.238.868 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:07.905.046 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:08.742.447 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. [WARNING] ME(22588:29712,MainProcess):2023-11-22-21:22:09.479.567 [mindspore\ops\primitive.py:819] The "use_copy_slice" is a constexpr function. The input arguments must be all constant value. Traceback (most recent call last): File "d:/晶晶宝贝/mindyolo-train/mindyolo-train/main.py", line 394, in
main()
File "d:/晶晶宝贝/mindyolo-train/mindyolo-train/main.py", line 361, in main
trainer.train(
File "d:\晶晶宝贝\mindyolo-train\mindyolo-train\mindyolo\utils\trainer_factory.py", line 170, in train
run_context.loss, run_context.lr = self.train_step(imgs, labels, segments,
File "d:\晶晶宝贝\mindyolo-train\mindyolo-train\mindyolo\utils\trainer_factory.py", line 366, in train_step
loss, lossitem, , grads_finite = self.train_step_fn(imgs, labels, True) File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 718, in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args, kwargs)
File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 121, in wrapper
results = fn(*arg, *kwargs) File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 350, in call
raise err File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 347, in call
phase = self.compile(self.fn.name, args_list, kwargs) File "C:\Users\76161.conda\envs\mindspore\lib\site-packages\mindspore\common\api.py", line 435, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, kwargs, phase, True) RuntimeError: bad allocation