search

PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.78k stars 2.88k forks source link

OSError: [Errno 5] Input/output error #6332

Open thongvhoang opened 2 years ago

thongvhoang commented 2 years ago

问题确认 Search before asking

bug描述 Describe the Bug

I train the ppyolo_r50vd_dcn_1x_coco model using Google Colab environment. I use the python command: !python tools/train.py -c configs/ppyolo/ppyolo_r50vd_dcn_1x_coco.yml --eval And the error generates after that

/usr/local/lib/python3.7/dist-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
/usr/local/lib/python3.7/dist-packages/scipy/fft/__init__.py:97: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
  from numpy.dual import register_func
/usr/local/lib/python3.7/dist-packages/scipy/sparse/sputils.py:17: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.
  supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/usr/local/lib/python3.7/dist-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
W0701 12:14:48.330195 169235 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 11.2, Runtime API Version: 10.2
W0701 12:14:48.335217 169235 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.
[07/01 12:14:55] ppdet.utils.checkpoint INFO: The shape (258,) in pretrained weight yolo_head.yolo_output.0.bias is unmatched with the shape [21] in model yolo_head.yolo_output.0.bias. And the weight yolo_head.yolo_output.0.bias will not be loaded
[07/01 12:14:55] ppdet.utils.checkpoint INFO: The shape (258, 1024, 1, 1) in pretrained weight yolo_head.yolo_output.0.weight is unmatched with the shape [21, 1024, 1, 1] in model yolo_head.yolo_output.0.weight. And the weight yolo_head.yolo_output.0.weight will not be loaded
[07/01 12:14:55] ppdet.utils.checkpoint INFO: The shape (258,) in pretrained weight yolo_head.yolo_output.1.bias is unmatched with the shape [21] in model yolo_head.yolo_output.1.bias. And the weight yolo_head.yolo_output.1.bias will not be loaded
[07/01 12:14:55] ppdet.utils.checkpoint INFO: The shape (258, 512, 1, 1) in pretrained weight yolo_head.yolo_output.1.weight is unmatched with the shape [21, 512, 1, 1] in model yolo_head.yolo_output.1.weight. And the weight yolo_head.yolo_output.1.weight will not be loaded
[07/01 12:14:55] ppdet.utils.checkpoint INFO: The shape (258,) in pretrained weight yolo_head.yolo_output.2.bias is unmatched with the shape [21] in model yolo_head.yolo_output.2.bias. And the weight yolo_head.yolo_output.2.bias will not be loaded
[07/01 12:14:55] ppdet.utils.checkpoint INFO: The shape (258, 256, 1, 1) in pretrained weight yolo_head.yolo_output.2.weight is unmatched with the shape [21, 256, 1, 1] in model yolo_head.yolo_output.2.weight. And the weight yolo_head.yolo_output.2.weight will not be loaded
[07/01 12:14:55] ppdet.utils.checkpoint INFO: Finish loading model weights: /content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/pretrained_model/ppyolo_r50vd_dcn_1x_coco.pdparams
[07/01 12:15:02] ppdet.engine INFO: Epoch: [0] [  0/526] learning_rate: 0.000000 loss_xy: 1.268517 loss_wh: 2.687380 loss_iou: 4.231785 loss_iou_aware: 0.679625 loss_obj: 20271.953125 loss_cls: 0.852223 loss: 20281.671875 eta: 2 days, 1:15:46 batch_cost: 6.7432 data_cost: 5.5761 ips: 1.4830 images/s
[07/01 12:16:13] ppdet.engine INFO: Epoch: [0] [  5/526] learning_rate: 0.000000 loss_xy: 1.177958 loss_wh: 2.413409 loss_iou: 4.251762 loss_iou_aware: 0.641516 loss_obj: 14273.924805 loss_cls: 0.703213 loss: 14283.142578 eta: 3 days, 21:57:12 batch_cost: 14.0870 data_cost: 13.2695 ips: 0.7099 images/s
[07/01 12:17:13] ppdet.engine INFO: Epoch: [0] [ 10/526] learning_rate: 0.000000 loss_xy: 1.104466 loss_wh: 2.132365 loss_iou: 3.897967 loss_iou_aware: 0.656381 loss_obj: 12482.313477 loss_cls: 0.641516 loss: 12490.728516 eta: 3 days, 19:22:27 batch_cost: 12.0913 data_cost: 11.2510 ips: 0.8270 images/s
[07/01 12:19:00] ppdet.engine INFO: Epoch: [0] [ 15/526] learning_rate: 0.000000 loss_xy: 0.915604 loss_wh: 2.491956 loss_iou: 4.159617 loss_iou_aware: 0.622869 loss_obj: 10716.389648 loss_cls: 0.736629 loss: 10724.900391 eta: 4 days, 15:27:38 batch_cost: 21.3232 data_cost: 20.4098 ips: 0.4690 images/s
[07/01 12:20:46] ppdet.engine INFO: Epoch: [0] [ 20/526] learning_rate: 0.000001 loss_xy: 1.121335 loss_wh: 2.470541 loss_iou: 4.441169 loss_iou_aware: 0.629464 loss_obj: 4258.397461 loss_cls: 0.621566 loss: 4267.644531 eta: 5 days, 1:50:17 batch_cost: 21.2485 data_cost: 20.3849 ips: 0.4706 images/s
[07/01 12:22:31] ppdet.engine INFO: Epoch: [0] [ 25/526] learning_rate: 0.000001 loss_xy: 1.092964 loss_wh: 2.442391 loss_iou: 4.365379 loss_iou_aware: 0.611063 loss_obj: 1514.855469 loss_cls: 0.755561 loss: 1524.373047 eta: 5 days, 7:39:50 batch_cost: 20.8574 data_cost: 20.1473 ips: 0.4794 images/s
[07/01 12:24:06] ppdet.engine INFO: Epoch: [0] [ 30/526] learning_rate: 0.000001 loss_xy: 1.245981 loss_wh: 2.342381 loss_iou: 4.491904 loss_iou_aware: 0.638886 loss_obj: 1469.394043 loss_cls: 0.715071 loss: 1479.742065 eta: 5 days, 9:22:20 batch_cost: 18.9637 data_cost: 18.0708 ips: 0.5273 images/s
[07/01 12:25:42] ppdet.engine INFO: Epoch: [0] [ 35/526] learning_rate: 0.000001 loss_xy: 1.144956 loss_wh: 2.406286 loss_iou: 4.320916 loss_iou_aware: 0.617093 loss_obj: 838.997192 loss_cls: 0.665849 loss: 847.173401 eta: 5 days, 10:47:43 batch_cost: 19.1575 data_cost: 18.4439 ips: 0.5220 images/s
[07/01 12:27:12] ppdet.engine INFO: Epoch: [0] [ 40/526] learning_rate: 0.000001 loss_xy: 1.138049 loss_wh: 2.127945 loss_iou: 4.030950 loss_iou_aware: 0.582580 loss_obj: 642.472351 loss_cls: 0.709636 loss: 650.463745 eta: 5 days, 10:44:47 batch_cost: 17.9005 data_cost: 17.1293 ips: 0.5586 images/s
[07/01 12:29:12] ppdet.engine INFO: Epoch: [0] [ 45/526] learning_rate: 0.000001 loss_xy: 0.956402 loss_wh: 2.319241 loss_iou: 4.204791 loss_iou_aware: 0.631527 loss_obj: 312.325195 loss_cls: 0.753860 loss: 320.881866 eta: 5 days, 15:31:27 batch_cost: 23.9826 data_cost: 23.3344 ips: 0.4170 images/s
[07/01 12:30:50] ppdet.engine INFO: Epoch: [0] [ 50/526] learning_rate: 0.000001 loss_xy: 1.231448 loss_wh: 2.558268 loss_iou: 4.370479 loss_iou_aware: 0.621082 loss_obj: 340.638580 loss_cls: 0.765347 loss: 349.412964 eta: 5 days, 16:12:59 batch_cost: 19.5872 data_cost: 18.8506 ips: 0.5105 images/s
[07/01 12:32:17] ppdet.engine INFO: Epoch: [0] [ 55/526] learning_rate: 0.000001 loss_xy: 1.223972 loss_wh: 2.396241 loss_iou: 4.393015 loss_iou_aware: 0.676423 loss_obj: 289.965118 loss_cls: 0.804316 loss: 299.275024 eta: 5 days, 15:18:19 batch_cost: 17.3212 data_cost: 16.6357 ips: 0.5773 images/s
[07/01 12:33:49] ppdet.engine INFO: Epoch: [0] [ 60/526] learning_rate: 0.000002 loss_xy: 1.095244 loss_wh: 2.720826 loss_iou: 4.326620 loss_iou_aware: 0.667595 loss_obj: 378.453705 loss_cls: 0.702839 loss: 388.153412 eta: 5 days, 15:10:41 batch_cost: 18.3898 data_cost: 17.5443 ips: 0.5438 images/s
[07/01 12:35:23] ppdet.engine INFO: Epoch: [0] [ 65/526] learning_rate: 0.000002 loss_xy: 0.988449 loss_wh: 2.104736 loss_iou: 4.186148 loss_iou_aware: 0.600449 loss_obj: 215.139709 loss_cls: 0.613115 loss: 224.098312 eta: 5 days, 15:14:21 batch_cost: 18.7034 data_cost: 18.0916 ips: 0.5347 images/s
[07/01 12:36:55] ppdet.engine INFO: Epoch: [0] [ 70/526] learning_rate: 0.000002 loss_xy: 1.078253 loss_wh: 2.615308 loss_iou: 4.237937 loss_iou_aware: 0.630623 loss_obj: 320.503784 loss_cls: 0.783672 loss: 331.229492 eta: 5 days, 15:09:30 batch_cost: 18.4503 data_cost: 17.5756 ips: 0.5420 images/s
[07/01 12:38:19] ppdet.engine INFO: Epoch: [0] [ 75/526] learning_rate: 0.000002 loss_xy: 1.161517 loss_wh: 2.456941 loss_iou: 4.451501 loss_iou_aware: 0.640539 loss_obj: 285.971924 loss_cls: 0.775499 loss: 295.907166 eta: 5 days, 14:15:29 batch_cost: 16.7253 data_cost: 15.8633 ips: 0.5979 images/s
[07/01 12:39:48] ppdet.engine INFO: Epoch: [0] [ 80/526] learning_rate: 0.000002 loss_xy: 1.221983 loss_wh: 2.509572 loss_iou: 4.425272 loss_iou_aware: 0.666793 loss_obj: 224.867050 loss_cls: 0.701793 loss: 234.744431 eta: 5 days, 13:55:23 batch_cost: 17.7417 data_cost: 16.9488 ips: 0.5636 images/s
[07/01 12:41:11] ppdet.engine INFO: Epoch: [0] [ 85/526] learning_rate: 0.000002 loss_xy: 0.996946 loss_wh: 2.555164 loss_iou: 4.233212 loss_iou_aware: 0.582177 loss_obj: 128.914963 loss_cls: 0.564532 loss: 137.997147 eta: 5 days, 13:06:02 batch_cost: 16.5054 data_cost: 15.9178 ips: 0.6059 images/s
[07/01 12:42:29] ppdet.engine INFO: Epoch: [0] [ 90/526] learning_rate: 0.000002 loss_xy: 1.281924 loss_wh: 2.155412 loss_iou: 4.224259 loss_iou_aware: 0.642223 loss_obj: 201.118118 loss_cls: 0.733224 loss: 211.086273 eta: 5 days, 12:02:38 batch_cost: 15.7005 data_cost: 14.7922 ips: 0.6369 images/s
[07/01 12:43:47] ppdet.engine INFO: Epoch: [0] [ 95/526] learning_rate: 0.000002 loss_xy: 1.254092 loss_wh: 2.572512 loss_iou: 4.429868 loss_iou_aware: 0.659593 loss_obj: 152.828247 loss_cls: 0.804853 loss: 162.739441 eta: 5 days, 10:59:13 batch_cost: 15.4150 data_cost: 14.5565 ips: 0.6487 images/s
[07/01 12:45:14] ppdet.engine INFO: Epoch: [0] [100/526] learning_rate: 0.000003 loss_xy: 0.996635 loss_wh: 2.301072 loss_iou: 4.120408 loss_iou_aware: 0.593843 loss_obj: 99.766457 loss_cls: 0.643620 loss: 108.784454 eta: 5 days, 10:45:15 batch_cost: 17.4179 data_cost: 16.6638 ips: 0.5741 images/s
[07/01 12:46:34] ppdet.engine INFO: Epoch: [0] [105/526] learning_rate: 0.000003 loss_xy: 1.063770 loss_wh: 2.539247 loss_iou: 4.360949 loss_iou_aware: 0.655446 loss_obj: 118.341606 loss_cls: 0.721550 loss: 127.873985 eta: 5 days, 10:03:09 batch_cost: 15.9947 data_cost: 15.0714 ips: 0.6252 images/s
[07/01 12:47:56] ppdet.engine INFO: Epoch: [0] [110/526] learning_rate: 0.000003 loss_xy: 1.177744 loss_wh: 2.230473 loss_iou: 4.356344 loss_iou_aware: 0.686642 loss_obj: 91.067383 loss_cls: 0.780575 loss: 100.362160 eta: 5 days, 9:29:54 batch_cost: 16.2577 data_cost: 15.5967 ips: 0.6151 images/s
[07/01 12:49:13] ppdet.engine INFO: Epoch: [0] [115/526] learning_rate: 0.000003 loss_xy: 1.158705 loss_wh: 2.281775 loss_iou: 4.029062 loss_iou_aware: 0.637969 loss_obj: 125.738983 loss_cls: 0.792050 loss: 135.915512 eta: 5 days, 8:44:32 batch_cost: 15.4675 data_cost: 14.5713 ips: 0.6465 images/s
[07/01 12:50:27] ppdet.engine INFO: Epoch: [0] [120/526] learning_rate: 0.000003 loss_xy: 0.991264 loss_wh: 1.821931 loss_iou: 3.765229 loss_iou_aware: 0.558607 loss_obj: 68.910866 loss_cls: 0.650912 loss: 75.308769 eta: 5 days, 7:48:29 batch_cost: 14.6733 data_cost: 13.8942 ips: 0.6815 images/s
[07/01 12:51:40] ppdet.engine INFO: Epoch: [0] [125/526] learning_rate: 0.000003 loss_xy: 1.258148 loss_wh: 2.423700 loss_iou: 4.441479 loss_iou_aware: 0.638279 loss_obj: 73.731422 loss_cls: 0.759794 loss: 83.374672 eta: 5 days, 6:55:32 batch_cost: 14.6012 data_cost: 13.7802 ips: 0.6849 images/s
[07/01 12:52:56] ppdet.engine INFO: Epoch: [0] [130/526] learning_rate: 0.000003 loss_xy: 1.090885 loss_wh: 2.375309 loss_iou: 4.019869 loss_iou_aware: 0.594898 loss_obj: 84.786713 loss_cls: 0.664097 loss: 93.494270 eta: 5 days, 6:15:10 batch_cost: 15.1191 data_cost: 14.2524 ips: 0.6614 images/s
[07/01 12:54:03] ppdet.engine INFO: Epoch: [0] [135/526] learning_rate: 0.000003 loss_xy: 1.244282 loss_wh: 2.057654 loss_iou: 4.130521 loss_iou_aware: 0.716851 loss_obj: 83.640106 loss_cls: 0.823826 loss: 92.512581 eta: 5 days, 5:11:28 batch_cost: 13.4848 data_cost: 12.4651 ips: 0.7416 images/s
[07/01 12:55:10] ppdet.engine INFO: Epoch: [0] [140/526] learning_rate: 0.000004 loss_xy: 1.110622 loss_wh: 2.304725 loss_iou: 4.162155 loss_iou_aware: 0.617140 loss_obj: 50.154106 loss_cls: 0.641885 loss: 59.076149 eta: 5 days, 4:07:40 batch_cost: 13.1908 data_cost: 12.4731 ips: 0.7581 images/s
[07/01 12:56:15] ppdet.engine INFO: Epoch: [0] [145/526] learning_rate: 0.000004 loss_xy: 1.159231 loss_wh: 2.232166 loss_iou: 4.176077 loss_iou_aware: 0.683062 loss_obj: 71.293312 loss_cls: 0.781162 loss: 79.995094 eta: 5 days, 3:06:57 batch_cost: 13.1099 data_cost: 12.0606 ips: 0.7628 images/s
[07/01 12:57:24] ppdet.engine INFO: Epoch: [0] [150/526] learning_rate: 0.000004 loss_xy: 1.132923 loss_wh: 2.046457 loss_iou: 4.106165 loss_iou_aware: 0.610845 loss_obj: 69.038246 loss_cls: 0.639172 loss: 76.659317 eta: 5 days, 2:17:18 batch_cost: 13.6038 data_cost: 12.6718 ips: 0.7351 images/s
[07/01 12:58:29] ppdet.engine INFO: Epoch: [0] [155/526] learning_rate: 0.000004 loss_xy: 1.089540 loss_wh: 2.326177 loss_iou: 4.250847 loss_iou_aware: 0.608413 loss_obj: 29.434540 loss_cls: 0.747675 loss: 40.095325 eta: 5 days, 1:21:58 batch_cost: 12.9737 data_cost: 12.4263 ips: 0.7708 images/s
[07/01 12:59:36] ppdet.engine INFO: Epoch: [0] [160/526] learning_rate: 0.000004 loss_xy: 0.864582 loss_wh: 2.029992 loss_iou: 3.836189 loss_iou_aware: 0.579900 loss_obj: 27.774757 loss_cls: 0.591898 loss: 35.567009 eta: 5 days, 0:37:18 batch_cost: 13.5130 data_cost: 12.8749 ips: 0.7400 images/s
[07/01 13:01:02] ppdet.engine INFO: Epoch: [0] [165/526] learning_rate: 0.000004 loss_xy: 1.316992 loss_wh: 2.169061 loss_iou: 4.145683 loss_iou_aware: 0.624901 loss_obj: 39.394150 loss_cls: 0.706548 loss: 47.988468 eta: 5 days, 0:41:15 batch_cost: 17.0185 data_cost: 16.2318 ips: 0.5876 images/s
[07/01 13:02:31] ppdet.engine INFO: Epoch: [0] [170/526] learning_rate: 0.000004 loss_xy: 1.174806 loss_wh: 2.012739 loss_iou: 4.157804 loss_iou_aware: 0.668963 loss_obj: 44.314423 loss_cls: 0.718145 loss: 52.917553 eta: 5 days, 0:54:20 batch_cost: 17.7605 data_cost: 16.8871 ips: 0.5630 images/s
[07/01 13:03:45] ppdet.engine INFO: Epoch: [0] [175/526] learning_rate: 0.000004 loss_xy: 1.247828 loss_wh: 2.023144 loss_iou: 4.336152 loss_iou_aware: 0.656447 loss_obj: 29.839693 loss_cls: 0.680164 loss: 38.585308 eta: 5 days, 0:29:42 batch_cost: 14.7787 data_cost: 14.0798 ips: 0.6767 images/s
[07/01 13:04:51] ppdet.engine INFO: Epoch: [0] [180/526] learning_rate: 0.000005 loss_xy: 1.149554 loss_wh: 2.169506 loss_iou: 3.785664 loss_iou_aware: 0.601007 loss_obj: 34.015617 loss_cls: 0.713665 loss: 43.458797 eta: 4 days, 23:47:12 batch_cost: 13.1845 data_cost: 12.5146 ips: 0.7585 images/s
[07/01 13:05:52] ppdet.engine INFO: Epoch: [0] [185/526] learning_rate: 0.000005 loss_xy: 1.223797 loss_wh: 2.178720 loss_iou: 4.150827 loss_iou_aware: 0.598653 loss_obj: 35.078701 loss_cls: 0.671679 loss: 43.485683 eta: 4 days, 22:55:08 batch_cost: 12.1778 data_cost: 11.4247 ips: 0.8212 images/s
[07/01 13:07:03] ppdet.engine INFO: Epoch: [0] [190/526] learning_rate: 0.000005 loss_xy: 1.295334 loss_wh: 1.855066 loss_iou: 4.324129 loss_iou_aware: 0.661017 loss_obj: 34.514874 loss_cls: 0.631759 loss: 42.388554 eta: 4 days, 22:28:53 batch_cost: 14.2077 data_cost: 13.4654 ips: 0.7038 images/s
[07/01 13:08:19] ppdet.engine INFO: Epoch: [0] [195/526] learning_rate: 0.000005 loss_xy: 1.134431 loss_wh: 2.360353 loss_iou: 4.451344 loss_iou_aware: 0.638183 loss_obj: 24.914568 loss_cls: 0.690286 loss: 33.923691 eta: 4 days, 22:14:03 batch_cost: 15.1232 data_cost: 14.4812 ips: 0.6612 images/s
[07/01 13:09:29] ppdet.engine INFO: Epoch: [0] [200/526] learning_rate: 0.000005 loss_xy: 1.286161 loss_wh: 1.787248 loss_iou: 4.156811 loss_iou_aware: 0.640577 loss_obj: 24.339710 loss_cls: 0.664834 loss: 32.167515 eta: 4 days, 21:47:40 batch_cost: 13.9926 data_cost: 13.3090 ips: 0.7147 images/s
[07/01 13:10:36] ppdet.engine INFO: Epoch: [0] [205/526] learning_rate: 0.000005 loss_xy: 1.184417 loss_wh: 2.029136 loss_iou: 3.975405 loss_iou_aware: 0.668309 loss_obj: 36.605125 loss_cls: 0.602948 loss: 45.886314 eta: 4 days, 21:15:23 batch_cost: 13.3168 data_cost: 12.5310 ips: 0.7509 images/s
[07/01 13:12:02] ppdet.engine INFO: Epoch: [0] [210/526] learning_rate: 0.000005 loss_xy: 1.081506 loss_wh: 1.724030 loss_iou: 3.893903 loss_iou_aware: 0.739715 loss_obj: 34.875927 loss_cls: 0.555110 loss: 42.719929 eta: 4 days, 21:23:04 batch_cost: 17.0525 data_cost: 16.1985 ips: 0.5864 images/s
[07/01 13:13:15] ppdet.engine INFO: Epoch: [0] [215/526] learning_rate: 0.000005 loss_xy: 1.145555 loss_wh: 1.895893 loss_iou: 4.057085 loss_iou_aware: 0.727120 loss_obj: 34.658497 loss_cls: 0.718911 loss: 43.180389 eta: 4 days, 21:05:31 batch_cost: 14.5880 data_cost: 13.6649 ips: 0.6855 images/s
[07/01 13:14:21] ppdet.engine INFO: Epoch: [0] [220/526] learning_rate: 0.000006 loss_xy: 1.106895 loss_wh: 1.921716 loss_iou: 4.197306 loss_iou_aware: 0.623526 loss_obj: 23.560707 loss_cls: 0.628223 loss: 32.653717 eta: 4 days, 20:34:38 batch_cost: 13.1562 data_cost: 12.4561 ips: 0.7601 images/s
[07/01 13:15:31] ppdet.engine INFO: Epoch: [0] [225/526] learning_rate: 0.000006 loss_xy: 1.197099 loss_wh: 1.876569 loss_iou: 3.978829 loss_iou_aware: 0.706580 loss_obj: 27.415630 loss_cls: 0.699091 loss: 35.734127 eta: 4 days, 20:13:23 batch_cost: 14.0210 data_cost: 13.2424 ips: 0.7132 images/s
[07/01 13:16:45] ppdet.engine INFO: Epoch: [0] [230/526] learning_rate: 0.000006 loss_xy: 1.057469 loss_wh: 1.776332 loss_iou: 3.924106 loss_iou_aware: 0.649162 loss_obj: 35.021446 loss_cls: 0.760346 loss: 43.127289 eta: 4 days, 20:00:20 batch_cost: 14.8005 data_cost: 13.8679 ips: 0.6757 images/s
[07/01 13:18:00] ppdet.engine INFO: Epoch: [0] [235/526] learning_rate: 0.000006 loss_xy: 1.198717 loss_wh: 1.827318 loss_iou: 4.267738 loss_iou_aware: 0.699909 loss_obj: 23.384451 loss_cls: 0.753166 loss: 31.450428 eta: 4 days, 19:49:35 batch_cost: 14.9973 data_cost: 14.1241 ips: 0.6668 images/s
[07/01 13:19:03] ppdet.engine INFO: Epoch: [0] [240/526] learning_rate: 0.000006 loss_xy: 1.282138 loss_wh: 1.899866 loss_iou: 4.010087 loss_iou_aware: 0.672560 loss_obj: 32.506649 loss_cls: 0.627504 loss: 41.963383 eta: 4 days, 19:16:41 batch_cost: 12.4951 data_cost: 11.6089 ips: 0.8003 images/s
[07/01 13:20:06] ppdet.engine INFO: Epoch: [0] [245/526] learning_rate: 0.000006 loss_xy: 1.028864 loss_wh: 1.826583 loss_iou: 3.988883 loss_iou_aware: 0.664884 loss_obj: 18.929489 loss_cls: 0.670509 loss: 27.673689 eta: 4 days, 18:45:00 batch_cost: 12.4847 data_cost: 11.7054 ips: 0.8010 images/s
[07/01 13:21:25] ppdet.engine INFO: Epoch: [0] [250/526] learning_rate: 0.000006 loss_xy: 1.235800 loss_wh: 1.777964 loss_iou: 4.063594 loss_iou_aware: 0.667364 loss_obj: 18.668900 loss_cls: 0.572790 loss: 26.589201 eta: 4 days, 18:43:32 batch_cost: 15.8379 data_cost: 15.0809 ips: 0.6314 images/s
[07/01 13:22:27] ppdet.engine INFO: Epoch: [0] [255/526] learning_rate: 0.000006 loss_xy: 1.208367 loss_wh: 2.075184 loss_iou: 4.206319 loss_iou_aware: 0.669621 loss_obj: 17.611246 loss_cls: 0.677594 loss: 26.703337 eta: 4 days, 18:12:12 batch_cost: 12.3145 data_cost: 11.6690 ips: 0.8120 images/s
[07/01 13:23:41] ppdet.engine INFO: Epoch: [0] [260/526] learning_rate: 0.000007 loss_xy: 1.085346 loss_wh: 1.668180 loss_iou: 4.177268 loss_iou_aware: 0.697426 loss_obj: 30.483238 loss_cls: 0.630117 loss: 38.814762 eta: 4 days, 18:02:18 batch_cost: 14.7525 data_cost: 13.7259 ips: 0.6779 images/s
[07/01 13:24:49] ppdet.engine INFO: Epoch: [0] [265/526] learning_rate: 0.000007 loss_xy: 1.088767 loss_wh: 1.783628 loss_iou: 4.164545 loss_iou_aware: 0.736160 loss_obj: 22.592388 loss_cls: 0.655318 loss: 31.026768 eta: 4 days, 17:43:42 batch_cost: 13.6480 data_cost: 12.8042 ips: 0.7327 images/s
[07/01 13:25:54] ppdet.engine INFO: Epoch: [0] [270/526] learning_rate: 0.000007 loss_xy: 1.029153 loss_wh: 1.677392 loss_iou: 3.650034 loss_iou_aware: 0.585184 loss_obj: 13.997475 loss_cls: 0.601826 loss: 21.570038 eta: 4 days, 17:19:13 batch_cost: 12.8302 data_cost: 12.1707 ips: 0.7794 images/s
[07/01 13:26:59] ppdet.engine INFO: Epoch: [0] [275/526] learning_rate: 0.000007 loss_xy: 1.096723 loss_wh: 1.582747 loss_iou: 3.985842 loss_iou_aware: 0.640106 loss_obj: 15.957102 loss_cls: 0.634033 loss: 23.852882 eta: 4 days, 16:56:51 batch_cost: 12.9921 data_cost: 12.3699 ips: 0.7697 images/s
[07/01 13:28:10] ppdet.engine INFO: Epoch: [0] [280/526] learning_rate: 0.000007 loss_xy: 1.029367 loss_wh: 1.631019 loss_iou: 4.059268 loss_iou_aware: 0.738247 loss_obj: 19.005415 loss_cls: 0.754441 loss: 28.284607 eta: 4 days, 16:43:40 batch_cost: 14.0849 data_cost: 13.1892 ips: 0.7100 images/s
[07/01 13:29:24] ppdet.engine INFO: Epoch: [0] [285/526] learning_rate: 0.000007 loss_xy: 0.972801 loss_wh: 1.615446 loss_iou: 4.031400 loss_iou_aware: 0.711254 loss_obj: 19.575111 loss_cls: 0.687294 loss: 27.518564 eta: 4 days, 16:37:11 batch_cost: 14.9121 data_cost: 13.9654 ips: 0.6706 images/s
[07/01 13:30:36] ppdet.engine INFO: Epoch: [0] [290/526] learning_rate: 0.000007 loss_xy: 1.141120 loss_wh: 1.821583 loss_iou: 3.844513 loss_iou_aware: 0.712960 loss_obj: 14.678379 loss_cls: 0.594878 loss: 22.830244 eta: 4 days, 16:25:48 batch_cost: 14.2301 data_cost: 13.5185 ips: 0.7027 images/s
[07/01 13:32:01] ppdet.engine INFO: Epoch: [0] [295/526] learning_rate: 0.000007 loss_xy: 1.059708 loss_wh: 1.558759 loss_iou: 3.763732 loss_iou_aware: 0.694737 loss_obj: 19.689251 loss_cls: 0.740966 loss: 27.290422 eta: 4 days, 16:35:50 batch_cost: 17.1097 data_cost: 16.2212 ips: 0.5845 images/s
[07/01 13:33:07] ppdet.engine INFO: Epoch: [0] [300/526] learning_rate: 0.000008 loss_xy: 1.199921 loss_wh: 1.615327 loss_iou: 4.127220 loss_iou_aware: 0.639887 loss_obj: 13.675007 loss_cls: 0.637481 loss: 22.013250 eta: 4 days, 16:16:50 batch_cost: 13.1272 data_cost: 12.5675 ips: 0.7618 images/s
[07/01 13:34:16] ppdet.engine INFO: Epoch: [0] [305/526] learning_rate: 0.000008 loss_xy: 1.050035 loss_wh: 1.681755 loss_iou: 3.872050 loss_iou_aware: 0.572892 loss_obj: 18.434284 loss_cls: 0.667585 loss: 25.835394 eta: 4 days, 16:01:59 batch_cost: 13.6314 data_cost: 12.7587 ips: 0.7336 images/s
[07/01 13:35:26] ppdet.engine INFO: Epoch: [0] [310/526] learning_rate: 0.000008 loss_xy: 0.997568 loss_wh: 1.509685 loss_iou: 3.722756 loss_iou_aware: 0.584143 loss_obj: 15.210144 loss_cls: 0.677971 loss: 22.759336 eta: 4 days, 15:50:47 batch_cost: 14.0941 data_cost: 13.3666 ips: 0.7095 images/s
[07/01 13:36:30] ppdet.engine INFO: Epoch: [0] [315/526] learning_rate: 0.000008 loss_xy: 1.045465 loss_wh: 1.656418 loss_iou: 3.877389 loss_iou_aware: 0.716924 loss_obj: 16.369495 loss_cls: 0.655378 loss: 24.386274 eta: 4 days, 15:30:22 batch_cost: 12.7003 data_cost: 12.0041 ips: 0.7874 images/s
[07/01 13:37:37] ppdet.engine INFO: Epoch: [0] [320/526] learning_rate: 0.000008 loss_xy: 1.154737 loss_wh: 1.728677 loss_iou: 4.048987 loss_iou_aware: 0.679962 loss_obj: 21.279150 loss_cls: 0.690357 loss: 29.280508 eta: 4 days, 15:15:21 batch_cost: 13.4124 data_cost: 12.4195 ips: 0.7456 images/s
[07/01 13:38:38] ppdet.engine INFO: Epoch: [0] [325/526] learning_rate: 0.000008 loss_xy: 1.082633 loss_wh: 1.326666 loss_iou: 3.693600 loss_iou_aware: 0.665069 loss_obj: 13.832065 loss_cls: 0.617947 loss: 21.107273 eta: 4 days, 14:52:22 batch_cost: 12.1507 data_cost: 11.4775 ips: 0.8230 images/s
[07/01 13:40:06] ppdet.engine INFO: Epoch: [0] [330/526] learning_rate: 0.000008 loss_xy: 1.102530 loss_wh: 1.402024 loss_iou: 3.685415 loss_iou_aware: 0.695506 loss_obj: 14.423463 loss_cls: 0.626651 loss: 22.490555 eta: 4 days, 15:04:41 batch_cost: 17.4448 data_cost: 16.7416 ips: 0.5732 images/s
[07/01 13:41:16] ppdet.engine INFO: Epoch: [0] [335/526] learning_rate: 0.000008 loss_xy: 1.136834 loss_wh: 1.368255 loss_iou: 3.694318 loss_iou_aware: 0.656776 loss_obj: 12.866175 loss_cls: 0.684606 loss: 21.235811 eta: 4 days, 14:54:13 batch_cost: 13.9724 data_cost: 13.1803 ips: 0.7157 images/s
[07/01 13:42:26] ppdet.engine INFO: Epoch: [0] [340/526] learning_rate: 0.000009 loss_xy: 1.002786 loss_wh: 1.762086 loss_iou: 3.859673 loss_iou_aware: 0.691230 loss_obj: 16.758720 loss_cls: 0.673387 loss: 25.358444 eta: 4 days, 14:43:41 batch_cost: 13.9185 data_cost: 13.0802 ips: 0.7185 images/s
[07/01 13:43:29] ppdet.engine INFO: Epoch: [0] [345/526] learning_rate: 0.000009 loss_xy: 1.104222 loss_wh: 1.643591 loss_iou: 4.099178 loss_iou_aware: 0.664676 loss_obj: 18.918755 loss_cls: 0.557912 loss: 26.811903 eta: 4 days, 14:25:41 batch_cost: 12.6807 data_cost: 11.7806 ips: 0.7886 images/s
[07/01 13:44:36] ppdet.engine INFO: Epoch: [0] [350/526] learning_rate: 0.000009 loss_xy: 1.141841 loss_wh: 1.622595 loss_iou: 3.870060 loss_iou_aware: 0.679749 loss_obj: 17.151106 loss_cls: 0.593945 loss: 25.032406 eta: 4 days, 14:11:32 batch_cost: 13.2250 data_cost: 12.3910 ips: 0.7561 images/s
[07/01 13:45:37] ppdet.engine INFO: Epoch: [0] [355/526] learning_rate: 0.000009 loss_xy: 1.052077 loss_wh: 1.664710 loss_iou: 3.881771 loss_iou_aware: 0.661104 loss_obj: 14.782454 loss_cls: 0.469873 loss: 22.958609 eta: 4 days, 13:51:20 batch_cost: 12.1702 data_cost: 11.4402 ips: 0.8217 images/s
[07/01 13:46:43] ppdet.engine INFO: Epoch: [0] [360/526] learning_rate: 0.000009 loss_xy: 1.093885 loss_wh: 1.406798 loss_iou: 3.811672 loss_iou_aware: 0.694201 loss_obj: 14.363710 loss_cls: 0.596051 loss: 22.274965 eta: 4 days, 13:37:47 batch_cost: 13.1939 data_cost: 12.4711 ips: 0.7579 images/s
[07/01 13:47:51] ppdet.engine INFO: Epoch: [0] [365/526] learning_rate: 0.000009 loss_xy: 1.124093 loss_wh: 1.381487 loss_iou: 3.874315 loss_iou_aware: 0.699272 loss_obj: 17.813011 loss_cls: 0.612456 loss: 25.545204 eta: 4 days, 13:26:30 batch_cost: 13.5193 data_cost: 12.4458 ips: 0.7397 images/s
[07/01 13:49:03] ppdet.engine INFO: Epoch: [0] [370/526] learning_rate: 0.000009 loss_xy: 1.188190 loss_wh: 1.624373 loss_iou: 3.864948 loss_iou_aware: 0.702230 loss_obj: 14.380863 loss_cls: 0.686754 loss: 22.397690 eta: 4 days, 13:21:04 batch_cost: 14.4740 data_cost: 13.5806 ips: 0.6909 images/s
[07/01 13:50:05] ppdet.engine INFO: Epoch: [0] [375/526] learning_rate: 0.000009 loss_xy: 1.113057 loss_wh: 1.414446 loss_iou: 3.888521 loss_iou_aware: 0.698814 loss_obj: 11.986423 loss_cls: 0.579011 loss: 18.496145 eta: 4 days, 13:03:37 batch_cost: 12.3672 data_cost: 11.5277 ips: 0.8086 images/s
[07/01 13:51:29] ppdet.engine INFO: Epoch: [0] [380/526] learning_rate: 0.000010 loss_xy: 0.991184 loss_wh: 1.621124 loss_iou: 3.902881 loss_iou_aware: 0.680126 loss_obj: 11.574911 loss_cls: 0.570284 loss: 20.008980 eta: 4 days, 13:11:17 batch_cost: 16.7181 data_cost: 16.0548 ips: 0.5982 images/s
[07/01 13:52:33] ppdet.engine INFO: Epoch: [0] [385/526] learning_rate: 0.000010 loss_xy: 1.156179 loss_wh: 1.454666 loss_iou: 3.837860 loss_iou_aware: 0.697370 loss_obj: 15.354259 loss_cls: 0.557391 loss: 22.213127 eta: 4 days, 12:56:43 batch_cost: 12.7873 data_cost: 11.8790 ips: 0.7820 images/s
[07/01 13:53:37] ppdet.engine INFO: Epoch: [0] [390/526] learning_rate: 0.000010 loss_xy: 1.044631 loss_wh: 1.496924 loss_iou: 3.769402 loss_iou_aware: 0.666927 loss_obj: 11.900872 loss_cls: 0.625725 loss: 18.992374 eta: 4 days, 12:42:10 batch_cost: 12.7279 data_cost: 12.0840 ips: 0.7857 images/s
[07/01 13:54:42] ppdet.engine INFO: Epoch: [0] [395/526] learning_rate: 0.000010 loss_xy: 1.177679 loss_wh: 1.506815 loss_iou: 4.003256 loss_iou_aware: 0.688290 loss_obj: 13.409214 loss_cls: 0.672994 loss: 21.275620 eta: 4 days, 12:28:31 batch_cost: 12.8296 data_cost: 11.9984 ips: 0.7794 images/s
[07/01 13:55:45] ppdet.engine INFO: Epoch: [0] [400/526] learning_rate: 0.000010 loss_xy: 1.173628 loss_wh: 1.515467 loss_iou: 4.000115 loss_iou_aware: 0.690052 loss_obj: 14.598661 loss_cls: 0.626922 loss: 22.702097 eta: 4 days, 12:14:20 batch_cost: 12.6730 data_cost: 11.6656 ips: 0.7891 images/s
[07/01 13:56:43] ppdet.engine INFO: Epoch: [0] [405/526] learning_rate: 0.000010 loss_xy: 1.100368 loss_wh: 1.575760 loss_iou: 4.194187 loss_iou_aware: 0.671972 loss_obj: 9.648710 loss_cls: 0.603631 loss: 17.500242 eta: 4 days, 11:54:35 batch_cost: 11.5648 data_cost: 11.0840 ips: 0.8647 images/s
[07/01 13:57:44] ppdet.engine INFO: Epoch: [0] [410/526] learning_rate: 0.000010 loss_xy: 1.195703 loss_wh: 1.437436 loss_iou: 3.927448 loss_iou_aware: 0.744302 loss_obj: 14.254375 loss_cls: 0.612301 loss: 21.800064 eta: 4 days, 11:37:59 batch_cost: 12.0772 data_cost: 11.2205 ips: 0.8280 images/s
[07/01 13:58:52] ppdet.engine INFO: Epoch: [0] [415/526] learning_rate: 0.000010 loss_xy: 1.065183 loss_wh: 1.467490 loss_iou: 4.010128 loss_iou_aware: 0.646286 loss_obj: 10.328963 loss_cls: 0.642096 loss: 18.596519 eta: 4 days, 11:28:58 batch_cost: 13.4697 data_cost: 12.7611 ips: 0.7424 images/s
[07/01 14:00:10] ppdet.engine INFO: Epoch: [0] [420/526] learning_rate: 0.000010 loss_xy: 1.073382 loss_wh: 1.399643 loss_iou: 3.956860 loss_iou_aware: 0.691939 loss_obj: 14.661393 loss_cls: 0.647687 loss: 22.744785 eta: 4 days, 11:31:40 batch_cost: 15.7167 data_cost: 14.6602 ips: 0.6363 images/s
[07/01 14:01:11] ppdet.engine INFO: Epoch: [0] [425/526] learning_rate: 0.000011 loss_xy: 1.110812 loss_wh: 1.475729 loss_iou: 3.955990 loss_iou_aware: 0.702570 loss_obj: 12.199990 loss_cls: 0.642899 loss: 19.389687 eta: 4 days, 11:16:09 batch_cost: 12.1384 data_cost: 11.2846 ips: 0.8238 images/s
[07/01 14:02:20] ppdet.engine INFO: Epoch: [0] [430/526] learning_rate: 0.000011 loss_xy: 1.133572 loss_wh: 1.311864 loss_iou: 3.688827 loss_iou_aware: 0.692649 loss_obj: 10.177580 loss_cls: 0.530342 loss: 17.044258 eta: 4 days, 11:08:35 batch_cost: 13.6620 data_cost: 12.9042 ips: 0.7320 images/s
[07/01 14:03:34] ppdet.engine INFO: Epoch: [0] [435/526] learning_rate: 0.000011 loss_xy: 1.053666 loss_wh: 1.387102 loss_iou: 3.845040 loss_iou_aware: 0.705124 loss_obj: 12.000084 loss_cls: 0.618547 loss: 19.661043 eta: 4 days, 11:07:13 batch_cost: 14.8847 data_cost: 14.0828 ips: 0.6718 images/s
[07/01 14:04:40] ppdet.engine INFO: Epoch: [0] [440/526] learning_rate: 0.000011 loss_xy: 1.012944 loss_wh: 1.303181 loss_iou: 3.869767 loss_iou_aware: 0.657314 loss_obj: 10.116856 loss_cls: 0.617109 loss: 16.418688 eta: 4 days, 10:56:55 batch_cost: 13.0560 data_cost: 12.3919 ips: 0.7659 images/s
[07/01 14:05:46] ppdet.engine INFO: Epoch: [0] [445/526] learning_rate: 0.000011 loss_xy: 1.020963 loss_wh: 1.274895 loss_iou: 3.828473 loss_iou_aware: 0.703915 loss_obj: 10.813742 loss_cls: 0.580715 loss: 18.155323 eta: 4 days, 10:47:02 batch_cost: 13.0975 data_cost: 12.2879 ips: 0.7635 images/s
[07/01 14:06:46] ppdet.engine INFO: Epoch: [0] [450/526] learning_rate: 0.000011 loss_xy: 1.147226 loss_wh: 1.537148 loss_iou: 3.797288 loss_iou_aware: 0.686343 loss_obj: 11.856258 loss_cls: 0.604958 loss: 19.863937 eta: 4 days, 10:32:20 batch_cost: 12.0528 data_cost: 11.1376 ips: 0.8297 images/s
[07/01 14:07:56] ppdet.engine INFO: Epoch: [0] [455/526] learning_rate: 0.000011 loss_xy: 1.034053 loss_wh: 1.613674 loss_iou: 3.838971 loss_iou_aware: 0.658241 loss_obj: 9.643236 loss_cls: 0.544299 loss: 17.847387 eta: 4 days, 10:27:15 batch_cost: 14.0231 data_cost: 13.3004 ips: 0.7131 images/s
[07/01 14:09:07] ppdet.engine INFO: Epoch: [0] [460/526] learning_rate: 0.000012 loss_xy: 1.040160 loss_wh: 1.295907 loss_iou: 3.821574 loss_iou_aware: 0.717952 loss_obj: 13.428027 loss_cls: 0.582457 loss: 21.589560 eta: 4 days, 10:22:28 batch_cost: 14.0663 data_cost: 13.0924 ips: 0.7109 images/s
[07/01 14:10:13] ppdet.engine INFO: Epoch: [0] [465/526] learning_rate: 0.000012 loss_xy: 1.152655 loss_wh: 1.342875 loss_iou: 3.947140 loss_iou_aware: 0.691189 loss_obj: 9.812326 loss_cls: 0.619066 loss: 17.257215 eta: 4 days, 10:13:22 batch_cost: 13.1199 data_cost: 12.3495 ips: 0.7622 images/s
[07/01 14:11:20] ppdet.engine INFO: Epoch: [0] [470/526] learning_rate: 0.000012 loss_xy: 1.009960 loss_wh: 1.367424 loss_iou: 3.860856 loss_iou_aware: 0.641173 loss_obj: 8.767929 loss_cls: 0.530717 loss: 16.264814 eta: 4 days, 10:05:49 batch_cost: 13.4169 data_cost: 12.7390 ips: 0.7453 images/s
[07/01 14:12:26] ppdet.engine INFO: Epoch: [0] [475/526] learning_rate: 0.000012 loss_xy: 0.880614 loss_wh: 1.468517 loss_iou: 3.700959 loss_iou_aware: 0.720627 loss_obj: 11.092227 loss_cls: 0.627311 loss: 18.430204 eta: 4 days, 9:56:36 batch_cost: 13.0224 data_cost: 12.0082 ips: 0.7679 images/s
[07/01 14:13:27] ppdet.engine INFO: Epoch: [0] [480/526] learning_rate: 0.000012 loss_xy: 1.061072 loss_wh: 1.287019 loss_iou: 3.872210 loss_iou_aware: 0.684550 loss_obj: 8.296071 loss_cls: 0.556529 loss: 15.575493 eta: 4 days, 9:44:15 batch_cost: 12.2817 data_cost: 11.5879 ips: 0.8142 images/s
[07/01 14:14:25] ppdet.engine INFO: Epoch: [0] [485/526] learning_rate: 0.000012 loss_xy: 0.908903 loss_wh: 1.104860 loss_iou: 3.683180 loss_iou_aware: 0.719687 loss_obj: 11.253470 loss_cls: 0.624695 loss: 18.605165 eta: 4 days, 9:28:30 batch_cost: 11.4633 data_cost: 10.5807 ips: 0.8724 images/s
[07/01 14:15:29] ppdet.engine INFO: Epoch: [0] [490/526] learning_rate: 0.000012 loss_xy: 1.040438 loss_wh: 1.338577 loss_iou: 3.615209 loss_iou_aware: 0.749816 loss_obj: 10.571490 loss_cls: 0.549764 loss: 17.733517 eta: 4 days, 9:18:47 batch_cost: 12.7700 data_cost: 11.8489 ips: 0.7831 images/s
[07/01 14:16:34] ppdet.engine INFO: Epoch: [0] [495/526] learning_rate: 0.000012 loss_xy: 0.972936 loss_wh: 1.131723 loss_iou: 3.537634 loss_iou_aware: 0.662659 loss_obj: 10.013592 loss_cls: 0.520124 loss: 16.848978 eta: 4 days, 9:10:33 batch_cost: 13.0737 data_cost: 12.1537 ips: 0.7649 images/s
[07/01 14:17:38] ppdet.engine INFO: Epoch: [0] [500/526] learning_rate: 0.000013 loss_xy: 1.043951 loss_wh: 1.071104 loss_iou: 3.648298 loss_iou_aware: 0.709077 loss_obj: 8.897619 loss_cls: 0.664455 loss: 15.998529 eta: 4 days, 9:01:07 batch_cost: 12.7595 data_cost: 11.9775 ips: 0.7837 images/s
[07/01 14:18:31] ppdet.engine INFO: Epoch: [0] [505/526] learning_rate: 0.000013 loss_xy: 0.995084 loss_wh: 1.141770 loss_iou: 3.911056 loss_iou_aware: 0.675935 loss_obj: 7.697985 loss_cls: 0.607199 loss: 14.856952 eta: 4 days, 8:41:53 batch_cost: 10.4142 data_cost: 9.7294 ips: 0.9602 images/s
[07/01 14:19:31] ppdet.engine INFO: Epoch: [0] [510/526] learning_rate: 0.000013 loss_xy: 1.022732 loss_wh: 1.114027 loss_iou: 3.688476 loss_iou_aware: 0.752083 loss_obj: 8.362950 loss_cls: 0.512267 loss: 16.175888 eta: 4 days, 8:29:39 batch_cost: 11.9926 data_cost: 11.2365 ips: 0.8338 images/s
[07/01 14:20:40] ppdet.engine INFO: Epoch: [0] [515/526] learning_rate: 0.000013 loss_xy: 0.999646 loss_wh: 1.286703 loss_iou: 3.516223 loss_iou_aware: 0.709584 loss_obj: 10.304377 loss_cls: 0.590871 loss: 17.952936 eta: 4 days, 8:25:17 batch_cost: 13.8291 data_cost: 12.8221 ips: 0.7231 images/s
[07/01 14:21:42] ppdet.engine INFO: Epoch: [0] [520/526] learning_rate: 0.000013 loss_xy: 0.960961 loss_wh: 1.245511 loss_iou: 3.809340 loss_iou_aware: 0.699583 loss_obj: 6.763412 loss_cls: 0.518565 loss: 13.833389 eta: 4 days, 8:14:33 batch_cost: 12.2701 data_cost: 11.5657 ips: 0.8150 images/s
[07/01 14:22:43] ppdet.engine INFO: Epoch: [0] [525/526] learning_rate: 0.000013 loss_xy: 0.961485 loss_wh: 1.279465 loss_iou: 3.649951 loss_iou_aware: 0.640815 loss_obj: 6.676243 loss_cls: 0.543096 loss: 14.173296 eta: 4 days, 8:03:37 batch_cost: 12.1764 data_cost: 11.5398 ips: 0.8213 images/s
[07/01 14:22:50] reader WARNING: fail to map sample transform [Decode_5b4f7f] with error: [Errno 5] Input/output error and stack:
Traceback (most recent call last):
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/reader.py", line 54, in __call__
    data = f(data)
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 103, in __call__
    sample[i] = self.apply(sample[i], context)
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 123, in apply
    sample['image'] = f.read()
OSError: [Errno 5] Input/output error

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/dataloader_iter.py", line 216, in _thread_loop
    self._thread_done_event)
  File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/fetcher.py", line 121, in fetch
    data.append(self.dataset[idx])
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/source/dataset.py", line 91, in __getitem__
    return self.transform(roidb)
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/reader.py", line 60, in __call__
    raise e
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/reader.py", line 54, in __call__
    data = f(data)
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 103, in __call__
    sample[i] = self.apply(sample[i], context)
  File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 123, in apply
    sample['image'] = f.read()
OSError: [Errno 5] Input/output error

[07/01 14:23:06] ppdet.utils.checkpoint INFO: Save checkpoint: output/ppyolo_r50vd_dcn_1x_coco/ppyolo_r50vd_dcn_1x_coco

I use PASCAL VOC format to train model. Does the bug above have any effect on the model's training? Thank you for your time to support me.

复现环境 Environment

Paddlepaddle-gpu==2.3.0 PaddleDetection: Release/2.4 Python 3.7.13 CUDA Version: 11.2 cuDNN Version: 7.6.

是否愿意提交PR Are you willing to submit a PR?

nemonameless commented 2 years ago

First you can train without --eval and make sure you finish the training first. Then make sure that there is no problem in your val json of eval dataset and the images can be read by OpenCV.

thongvhoang commented 2 years ago

First you can train without --eval and make sure you finish the training first. Then make sure that there is no problem in your val json of eval dataset and the images can be read by OpenCV.

I don't think so that my val json is a problem because I train the Picodet model with my val json and it works fine.

/usr/local/lib/python3.7/dist-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
/usr/local/lib/python3.7/dist-packages/scipy/fft/__init__.py:97: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
  from numpy.dual import register_func
/usr/local/lib/python3.7/dist-packages/scipy/sparse/sputils.py:17: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.
  supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/usr/local/lib/python3.7/dist-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
loading annotations into memory...
Done (t=0.04s)
creating index...
index created!
W0704 08:17:58.320560 31923 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 6.0, Driver API Version: 11.2, Runtime API Version: 10.2
W0704 08:17:58.329401 31923 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.
[07/04 08:18:08] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/picodet_m_416_coco_lcnet/model_final.pdparams
[07/04 08:18:13] ppdet.engine INFO: Epoch: [30] [  0/526] learning_rate: 0.000234 loss_vfl: 0.429801 loss_bbox: 0.216690 loss_dfl: 0.281943 loss: 0.928434 eta: 0:36:49 batch_cost: 0.4200 data_cost: 0.0035 ips: 23.8099 images/s
[07/04 08:18:29] ppdet.engine INFO: Epoch: [30] [  5/526] learning_rate: 0.000234 loss_vfl: 0.410616 loss_bbox: 0.226801 loss_dfl: 0.272752 loss: 0.922487 eta: 4:05:52 batch_cost: 3.2847 data_cost: 3.0771 ips: 3.0444 images/s
[07/04 08:19:02] ppdet.engine INFO: Epoch: [30] [ 10/526] learning_rate: 0.000234 loss_vfl: 0.382467 loss_bbox: 0.213267 loss_dfl: 0.262357 loss: 0.862539 eta: 6:30:58 batch_cost: 6.4616 data_cost: 6.2654 ips: 1.5476 images/s
[07/04 08:19:34] ppdet.engine INFO: Epoch: [30] [ 15/526] learning_rate: 0.000234 loss_vfl: 0.362904 loss_bbox: 0.175328 loss_dfl: 0.270935 loss: 0.782068 eta: 7:19:13 batch_cost: 6.2478 data_cost: 6.0377 ips: 1.6006 images/s
[07/04 08:20:06] ppdet.engine INFO: Epoch: [30] [ 20/526] learning_rate: 0.000234 loss_vfl: 0.405658 loss_bbox: 0.220620 loss_dfl: 0.274387 loss: 0.886294 eta: 7:45:06 batch_cost: 6.2897 data_cost: 6.0679 ips: 1.5899 images/s
[07/04 08:20:40] ppdet.engine INFO: Epoch: [30] [ 25/526] learning_rate: 0.000234 loss_vfl: 0.351791 loss_bbox: 0.214690 loss_dfl: 0.261170 loss: 0.849986 eta: 8:08:41 batch_cost: 6.7575 data_cost: 6.5551 ips: 1.4798 images/s
[07/04 08:21:13] ppdet.engine INFO: Epoch: [30] [ 30/526] learning_rate: 0.000234 loss_vfl: 0.367283 loss_bbox: 0.226409 loss_dfl: 0.270896 loss: 0.848719 eta: 8:21:40 batch_cost: 6.5575 data_cost: 6.3554 ips: 1.5250 images/s
[07/04 08:21:46] ppdet.engine INFO: Epoch: [30] [ 35/526] learning_rate: 0.000234 loss_vfl: 0.385411 loss_bbox: 0.186166 loss_dfl: 0.271677 loss: 0.828303 eta: 8:30:13 batch_cost: 6.5019 data_cost: 6.3064 ips: 1.5380 images/s
[07/04 08:22:17] ppdet.engine INFO: Epoch: [30] [ 40/526] learning_rate: 0.000234 loss_vfl: 0.440741 loss_bbox: 0.191622 loss_dfl: 0.241397 loss: 0.869412 eta: 8:33:35 batch_cost: 6.2229 data_cost: 6.0136 ips: 1.6070 images/s
[07/04 08:23:10] ppdet.engine INFO: Epoch: [30] [ 45/526] learning_rate: 0.000234 loss_vfl: 0.411759 loss_bbox: 0.217600 loss_dfl: 0.274928 loss: 0.929484 eta: 9:16:17 batch_cost: 10.4747 data_cost: 10.2710 ips: 0.9547 images/s
[07/04 08:23:43] ppdet.engine INFO: Epoch: [30] [ 50/526] learning_rate: 0.000234 loss_vfl: 0.378255 loss_bbox: 0.230514 loss_dfl: 0.274903 loss: 0.904595 eta: 9:16:35 batch_cost: 6.4981 data_cost: 6.3042 ips: 1.5389 images/s
[07/04 08:24:14] ppdet.engine INFO: Epoch: [30] [ 55/526] learning_rate: 0.000234 loss_vfl: 0.375699 loss_bbox: 0.216554 loss_dfl: 0.270524 loss: 0.879218 eta: 9:13:03 batch_cost: 6.0219 data_cost: 5.8331 ips: 1.6606 images/s
[07/04 08:24:44] ppdet.engine INFO: Epoch: [30] [ 60/526] learning_rate: 0.000234 loss_vfl: 0.397382 loss_bbox: 0.232140 loss_dfl: 0.270320 loss: 0.910500 eta: 9:10:27 batch_cost: 6.0851 data_cost: 5.8873 ips: 1.6434 images/s
[07/04 08:25:17] ppdet.engine INFO: Epoch: [30] [ 65/526] learning_rate: 0.000234 loss_vfl: 0.399165 loss_bbox: 0.213154 loss_dfl: 0.267611 loss: 0.870209 eta: 9:10:20 batch_cost: 6.4152 data_cost: 6.2111 ips: 1.5588 images/s
[07/04 08:25:49] ppdet.engine INFO: Epoch: [30] [ 70/526] learning_rate: 0.000234 loss_vfl: 0.338821 loss_bbox: 0.174160 loss_dfl: 0.273611 loss: 0.795376 eta: 9:09:10 batch_cost: 6.2494 data_cost: 6.0452 ips: 1.6001 images/s
[07/04 08:26:24] ppdet.engine INFO: Epoch: [30] [ 75/526] learning_rate: 0.000234 loss_vfl: 0.369768 loss_bbox: 0.225486 loss_dfl: 0.246591 loss: 0.848217 eta: 9:12:31 batch_cost: 7.0325 data_cost: 6.8245 ips: 1.4220 images/s
[07/04 08:26:59] ppdet.engine INFO: Epoch: [30] [ 80/526] learning_rate: 0.000234 loss_vfl: 0.375087 loss_bbox: 0.192145 loss_dfl: 0.276981 loss: 0.841399 eta: 9:14:29 batch_cost: 6.8612 data_cost: 6.6697 ips: 1.4575 images/s
[07/04 08:27:29] ppdet.engine INFO: Epoch: [30] [ 85/526] learning_rate: 0.000234 loss_vfl: 0.319496 loss_bbox: 0.162922 loss_dfl: 0.256047 loss: 0.726528 eta: 9:11:46 batch_cost: 5.9876 data_cost: 5.7941 ips: 1.6701 images/s
[07/04 08:28:01] ppdet.engine INFO: Epoch: [30] [ 90/526] learning_rate: 0.000234 loss_vfl: 0.409093 loss_bbox: 0.257046 loss_dfl: 0.268151 loss: 0.940789 eta: 9:10:21 batch_cost: 6.2107 data_cost: 6.0144 ips: 1.6101 images/s
[07/04 08:28:32] ppdet.engine INFO: Epoch: [30] [ 95/526] learning_rate: 0.000234 loss_vfl: 0.351439 loss_bbox: 0.158058 loss_dfl: 0.273113 loss: 0.775364 eta: 9:08:45 batch_cost: 6.1509 data_cost: 5.9548 ips: 1.6258 images/s
[07/04 08:29:02] ppdet.engine INFO: Epoch: [30] [100/526] learning_rate: 0.000234 loss_vfl: 0.423639 loss_bbox: 0.234954 loss_dfl: 0.280912 loss: 0.953623 eta: 9:06:35 batch_cost: 5.9890 data_cost: 5.7801 ips: 1.6697 images/s
[07/04 08:29:32] ppdet.engine INFO: Epoch: [30] [105/526] learning_rate: 0.000234 loss_vfl: 0.360931 loss_bbox: 0.206550 loss_dfl: 0.269648 loss: 0.815192 eta: 9:04:16 batch_cost: 5.9148 data_cost: 5.7105 ips: 1.6907 images/s
[07/04 08:30:05] ppdet.engine INFO: Epoch: [30] [110/526] learning_rate: 0.000234 loss_vfl: 0.371823 loss_bbox: 0.201012 loss_dfl: 0.275971 loss: 0.819592 eta: 9:04:09 batch_cost: 6.4443 data_cost: 6.2407 ips: 1.5518 images/s
[07/04 08:30:38] ppdet.engine INFO: Epoch: [30] [115/526] learning_rate: 0.000234 loss_vfl: 0.338809 loss_bbox: 0.176295 loss_dfl: 0.262250 loss: 0.760780 eta: 9:04:40 batch_cost: 6.6208 data_cost: 6.4190 ips: 1.5104 images/s
[07/04 08:31:11] ppdet.engine INFO: Epoch: [30] [120/526] learning_rate: 0.000234 loss_vfl: 0.381612 loss_bbox: 0.228308 loss_dfl: 0.277745 loss: 0.912484 eta: 9:04:44 batch_cost: 6.5185 data_cost: 6.3231 ips: 1.5341 images/s
[07/04 08:31:43] ppdet.engine INFO: Epoch: [30] [125/526] learning_rate: 0.000234 loss_vfl: 0.348302 loss_bbox: 0.193593 loss_dfl: 0.273023 loss: 0.814918 eta: 9:03:54 batch_cost: 6.2724 data_cost: 6.0702 ips: 1.5943 images/s
[07/04 08:32:46] ppdet.engine INFO: Epoch: [30] [130/526] learning_rate: 0.000234 loss_vfl: 0.356520 loss_bbox: 0.200975 loss_dfl: 0.257538 loss: 0.798028 eta: 9:23:32 batch_cost: 12.5342 data_cost: 12.3390 ips: 0.7978 images/s
[07/04 08:33:18] ppdet.engine INFO: Epoch: [30] [135/526] learning_rate: 0.000234 loss_vfl: 0.324831 loss_bbox: 0.179908 loss_dfl: 0.267682 loss: 0.760818 eta: 9:21:54 batch_cost: 6.2423 data_cost: 6.0426 ips: 1.6020 images/s
[07/04 08:33:49] ppdet.engine INFO: Epoch: [30] [140/526] learning_rate: 0.000234 loss_vfl: 0.392030 loss_bbox: 0.184235 loss_dfl: 0.283842 loss: 0.835452 eta: 9:19:56 batch_cost: 6.1122 data_cost: 5.9020 ips: 1.6361 images/s
[07/04 08:34:19] ppdet.engine INFO: Epoch: [30] [145/526] learning_rate: 0.000234 loss_vfl: 0.393940 loss_bbox: 0.205640 loss_dfl: 0.264782 loss: 0.935550 eta: 9:17:50 batch_cost: 6.0284 data_cost: 5.8239 ips: 1.6588 images/s
[07/04 08:34:53] ppdet.engine INFO: Epoch: [30] [150/526] learning_rate: 0.000234 loss_vfl: 0.371489 loss_bbox: 0.206642 loss_dfl: 0.246825 loss: 0.826814 eta: 9:17:56 batch_cost: 6.7745 data_cost: 6.5769 ips: 1.4761 images/s
[07/04 08:35:26] ppdet.engine INFO: Epoch: [30] [155/526] learning_rate: 0.000234 loss_vfl: 0.403456 loss_bbox: 0.203092 loss_dfl: 0.265223 loss: 0.871771 eta: 9:16:55 batch_cost: 6.3764 data_cost: 6.1692 ips: 1.5683 images/s
[07/04 08:35:58] ppdet.engine INFO: Epoch: [30] [160/526] learning_rate: 0.000234 loss_vfl: 0.409780 loss_bbox: 0.265978 loss_dfl: 0.277989 loss: 0.953893 eta: 9:15:50 batch_cost: 6.3401 data_cost: 6.1403 ips: 1.5773 images/s
[07/04 08:36:30] ppdet.engine INFO: Epoch: [30] [165/526] learning_rate: 0.000234 loss_vfl: 0.329531 loss_bbox: 0.196506 loss_dfl: 0.282184 loss: 0.808221 eta: 9:15:00 batch_cost: 6.4316 data_cost: 6.2269 ips: 1.5548 images/s
[07/04 08:37:09] ppdet.engine INFO: Epoch: [30] [170/526] learning_rate: 0.000234 loss_vfl: 0.379238 loss_bbox: 0.213273 loss_dfl: 0.284255 loss: 0.876984 eta: 9:17:09 batch_cost: 7.6173 data_cost: 7.4226 ips: 1.3128 images/s
[07/04 08:37:39] ppdet.engine INFO: Epoch: [30] [175/526] learning_rate: 0.000234 loss_vfl: 0.406193 loss_bbox: 0.202611 loss_dfl: 0.264923 loss: 0.906799 eta: 9:15:00 batch_cost: 5.9019 data_cost: 5.7091 ips: 1.6944 images/s
[07/04 08:38:13] ppdet.engine INFO: Epoch: [30] [180/526] learning_rate: 0.000234 loss_vfl: 0.428965 loss_bbox: 0.206625 loss_dfl: 0.271385 loss: 0.887883 eta: 9:14:58 batch_cost: 6.7730 data_cost: 6.5747 ips: 1.4764 images/s
[07/04 08:38:44] ppdet.engine INFO: Epoch: [30] [185/526] learning_rate: 0.000234 loss_vfl: 0.408150 loss_bbox: 0.257625 loss_dfl: 0.278183 loss: 0.934830 eta: 9:13:37 batch_cost: 6.1976 data_cost: 5.9936 ips: 1.6135 images/s
[07/04 08:39:16] ppdet.engine INFO: Epoch: [30] [190/526] learning_rate: 0.000234 loss_vfl: 0.399427 loss_bbox: 0.217459 loss_dfl: 0.268942 loss: 0.877793 eta: 9:12:15 batch_cost: 6.1763 data_cost: 5.9777 ips: 1.6191 images/s
[07/04 08:39:48] ppdet.engine INFO: Epoch: [30] [195/526] learning_rate: 0.000234 loss_vfl: 0.367440 loss_bbox: 0.212307 loss_dfl: 0.274233 loss: 0.871385 eta: 9:11:24 batch_cost: 6.3912 data_cost: 6.2032 ips: 1.5647 images/s
[07/04 08:40:21] ppdet.engine INFO: Epoch: [30] [200/526] learning_rate: 0.000234 loss_vfl: 0.366338 loss_bbox: 0.184318 loss_dfl: 0.265318 loss: 0.807220 eta: 9:10:37 batch_cost: 6.4211 data_cost: 6.2129 ips: 1.5574 images/s
[07/04 08:40:51] ppdet.engine INFO: Epoch: [30] [205/526] learning_rate: 0.000234 loss_vfl: 0.362534 loss_bbox: 0.189367 loss_dfl: 0.258760 loss: 0.816139 eta: 9:09:06 batch_cost: 6.0495 data_cost: 5.8536 ips: 1.6530 images/s
[07/04 08:41:23] ppdet.engine INFO: Epoch: [30] [210/526] learning_rate: 0.000234 loss_vfl: 0.420306 loss_bbox: 0.221630 loss_dfl: 0.273549 loss: 0.902906 eta: 9:07:55 batch_cost: 6.1998 data_cost: 6.0037 ips: 1.6130 images/s
[07/04 08:41:55] ppdet.engine INFO: Epoch: [30] [215/526] learning_rate: 0.000234 loss_vfl: 0.397242 loss_bbox: 0.239314 loss_dfl: 0.272779 loss: 0.896754 eta: 9:07:15 batch_cost: 6.4492 data_cost: 6.2461 ips: 1.5506 images/s
[07/04 08:42:26] ppdet.engine INFO: Epoch: [30] [220/526] learning_rate: 0.000234 loss_vfl: 0.367279 loss_bbox: 0.207655 loss_dfl: 0.258904 loss: 0.852474 eta: 9:05:46 batch_cost: 6.0090 data_cost: 5.7985 ips: 1.6642 images/s
[07/04 08:42:56] ppdet.engine INFO: Epoch: [30] [225/526] learning_rate: 0.000234 loss_vfl: 0.365210 loss_bbox: 0.201924 loss_dfl: 0.267399 loss: 0.832329 eta: 9:04:24 batch_cost: 6.0512 data_cost: 5.8455 ips: 1.6526 images/s
[07/04 08:43:29] ppdet.engine INFO: Epoch: [30] [230/526] learning_rate: 0.000234 loss_vfl: 0.337811 loss_bbox: 0.223935 loss_dfl: 0.253527 loss: 0.847874 eta: 9:03:46 batch_cost: 6.4358 data_cost: 6.2284 ips: 1.5538 images/s
[07/04 08:44:00] ppdet.engine INFO: Epoch: [30] [235/526] learning_rate: 0.000234 loss_vfl: 0.377923 loss_bbox: 0.235267 loss_dfl: 0.266141 loss: 0.838544 eta: 9:02:39 batch_cost: 6.1607 data_cost: 5.9589 ips: 1.6232 images/s
[07/04 08:44:42] ppdet.engine INFO: Epoch: [30] [240/526] learning_rate: 0.000234 loss_vfl: 0.365305 loss_bbox: 0.204567 loss_dfl: 0.279140 loss: 0.889250 eta: 9:05:18 batch_cost: 8.3174 data_cost: 8.1052 ips: 1.2023 images/s
[07/04 08:45:25] ppdet.engine INFO: Epoch: [30] [245/526] learning_rate: 0.000234 loss_vfl: 0.376163 loss_bbox: 0.189279 loss_dfl: 0.281396 loss: 0.861057 eta: 9:08:01 batch_cost: 8.4416 data_cost: 8.2412 ips: 1.1846 images/s
[07/04 08:45:57] ppdet.engine INFO: Epoch: [30] [250/526] learning_rate: 0.000234 loss_vfl: 0.397535 loss_bbox: 0.220459 loss_dfl: 0.268722 loss: 0.899607 eta: 9:07:12 batch_cost: 6.3918 data_cost: 6.1853 ips: 1.5645 images/s
[07/04 08:46:30] ppdet.engine INFO: Epoch: [30] [255/526] learning_rate: 0.000234 loss_vfl: 0.361986 loss_bbox: 0.188704 loss_dfl: 0.275878 loss: 0.819698 eta: 9:06:39 batch_cost: 6.5513 data_cost: 6.3584 ips: 1.5264 images/s
[07/04 08:47:12] ppdet.engine INFO: Epoch: [30] [260/526] learning_rate: 0.000234 loss_vfl: 0.422700 loss_bbox: 0.218101 loss_dfl: 0.284449 loss: 0.942230 eta: 9:08:57 batch_cost: 8.3346 data_cost: 8.1397 ips: 1.1998 images/s
[07/04 08:48:04] ppdet.engine INFO: Epoch: [30] [265/526] learning_rate: 0.000234 loss_vfl: 0.375547 loss_bbox: 0.220359 loss_dfl: 0.275944 loss: 0.859988 eta: 9:14:01 batch_cost: 10.1797 data_cost: 9.9883 ips: 0.9824 images/s
[07/04 08:48:35] ppdet.engine INFO: Epoch: [30] [270/526] learning_rate: 0.000234 loss_vfl: 0.392788 loss_bbox: 0.201413 loss_dfl: 0.264923 loss: 0.835495 eta: 9:12:49 batch_cost: 6.2320 data_cost: 6.0129 ips: 1.6046 images/s
[07/04 08:49:08] ppdet.engine INFO: Epoch: [30] [275/526] learning_rate: 0.000234 loss_vfl: 0.387286 loss_bbox: 0.213138 loss_dfl: 0.263240 loss: 0.854135 eta: 9:12:02 batch_cost: 6.4979 data_cost: 6.2993 ips: 1.5389 images/s
[07/04 08:49:51] ppdet.engine INFO: Epoch: [30] [280/526] learning_rate: 0.000234 loss_vfl: 0.423613 loss_bbox: 0.258676 loss_dfl: 0.267565 loss: 0.971484 eta: 9:14:08 batch_cost: 8.4346 data_cost: 8.2274 ips: 1.1856 images/s
[07/04 08:50:21] ppdet.engine INFO: Epoch: [30] [285/526] learning_rate: 0.000234 loss_vfl: 0.400973 loss_bbox: 0.234442 loss_dfl: 0.268832 loss: 0.838679 eta: 9:12:39 batch_cost: 6.0361 data_cost: 5.8262 ips: 1.6567 images/s
[07/04 08:50:53] ppdet.engine INFO: Epoch: [30] [290/526] learning_rate: 0.000234 loss_vfl: 0.346771 loss_bbox: 0.206547 loss_dfl: 0.273426 loss: 0.844545 eta: 9:11:22 batch_cost: 6.1543 data_cost: 5.9393 ips: 1.6249 images/s
[07/04 08:51:25] ppdet.engine INFO: Epoch: [30] [295/526] learning_rate: 0.000234 loss_vfl: 0.323033 loss_bbox: 0.180434 loss_dfl: 0.250310 loss: 0.790440 eta: 9:10:30 batch_cost: 6.4398 data_cost: 6.2369 ips: 1.5528 images/s
[07/04 08:51:57] ppdet.engine INFO: Epoch: [30] [300/526] learning_rate: 0.000234 loss_vfl: 0.360624 loss_bbox: 0.182758 loss_dfl: 0.281024 loss: 0.816123 eta: 9:09:26 batch_cost: 6.2733 data_cost: 6.0673 ips: 1.5941 images/s
[07/04 08:52:29] ppdet.engine INFO: Epoch: [30] [305/526] learning_rate: 0.000234 loss_vfl: 0.381917 loss_bbox: 0.205289 loss_dfl: 0.257753 loss: 0.822699 eta: 9:08:19 batch_cost: 6.2395 data_cost: 6.0341 ips: 1.6027 images/s
[07/04 08:53:01] ppdet.engine INFO: Epoch: [30] [310/526] learning_rate: 0.000234 loss_vfl: 0.379168 loss_bbox: 0.190178 loss_dfl: 0.272284 loss: 0.936051 eta: 9:07:26 batch_cost: 6.3886 data_cost: 6.1975 ips: 1.5653 images/s
[07/04 08:53:33] ppdet.engine INFO: Epoch: [30] [315/526] learning_rate: 0.000234 loss_vfl: 0.403984 loss_bbox: 0.252688 loss_dfl: 0.279679 loss: 0.908417 eta: 9:06:32 batch_cost: 6.3668 data_cost: 6.1672 ips: 1.5706 images/s
[07/04 08:54:07] ppdet.engine INFO: Epoch: [30] [320/526] learning_rate: 0.000234 loss_vfl: 0.372247 loss_bbox: 0.196899 loss_dfl: 0.280508 loss: 0.849654 eta: 9:05:57 batch_cost: 6.6134 data_cost: 6.4064 ips: 1.5121 images/s
[07/04 08:54:38] ppdet.engine INFO: Epoch: [30] [325/526] learning_rate: 0.000234 loss_vfl: 0.354984 loss_bbox: 0.189127 loss_dfl: 0.267965 loss: 0.805358 eta: 9:04:52 batch_cost: 6.2090 data_cost: 6.0066 ips: 1.6106 images/s
[07/04 08:55:10] ppdet.engine INFO: Epoch: [30] [330/526] learning_rate: 0.000234 loss_vfl: 0.333515 loss_bbox: 0.182846 loss_dfl: 0.274709 loss: 0.775315 eta: 9:03:51 batch_cost: 6.2511 data_cost: 6.0504 ips: 1.5997 images/s
[07/04 08:55:40] ppdet.engine INFO: Epoch: [30] [335/526] learning_rate: 0.000234 loss_vfl: 0.366853 loss_bbox: 0.199308 loss_dfl: 0.279395 loss: 0.831633 eta: 9:02:27 batch_cost: 5.9206 data_cost: 5.7143 ips: 1.6890 images/s
[07/04 08:56:11] ppdet.engine INFO: Epoch: [30] [340/526] learning_rate: 0.000234 loss_vfl: 0.387321 loss_bbox: 0.199890 loss_dfl: 0.253732 loss: 0.857532 eta: 9:01:18 batch_cost: 6.1094 data_cost: 5.9076 ips: 1.6368 images/s
[07/04 08:56:41] ppdet.engine INFO: Epoch: [30] [345/526] learning_rate: 0.000234 loss_vfl: 0.392649 loss_bbox: 0.204637 loss_dfl: 0.263643 loss: 0.845812 eta: 9:00:03 batch_cost: 6.0056 data_cost: 5.7992 ips: 1.6651 images/s
[07/04 08:57:14] ppdet.engine INFO: Epoch: [30] [350/526] learning_rate: 0.000234 loss_vfl: 0.414876 loss_bbox: 0.217972 loss_dfl: 0.263249 loss: 0.917983 eta: 8:59:27 batch_cost: 6.5555 data_cost: 6.3612 ips: 1.5254 images/s
[07/04 08:57:45] ppdet.engine INFO: Epoch: [30] [355/526] learning_rate: 0.000234 loss_vfl: 0.382967 loss_bbox: 0.205519 loss_dfl: 0.270750 loss: 0.849254 eta: 8:58:18 batch_cost: 6.0683 data_cost: 5.8727 ips: 1.6479 images/s
[07/04 08:58:17] ppdet.engine INFO: Epoch: [30] [360/526] learning_rate: 0.000234 loss_vfl: 0.402333 loss_bbox: 0.213123 loss_dfl: 0.265559 loss: 0.886402 eta: 8:57:30 batch_cost: 6.3577 data_cost: 6.1632 ips: 1.5729 images/s
[07/04 08:58:47] ppdet.engine INFO: Epoch: [30] [365/526] learning_rate: 0.000234 loss_vfl: 0.433499 loss_bbox: 0.210310 loss_dfl: 0.264096 loss: 0.917488 eta: 8:56:11 batch_cost: 5.9026 data_cost: 5.6989 ips: 1.6942 images/s
[07/04 08:59:19] ppdet.engine INFO: Epoch: [30] [370/526] learning_rate: 0.000234 loss_vfl: 0.422219 loss_bbox: 0.239367 loss_dfl: 0.276189 loss: 0.937775 eta: 8:55:15 batch_cost: 6.2089 data_cost: 5.9908 ips: 1.6106 images/s
[07/04 08:59:49] ppdet.engine INFO: Epoch: [30] [375/526] learning_rate: 0.000234 loss_vfl: 0.356674 loss_bbox: 0.195882 loss_dfl: 0.259575 loss: 0.820024 eta: 8:54:04 batch_cost: 5.9900 data_cost: 5.8037 ips: 1.6694 images/s
[07/04 09:00:22] ppdet.engine INFO: Epoch: [30] [380/526] learning_rate: 0.000234 loss_vfl: 0.414729 loss_bbox: 0.230593 loss_dfl: 0.266254 loss: 0.951108 eta: 8:53:30 batch_cost: 6.5421 data_cost: 6.3360 ips: 1.5286 images/s
[07/04 09:00:55] ppdet.engine INFO: Epoch: [30] [385/526] learning_rate: 0.000234 loss_vfl: 0.366323 loss_bbox: 0.225326 loss_dfl: 0.290089 loss: 0.881664 eta: 8:52:47 batch_cost: 6.3972 data_cost: 6.1945 ips: 1.5632 images/s
[07/04 09:01:26] ppdet.engine INFO: Epoch: [30] [390/526] learning_rate: 0.000234 loss_vfl: 0.375917 loss_bbox: 0.215062 loss_dfl: 0.272625 loss: 0.884654 eta: 8:51:52 batch_cost: 6.2010 data_cost: 5.9959 ips: 1.6126 images/s
[07/04 09:01:55] ppdet.engine INFO: Epoch: [30] [395/526] learning_rate: 0.000234 loss_vfl: 0.394455 loss_bbox: 0.186490 loss_dfl: 0.273544 loss: 0.877967 eta: 8:50:34 batch_cost: 5.8183 data_cost: 5.6159 ips: 1.7187 images/s
[07/04 09:02:28] ppdet.engine INFO: Epoch: [30] [400/526] learning_rate: 0.000234 loss_vfl: 0.361132 loss_bbox: 0.196059 loss_dfl: 0.284260 loss: 0.815007 eta: 8:49:50 batch_cost: 6.3548 data_cost: 6.1523 ips: 1.5736 images/s
[07/04 09:02:59] ppdet.engine INFO: Epoch: [30] [405/526] learning_rate: 0.000234 loss_vfl: 0.374947 loss_bbox: 0.219281 loss_dfl: 0.288136 loss: 0.882364 eta: 8:48:57 batch_cost: 6.1925 data_cost: 5.9572 ips: 1.6149 images/s
[07/04 09:03:31] ppdet.engine INFO: Epoch: [30] [410/526] learning_rate: 0.000234 loss_vfl: 0.352020 loss_bbox: 0.213135 loss_dfl: 0.275756 loss: 0.852460 eta: 8:48:09 batch_cost: 6.2869 data_cost: 6.0762 ips: 1.5906 images/s
[07/04 09:04:03] ppdet.engine INFO: Epoch: [30] [415/526] learning_rate: 0.000234 loss_vfl: 0.353514 loss_bbox: 0.215985 loss_dfl: 0.272461 loss: 0.815334 eta: 8:47:25 batch_cost: 6.3393 data_cost: 6.1345 ips: 1.5775 images/s
[07/04 09:04:36] ppdet.engine INFO: Epoch: [30] [420/526] learning_rate: 0.000234 loss_vfl: 0.408100 loss_bbox: 0.234703 loss_dfl: 0.269753 loss: 0.914067 eta: 8:46:48 batch_cost: 6.4527 data_cost: 6.2533 ips: 1.5497 images/s
[07/04 09:05:09] ppdet.engine INFO: Epoch: [30] [425/526] learning_rate: 0.000234 loss_vfl: 0.416435 loss_bbox: 0.224434 loss_dfl: 0.251133 loss: 0.873551 eta: 8:46:15 batch_cost: 6.5260 data_cost: 6.3345 ips: 1.5323 images/s
[07/04 09:06:10] ppdet.engine INFO: Epoch: [30] [430/526] learning_rate: 0.000234 loss_vfl: 0.389648 loss_bbox: 0.178283 loss_dfl: 0.253066 loss: 0.834921 eta: 8:51:02 batch_cost: 12.2418 data_cost: 12.0285 ips: 0.8169 images/s
[07/04 09:06:42] ppdet.engine INFO: Epoch: [30] [435/526] learning_rate: 0.000234 loss_vfl: 0.378850 loss_bbox: 0.219586 loss_dfl: 0.282447 loss: 0.905134 eta: 8:50:12 batch_cost: 6.2870 data_cost: 6.0908 ips: 1.5906 images/s
[07/04 09:07:11] ppdet.engine INFO: Epoch: [30] [440/526] learning_rate: 0.000234 loss_vfl: 0.377423 loss_bbox: 0.202711 loss_dfl: 0.251702 loss: 0.819281 eta: 8:48:54 batch_cost: 5.7638 data_cost: 5.5583 ips: 1.7350 images/s
[07/04 09:07:45] ppdet.engine INFO: Epoch: [30] [445/526] learning_rate: 0.000234 loss_vfl: 0.340947 loss_bbox: 0.208815 loss_dfl: 0.263789 loss: 0.829212 eta: 8:48:29 batch_cost: 6.7249 data_cost: 6.5214 ips: 1.4870 images/s
[07/04 09:08:19] ppdet.engine INFO: Epoch: [30] [450/526] learning_rate: 0.000234 loss_vfl: 0.306914 loss_bbox: 0.183233 loss_dfl: 0.270282 loss: 0.737697 eta: 8:48:02 batch_cost: 6.7086 data_cost: 6.5032 ips: 1.4906 images/s
[07/04 09:08:54] ppdet.engine INFO: Epoch: [30] [455/526] learning_rate: 0.000234 loss_vfl: 0.366436 loss_bbox: 0.203972 loss_dfl: 0.281986 loss: 0.825535 eta: 8:47:44 batch_cost: 6.8709 data_cost: 6.6673 ips: 1.4554 images/s
[07/04 09:09:26] ppdet.engine INFO: Epoch: [30] [460/526] learning_rate: 0.000234 loss_vfl: 0.369893 loss_bbox: 0.228326 loss_dfl: 0.275160 loss: 0.857507 eta: 8:46:56 batch_cost: 6.2900 data_cost: 6.0841 ips: 1.5898 images/s
[07/04 09:09:58] ppdet.engine INFO: Epoch: [30] [465/526] learning_rate: 0.000234 loss_vfl: 0.398390 loss_bbox: 0.233455 loss_dfl: 0.284600 loss: 0.902023 eta: 8:46:08 batch_cost: 6.3034 data_cost: 6.1091 ips: 1.5865 images/s
[07/04 09:10:29] ppdet.engine INFO: Epoch: [30] [470/526] learning_rate: 0.000234 loss_vfl: 0.359512 loss_bbox: 0.193861 loss_dfl: 0.266186 loss: 0.828418 eta: 8:45:11 batch_cost: 6.1119 data_cost: 5.9059 ips: 1.6361 images/s
[07/04 09:11:01] ppdet.engine INFO: Epoch: [30] [475/526] learning_rate: 0.000234 loss_vfl: 0.344352 loss_bbox: 0.201831 loss_dfl: 0.268511 loss: 0.862905 eta: 8:44:26 batch_cost: 6.3307 data_cost: 6.1250 ips: 1.5796 images/s
[07/04 09:11:33] ppdet.engine INFO: Epoch: [30] [480/526] learning_rate: 0.000234 loss_vfl: 0.360266 loss_bbox: 0.195175 loss_dfl: 0.266284 loss: 0.811731 eta: 8:43:37 batch_cost: 6.2581 data_cost: 6.0601 ips: 1.5979 images/s
[07/04 09:12:02] ppdet.engine INFO: Epoch: [30] [485/526] learning_rate: 0.000234 loss_vfl: 0.391112 loss_bbox: 0.260875 loss_dfl: 0.273799 loss: 0.925785 eta: 8:42:31 batch_cost: 5.8835 data_cost: 5.6703 ips: 1.6997 images/s
[07/04 09:12:35] ppdet.engine INFO: Epoch: [30] [490/526] learning_rate: 0.000234 loss_vfl: 0.340247 loss_bbox: 0.188914 loss_dfl: 0.263015 loss: 0.800877 eta: 8:41:54 batch_cost: 6.4934 data_cost: 6.2919 ips: 1.5400 images/s
[07/04 09:13:07] ppdet.engine INFO: Epoch: [30] [495/526] learning_rate: 0.000234 loss_vfl: 0.388295 loss_bbox: 0.221719 loss_dfl: 0.288300 loss: 0.906224 eta: 8:41:02 batch_cost: 6.1651 data_cost: 5.9585 ips: 1.6220 images/s
[07/04 09:13:43] ppdet.engine INFO: Epoch: [30] [500/526] learning_rate: 0.000234 loss_vfl: 0.404059 loss_bbox: 0.200618 loss_dfl: 0.277728 loss: 0.872636 eta: 8:40:57 batch_cost: 7.1494 data_cost: 6.9588 ips: 1.3987 images/s
[07/04 09:14:21] ppdet.engine INFO: Epoch: [30] [505/526] learning_rate: 0.000234 loss_vfl: 0.434132 loss_bbox: 0.267540 loss_dfl: 0.274427 loss: 0.960191 eta: 8:41:13 batch_cost: 7.5900 data_cost: 7.3898 ips: 1.3175 images/s
[07/04 09:14:54] ppdet.engine INFO: Epoch: [30] [510/526] learning_rate: 0.000234 loss_vfl: 0.365376 loss_bbox: 0.216388 loss_dfl: 0.262516 loss: 0.851588 eta: 8:40:34 batch_cost: 6.4613 data_cost: 6.2501 ips: 1.5477 images/s
[07/04 09:15:27] ppdet.engine INFO: Epoch: [30] [515/526] learning_rate: 0.000234 loss_vfl: 0.377622 loss_bbox: 0.189161 loss_dfl: 0.261540 loss: 0.805966 eta: 8:39:57 batch_cost: 6.4716 data_cost: 6.2723 ips: 1.5452 images/s
[07/04 09:16:26] ppdet.engine INFO: Epoch: [30] [520/526] learning_rate: 0.000234 loss_vfl: 0.342163 loss_bbox: 0.164112 loss_dfl: 0.269315 loss: 0.778167 eta: 8:43:19 batch_cost: 11.7493 data_cost: 11.5312 ips: 0.8511 images/s
[07/04 09:16:57] ppdet.engine INFO: Epoch: [30] [525/526] learning_rate: 0.000234 loss_vfl: 0.409647 loss_bbox: 0.213941 loss_dfl: 0.277686 loss: 0.868590 eta: 8:42:25 batch_cost: 6.1562 data_cost: 5.9479 ips: 1.6244 images/s
[07/04 09:17:05] ppdet.utils.checkpoint INFO: Save checkpoint: output/picodet_m_416_coco_lcnet
loading annotations into memory...
Done (t=0.52s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
[07/04 09:22:02] ppdet.engine INFO: Eval iter: 0
[07/04 09:31:21] ppdet.engine INFO: Eval iter: 100
[07/04 09:33:51] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.04s)
creating index...
index created!
[07/04 09:33:53] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=0.88s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.81s).
Accumulating evaluation results...
DONE (t=0.16s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.861
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.988
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.772
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.870
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.885
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.885
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.885
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.809
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.895
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
[07/04 09:33:55] ppdet.engine INFO: Total sample number: 1000, averge FPS: 1.4042449080827668
[07/04 09:33:55] ppdet.engine INFO: Best test bbox ap is 0.861.
[07/04 09:34:09] ppdet.utils.checkpoint INFO: Save checkpoint: output/picodet_m_416_coco_lcnet
[07/04 09:34:10] ppdet.engine INFO: Epoch: [31] [  0/526] learning_rate: 0.000234 loss_vfl: 0.409647 loss_bbox: 0.213941 loss_dfl: 0.269499 loss: 0.868590 eta: 8:41:21 batch_cost: 4.9662 data_cost: 4.7598 ips: 2.0136 images/s
[07/04 09:34:20] ppdet.engine INFO: Epoch: [31] [  5/526] learning_rate: 0.000234 loss_vfl: 0.378815 loss_bbox: 0.230606 loss_dfl: 0.285081 loss: 0.894373 eta: 8:37:19 batch_cost: 1.8999 data_cost: 1.6849 ips: 5.2635 images/s

However, the PP-YOLO model does not work, and it generates the error like that below. After "Save checkpoint: output/ppyolo_r50vd_dcn_1x_coco", the scripts continue but it does not generate anything.

/usr/local/lib/python3.7/dist-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
        Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
          if data.dtype == np.object:
        /usr/local/lib/python3.7/dist-packages/scipy/fft/__init__.py:97: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
          from numpy.dual import register_func
        /usr/local/lib/python3.7/dist-packages/scipy/sparse/sputils.py:17: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.
          supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
        /usr/local/lib/python3.7/dist-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
        Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
          from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
        loading annotations into memory...
        Done (t=1.25s)
        creating index...
        index created!
        W0704 07:56:05.869805  3037 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 6.0, Driver API Version: 11.2, Runtime API Version: 10.2
        W0704 07:56:06.230222  3037 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.
        [07/04 07:56:23] ppdet.utils.checkpoint INFO: Finish resuming model weights: output/ppyolo_r50vd_dcn_1x_coco/6.pdparams
        [07/04 07:56:28] ppdet.engine INFO: Epoch: [7] [  0/526] learning_rate: 0.000092 loss_xy: 0.271238 loss_wh: 0.357589 loss_iou: 1.576589 loss_iou_aware: 0.466314 loss_obj: 0.966710 loss_cls: 0.038763 loss: 3.677203 eta: 2:15:35 batch_cost: 0.6725 data_cost: 0.0002 ips: 14.8699 images/s
        [07/04 07:58:19] ppdet.engine INFO: Epoch: [7] [  5/526] learning_rate: 0.000092 loss_xy: 0.393540 loss_wh: 0.393296 loss_iou: 1.853384 loss_iou_aware: 0.507917 loss_obj: 1.164704 loss_cls: 0.027172 loss: 4.602396 eta: 2 days, 14:31:10 batch_cost: 22.1995 data_cost: 21.5795 ips: 0.4505 images/s
        [07/04 08:00:05] ppdet.engine INFO: Epoch: [7] [ 10/526] learning_rate: 0.000092 loss_xy: 0.296190 loss_wh: 0.355718 loss_iou: 1.808403 loss_iou_aware: 0.511291 loss_obj: 1.258633 loss_cls: 0.036853 loss: 4.278323 eta: 2 days, 18:16:28 batch_cost: 21.0889 data_cost: 20.6761 ips: 0.4742 images/s
        [07/04 08:01:51] ppdet.engine INFO: Epoch: [7] [ 15/526] learning_rate: 0.000092 loss_xy: 0.349579 loss_wh: 0.397204 loss_iou: 1.813698 loss_iou_aware: 0.490712 loss_obj: 0.934151 loss_cls: 0.027935 loss: 3.974364 eta: 2 days, 19:48:32 batch_cost: 21.2266 data_cost: 20.8666 ips: 0.4711 images/s
        [07/04 08:03:32] ppdet.engine INFO: Epoch: [7] [ 20/526] learning_rate: 0.000093 loss_xy: 0.266105 loss_wh: 0.413637 loss_iou: 1.843661 loss_iou_aware: 0.492841 loss_obj: 0.869180 loss_cls: 0.028634 loss: 4.009541 eta: 2 days, 19:38:12 batch_cost: 20.0227 data_cost: 19.5911 ips: 0.4994 images/s
        [07/04 08:05:10] ppdet.engine INFO: Epoch: [7] [ 25/526] learning_rate: 0.000093 loss_xy: 0.342779 loss_wh: 0.388728 loss_iou: 1.959213 loss_iou_aware: 0.528961 loss_obj: 1.474920 loss_cls: 0.036803 loss: 4.570692 eta: 2 days, 19:14:28 batch_cost: 19.5898 data_cost: 19.1491 ips: 0.5105 images/s
        [07/04 08:06:51] ppdet.engine INFO: Epoch: [7] [ 30/526] learning_rate: 0.000093 loss_xy: 0.262813 loss_wh: 0.490076 loss_iou: 2.170886 loss_iou_aware: 0.535854 loss_obj: 0.599896 loss_cls: 0.022922 loss: 4.201983 eta: 2 days, 19:18:15 batch_cost: 20.2186 data_cost: 19.7713 ips: 0.4946 images/s
        [07/04 08:08:28] ppdet.engine INFO: Epoch: [7] [ 35/526] learning_rate: 0.000093 loss_xy: 0.354492 loss_wh: 0.440776 loss_iou: 1.925000 loss_iou_aware: 0.505434 loss_obj: 1.267373 loss_cls: 0.032808 loss: 4.320576 eta: 2 days, 18:51:40 batch_cost: 19.1853 data_cost: 18.7586 ips: 0.5212 images/s
        [07/04 08:10:07] ppdet.engine INFO: Epoch: [7] [ 40/526] learning_rate: 0.000093 loss_xy: 0.290053 loss_wh: 0.384202 loss_iou: 1.844284 loss_iou_aware: 0.509619 loss_obj: 1.169575 loss_cls: 0.034189 loss: 4.214235 eta: 2 days, 18:47:09 batch_cost: 19.8371 data_cost: 19.3097 ips: 0.5041 images/s
        [07/04 08:11:43] ppdet.engine INFO: Epoch: [7] [ 45/526] learning_rate: 0.000093 loss_xy: 0.388575 loss_wh: 0.415760 loss_iou: 1.850632 loss_iou_aware: 0.503393 loss_obj: 1.254367 loss_cls: 0.054199 loss: 4.597940 eta: 2 days, 18:28:53 batch_cost: 19.1796 data_cost: 18.6028 ips: 0.5214 images/s
        [07/04 08:13:16] ppdet.engine INFO: Epoch: [7] [ 50/526] learning_rate: 0.000093 loss_xy: 0.322385 loss_wh: 0.408355 loss_iou: 1.852846 loss_iou_aware: 0.482248 loss_obj: 1.192866 loss_cls: 0.032046 loss: 4.015693 eta: 2 days, 17:59:02 batch_cost: 18.4240 data_cost: 17.9292 ips: 0.5428 images/s
        [07/04 08:14:52] ppdet.engine INFO: Epoch: [7] [ 55/526] learning_rate: 0.000093 loss_xy: 0.365895 loss_wh: 0.344703 loss_iou: 1.800770 loss_iou_aware: 0.487615 loss_obj: 0.812750 loss_cls: 0.038883 loss: 4.197926 eta: 2 days, 17:48:06 batch_cost: 19.1982 data_cost: 18.6566 ips: 0.5209 images/s
        [07/04 08:16:39] ppdet.engine INFO: Epoch: [7] [ 60/526] learning_rate: 0.000094 loss_xy: 0.323396 loss_wh: 0.356192 loss_iou: 1.771033 loss_iou_aware: 0.495836 loss_obj: 0.977654 loss_cls: 0.024881 loss: 4.188312 eta: 2 days, 18:12:15 batch_cost: 21.2381 data_cost: 20.7056 ips: 0.4709 images/s
        [07/04 08:18:24] ppdet.engine INFO: Epoch: [7] [ 65/526] learning_rate: 0.000094 loss_xy: 0.302803 loss_wh: 0.403184 loss_iou: 1.620388 loss_iou_aware: 0.476786 loss_obj: 1.081181 loss_cls: 0.025496 loss: 4.067637 eta: 2 days, 18:28:49 batch_cost: 20.9976 data_cost: 20.3531 ips: 0.4762 images/s
        [07/04 08:20:03] ppdet.engine INFO: Epoch: [7] [ 70/526] learning_rate: 0.000094 loss_xy: 0.272057 loss_wh: 0.359453 loss_iou: 1.717737 loss_iou_aware: 0.512510 loss_obj: 1.106217 loss_cls: 0.029591 loss: 3.957222 eta: 2 days, 18:26:15 batch_cost: 19.8250 data_cost: 19.2783 ips: 0.5044 images/s
        [07/04 08:21:40] ppdet.engine INFO: Epoch: [7] [ 75/526] learning_rate: 0.000094 loss_xy: 0.364607 loss_wh: 0.509280 loss_iou: 2.173105 loss_iou_aware: 0.538300 loss_obj: 1.016085 loss_cls: 0.031052 loss: 4.744166 eta: 2 days, 18:16:59 batch_cost: 19.3082 data_cost: 18.8394 ips: 0.5179 images/s
        [07/04 08:23:09] ppdet.engine INFO: Epoch: [7] [ 80/526] learning_rate: 0.000094 loss_xy: 0.257277 loss_wh: 0.375538 loss_iou: 1.660879 loss_iou_aware: 0.457092 loss_obj: 0.781248 loss_cls: 0.025709 loss: 3.604205 eta: 2 days, 17:48:01 batch_cost: 17.6373 data_cost: 17.1693 ips: 0.5670 images/s
        [07/04 08:24:47] ppdet.engine INFO: Epoch: [7] [ 85/526] learning_rate: 0.000094 loss_xy: 0.338342 loss_wh: 0.438676 loss_iou: 1.810882 loss_iou_aware: 0.521993 loss_obj: 1.391615 loss_cls: 0.038322 loss: 4.830496 eta: 2 days, 17:45:34 batch_cost: 19.6407 data_cost: 19.0641 ips: 0.5091 images/s
        [07/04 08:26:29] ppdet.engine INFO: Epoch: [7] [ 90/526] learning_rate: 0.000094 loss_xy: 0.297272 loss_wh: 0.411607 loss_iou: 1.937959 loss_iou_aware: 0.492738 loss_obj: 1.045178 loss_cls: 0.032997 loss: 4.195816 eta: 2 days, 17:49:35 batch_cost: 20.2217 data_cost: 19.7197 ips: 0.4945 images/s
        [07/04 08:28:05] ppdet.engine INFO: Epoch: [7] [ 95/526] learning_rate: 0.000094 loss_xy: 0.281842 loss_wh: 0.503617 loss_iou: 2.080297 loss_iou_aware: 0.541769 loss_obj: 1.209590 loss_cls: 0.034395 loss: 4.584342 eta: 2 days, 17:42:49 batch_cost: 19.2440 data_cost: 18.6377 ips: 0.5196 images/s
        [07/04 08:29:40] ppdet.engine INFO: Epoch: [7] [100/526] learning_rate: 0.000095 loss_xy: 0.358479 loss_wh: 0.559437 loss_iou: 2.222184 loss_iou_aware: 0.561517 loss_obj: 1.193535 loss_cls: 0.028179 loss: 4.959227 eta: 2 days, 17:32:36 batch_cost: 18.8426 data_cost: 18.2427 ips: 0.5307 images/s
        [07/04 08:31:20] ppdet.engine INFO: Epoch: [7] [105/526] learning_rate: 0.000095 loss_xy: 0.290664 loss_wh: 0.389840 loss_iou: 1.862712 loss_iou_aware: 0.503922 loss_obj: 1.080857 loss_cls: 0.030670 loss: 4.402097 eta: 2 days, 17:33:49 batch_cost: 19.9694 data_cost: 19.5400 ips: 0.5008 images/s
        [07/04 08:32:52] ppdet.engine INFO: Epoch: [7] [110/526] learning_rate: 0.000095 loss_xy: 0.270563 loss_wh: 0.424142 loss_iou: 1.926599 loss_iou_aware: 0.529753 loss_obj: 0.971601 loss_cls: 0.027508 loss: 4.074944 eta: 2 days, 17:20:02 batch_cost: 18.3311 data_cost: 17.7899 ips: 0.5455 images/s
        [07/04 08:34:31] ppdet.engine INFO: Epoch: [7] [115/526] learning_rate: 0.000095 loss_xy: 0.336742 loss_wh: 0.445517 loss_iou: 1.844308 loss_iou_aware: 0.523903 loss_obj: 1.162706 loss_cls: 0.035103 loss: 4.854972 eta: 2 days, 17:19:32 batch_cost: 19.7514 data_cost: 19.2809 ips: 0.5063 images/s
        [07/04 08:36:01] ppdet.engine INFO: Epoch: [7] [120/526] learning_rate: 0.000095 loss_xy: 0.323625 loss_wh: 0.443975 loss_iou: 1.987566 loss_iou_aware: 0.531336 loss_obj: 0.864106 loss_cls: 0.032928 loss: 4.180026 eta: 2 days, 17:04:18 batch_cost: 17.9772 data_cost: 17.4612 ips: 0.5563 images/s
        [07/04 08:37:37] ppdet.engine INFO: Epoch: [7] [125/526] learning_rate: 0.000095 loss_xy: 0.272251 loss_wh: 0.446770 loss_iou: 1.835055 loss_iou_aware: 0.502560 loss_obj: 0.952061 loss_cls: 0.028709 loss: 4.316224 eta: 2 days, 16:58:23 batch_cost: 19.0160 data_cost: 18.5340 ips: 0.5259 images/s
        [07/04 08:39:10] ppdet.engine INFO: Epoch: [7] [130/526] learning_rate: 0.000095 loss_xy: 0.320515 loss_wh: 0.504043 loss_iou: 2.068436 loss_iou_aware: 0.558140 loss_obj: 1.086174 loss_cls: 0.024959 loss: 4.700398 eta: 2 days, 16:50:02 batch_cost: 18.6520 data_cost: 18.1442 ips: 0.5361 images/s
        [07/04 08:40:48] ppdet.engine INFO: Epoch: [7] [135/526] learning_rate: 0.000095 loss_xy: 0.277923 loss_wh: 0.415701 loss_iou: 1.783135 loss_iou_aware: 0.500629 loss_obj: 1.064106 loss_cls: 0.024428 loss: 4.172996 eta: 2 days, 16:47:50 batch_cost: 19.4251 data_cost: 18.8585 ips: 0.5148 images/s
        [07/04 08:42:18] ppdet.engine INFO: Epoch: [7] [140/526] learning_rate: 0.000096 loss_xy: 0.261334 loss_wh: 0.397305 loss_iou: 1.819752 loss_iou_aware: 0.508419 loss_obj: 0.813551 loss_cls: 0.034020 loss: 4.046906 eta: 2 days, 16:35:10 batch_cost: 17.9363 data_cost: 17.4196 ips: 0.5575 images/s
        [07/04 08:43:57] ppdet.engine INFO: Epoch: [7] [145/526] learning_rate: 0.000096 loss_xy: 0.267708 loss_wh: 0.439971 loss_iou: 1.937807 loss_iou_aware: 0.516805 loss_obj: 0.881077 loss_cls: 0.022074 loss: 4.369938 eta: 2 days, 16:36:33 batch_cost: 19.8855 data_cost: 19.3884 ips: 0.5029 images/s
        [07/04 08:45:29] ppdet.engine INFO: Epoch: [7] [150/526] learning_rate: 0.000096 loss_xy: 0.302159 loss_wh: 0.410837 loss_iou: 1.829268 loss_iou_aware: 0.522165 loss_obj: 0.884524 loss_cls: 0.030697 loss: 4.217772 eta: 2 days, 16:26:45 batch_cost: 18.2189 data_cost: 17.7389 ips: 0.5489 images/s
        [07/04 08:46:56] ppdet.engine INFO: Epoch: [7] [155/526] learning_rate: 0.000096 loss_xy: 0.294403 loss_wh: 0.449907 loss_iou: 1.951491 loss_iou_aware: 0.511063 loss_obj: 1.005861 loss_cls: 0.031593 loss: 4.264506 eta: 2 days, 16:12:21 batch_cost: 17.4135 data_cost: 16.9068 ips: 0.5743 images/s
        [07/04 08:48:28] ppdet.engine INFO: Epoch: [7] [160/526] learning_rate: 0.000096 loss_xy: 0.333069 loss_wh: 0.455826 loss_iou: 1.911201 loss_iou_aware: 0.523360 loss_obj: 1.250967 loss_cls: 0.037812 loss: 4.597150 eta: 2 days, 16:03:59 batch_cost: 18.2594 data_cost: 17.7629 ips: 0.5477 images/s
        [07/04 08:49:48] ppdet.engine INFO: Epoch: [7] [165/526] learning_rate: 0.000096 loss_xy: 0.285064 loss_wh: 0.460953 loss_iou: 1.938991 loss_iou_aware: 0.524561 loss_obj: 1.481674 loss_cls: 0.030693 loss: 4.903224 eta: 2 days, 15:42:21 batch_cost: 15.9772 data_cost: 15.4923 ips: 0.6259 images/s
        [07/04 08:51:28] ppdet.engine INFO: Epoch: [7] [170/526] learning_rate: 0.000096 loss_xy: 0.296580 loss_wh: 0.438477 loss_iou: 1.945454 loss_iou_aware: 0.527978 loss_obj: 1.300317 loss_cls: 0.024111 loss: 4.631711 eta: 2 days, 15:44:28 batch_cost: 19.8611 data_cost: 19.4522 ips: 0.5035 images/s
        [07/04 08:53:04] ppdet.engine INFO: Epoch: [7] [175/526] learning_rate: 0.000096 loss_xy: 0.261097 loss_wh: 0.409652 loss_iou: 1.876185 loss_iou_aware: 0.500499 loss_obj: 0.949409 loss_cls: 0.027298 loss: 4.010798 eta: 2 days, 15:42:46 batch_cost: 19.2197 data_cost: 18.6716 ips: 0.5203 images/s
        [07/04 08:54:25] ppdet.engine INFO: Epoch: [7] [180/526] learning_rate: 0.000097 loss_xy: 0.244751 loss_wh: 0.326543 loss_iou: 1.684549 loss_iou_aware: 0.477246 loss_obj: 0.867979 loss_cls: 0.019974 loss: 3.445610 eta: 2 days, 15:24:38 batch_cost: 16.2259 data_cost: 15.7292 ips: 0.6163 images/s
        [07/04 08:55:54] ppdet.engine INFO: Epoch: [7] [185/526] learning_rate: 0.000097 loss_xy: 0.302227 loss_wh: 0.412029 loss_iou: 2.004558 loss_iou_aware: 0.531182 loss_obj: 1.017971 loss_cls: 0.031512 loss: 4.420315 eta: 2 days, 15:14:41 batch_cost: 17.5860 data_cost: 17.0642 ips: 0.5686 images/s
        [07/04 08:57:15] ppdet.engine INFO: Epoch: [7] [190/526] learning_rate: 0.000097 loss_xy: 0.277390 loss_wh: 0.375625 loss_iou: 1.787648 loss_iou_aware: 0.489875 loss_obj: 1.042646 loss_cls: 0.027224 loss: 4.044694 eta: 2 days, 14:58:15 batch_cost: 16.2555 data_cost: 15.7897 ips: 0.6152 images/s
        [07/04 08:58:33] ppdet.engine INFO: Epoch: [7] [195/526] learning_rate: 0.000097 loss_xy: 0.374124 loss_wh: 0.449620 loss_iou: 1.844166 loss_iou_aware: 0.508744 loss_obj: 0.843660 loss_cls: 0.030259 loss: 4.087278 eta: 2 days, 14:38:44 batch_cost: 15.4966 data_cost: 14.9542 ips: 0.6453 images/s
        [07/04 09:00:16] ppdet.engine INFO: Epoch: [7] [200/526] learning_rate: 0.000097 loss_xy: 0.343149 loss_wh: 0.526326 loss_iou: 2.347107 loss_iou_aware: 0.565511 loss_obj: 1.125885 loss_cls: 0.027738 loss: 4.880188 eta: 2 days, 14:45:17 batch_cost: 20.5930 data_cost: 20.0949 ips: 0.4856 images/s
        [07/04 09:01:42] ppdet.engine INFO: Epoch: [7] [205/526] learning_rate: 0.000097 loss_xy: 0.261592 loss_wh: 0.389280 loss_iou: 1.730122 loss_iou_aware: 0.490115 loss_obj: 0.942802 loss_cls: 0.029430 loss: 3.885148 eta: 2 days, 14:34:02 batch_cost: 16.9777 data_cost: 16.4722 ips: 0.5890 images/s
        [07/04 09:03:14] ppdet.engine INFO: Epoch: [7] [210/526] learning_rate: 0.000097 loss_xy: 0.261652 loss_wh: 0.361626 loss_iou: 1.876572 loss_iou_aware: 0.515457 loss_obj: 1.111798 loss_cls: 0.023007 loss: 4.210696 eta: 2 days, 14:29:55 batch_cost: 18.4018 data_cost: 17.9000 ips: 0.5434 images/s
        [07/04 09:04:35] ppdet.engine INFO: Epoch: [7] [215/526] learning_rate: 0.000097 loss_xy: 0.325030 loss_wh: 0.367326 loss_iou: 1.797746 loss_iou_aware: 0.478234 loss_obj: 1.167450 loss_cls: 0.028474 loss: 3.948726 eta: 2 days, 14:15:39 batch_cost: 16.1564 data_cost: 15.6647 ips: 0.6189 images/s
        [07/04 09:05:52] ppdet.engine INFO: Epoch: [7] [220/526] learning_rate: 0.000098 loss_xy: 0.310715 loss_wh: 0.437046 loss_iou: 1.924333 loss_iou_aware: 0.514065 loss_obj: 1.181035 loss_cls: 0.033214 loss: 4.502135 eta: 2 days, 13:58:45 batch_cost: 15.4405 data_cost: 15.0305 ips: 0.6476 images/s
        [07/04 09:07:14] ppdet.engine INFO: Epoch: [7] [225/526] learning_rate: 0.000098 loss_xy: 0.307318 loss_wh: 0.339045 loss_iou: 1.859636 loss_iou_aware: 0.488805 loss_obj: 1.258562 loss_cls: 0.033871 loss: 4.121819 eta: 2 days, 13:46:05 batch_cost: 16.2501 data_cost: 15.7163 ips: 0.6154 images/s
        [07/04 09:08:31] ppdet.engine INFO: Epoch: [7] [230/526] learning_rate: 0.000098 loss_xy: 0.335790 loss_wh: 0.391563 loss_iou: 2.035825 loss_iou_aware: 0.506498 loss_obj: 1.079076 loss_cls: 0.032927 loss: 4.267739 eta: 2 days, 13:30:21 batch_cost: 15.4150 data_cost: 14.9257 ips: 0.6487 images/s
        [07/04 09:09:55] ppdet.engine INFO: Epoch: [7] [235/526] learning_rate: 0.000098 loss_xy: 0.273100 loss_wh: 0.385736 loss_iou: 1.810737 loss_iou_aware: 0.504320 loss_obj: 1.299198 loss_cls: 0.028543 loss: 4.216803 eta: 2 days, 13:20:10 batch_cost: 16.5971 data_cost: 15.9934 ips: 0.6025 images/s
        [07/04 09:11:12] ppdet.engine INFO: Epoch: [7] [240/526] learning_rate: 0.000098 loss_xy: 0.376021 loss_wh: 0.472867 loss_iou: 2.046918 loss_iou_aware: 0.545379 loss_obj: 1.354846 loss_cls: 0.041773 loss: 4.938355 eta: 2 days, 13:05:16 batch_cost: 15.3609 data_cost: 14.7778 ips: 0.6510 images/s
        [07/04 09:12:33] ppdet.engine INFO: Epoch: [7] [245/526] learning_rate: 0.000098 loss_xy: 0.270698 loss_wh: 0.385635 loss_iou: 1.851159 loss_iou_aware: 0.504134 loss_obj: 0.915318 loss_cls: 0.021576 loss: 3.918638 eta: 2 days, 12:54:08 batch_cost: 16.1567 data_cost: 15.6123 ips: 0.6189 images/s
        [07/04 09:13:53] ppdet.engine INFO: Epoch: [7] [250/526] learning_rate: 0.000098 loss_xy: 0.304756 loss_wh: 0.413156 loss_iou: 1.817480 loss_iou_aware: 0.509742 loss_obj: 1.042281 loss_cls: 0.024843 loss: 4.121634 eta: 2 days, 12:42:27 batch_cost: 15.9175 data_cost: 15.4445 ips: 0.6282 images/s
        [07/04 09:15:12] ppdet.engine INFO: Epoch: [7] [255/526] learning_rate: 0.000098 loss_xy: 0.296441 loss_wh: 0.378142 loss_iou: 1.758487 loss_iou_aware: 0.502587 loss_obj: 0.976710 loss_cls: 0.029148 loss: 4.023067 eta: 2 days, 12:30:54 batch_cost: 15.8513 data_cost: 15.2941 ips: 0.6309 images/s
        [07/04 09:16:31] ppdet.engine INFO: Epoch: [7] [260/526] learning_rate: 0.000099 loss_xy: 0.302066 loss_wh: 0.363873 loss_iou: 1.888663 loss_iou_aware: 0.510169 loss_obj: 0.877994 loss_cls: 0.025846 loss: 3.763185 eta: 2 days, 12:19:08 batch_cost: 15.6881 data_cost: 15.2634 ips: 0.6374 images/s
        [07/04 09:17:48] ppdet.engine INFO: Epoch: [7] [265/526] learning_rate: 0.000099 loss_xy: 0.284156 loss_wh: 0.423852 loss_iou: 1.969575 loss_iou_aware: 0.519264 loss_obj: 0.938604 loss_cls: 0.027907 loss: 4.515782 eta: 2 days, 12:06:23 batch_cost: 15.3162 data_cost: 14.9079 ips: 0.6529 images/s
        [07/04 09:19:02] ppdet.engine INFO: Epoch: [7] [270/526] learning_rate: 0.000099 loss_xy: 0.321137 loss_wh: 0.434120 loss_iou: 1.981202 loss_iou_aware: 0.535011 loss_obj: 1.103608 loss_cls: 0.028343 loss: 4.492757 eta: 2 days, 11:52:08 batch_cost: 14.7838 data_cost: 14.3053 ips: 0.6764 images/s
        [07/04 09:20:58] ppdet.engine INFO: Epoch: [7] [275/526] learning_rate: 0.000099 loss_xy: 0.297710 loss_wh: 0.374889 loss_iou: 1.719362 loss_iou_aware: 0.480011 loss_obj: 1.066309 loss_cls: 0.022990 loss: 3.727151 eta: 2 days, 12:07:38 batch_cost: 22.9907 data_cost: 22.4361 ips: 0.4350 images/s
        [07/04 09:22:23] ppdet.engine INFO: Epoch: [7] [280/526] learning_rate: 0.000099 loss_xy: 0.300287 loss_wh: 0.445249 loss_iou: 1.892323 loss_iou_aware: 0.518732 loss_obj: 1.241344 loss_cls: 0.021275 loss: 4.325629 eta: 2 days, 12:01:33 batch_cost: 17.0099 data_cost: 16.4547 ips: 0.5879 images/s
        [07/04 09:23:40] ppdet.engine INFO: Epoch: [7] [285/526] learning_rate: 0.000099 loss_xy: 0.318457 loss_wh: 0.433496 loss_iou: 2.020114 loss_iou_aware: 0.520272 loss_obj: 1.071694 loss_cls: 0.028833 loss: 4.433629 eta: 2 days, 11:50:14 batch_cost: 15.4368 data_cost: 14.8970 ips: 0.6478 images/s
        [07/04 09:25:00] ppdet.engine INFO: Epoch: [7] [290/526] learning_rate: 0.000099 loss_xy: 0.254801 loss_wh: 0.425040 loss_iou: 1.802317 loss_iou_aware: 0.508445 loss_obj: 0.953223 loss_cls: 0.023918 loss: 3.953820 eta: 2 days, 11:40:22 batch_cost: 15.7683 data_cost: 15.3503 ips: 0.6342 images/s
        [07/04 09:26:13] ppdet.engine INFO: Epoch: [7] [295/526] learning_rate: 0.000099 loss_xy: 0.328503 loss_wh: 0.424514 loss_iou: 2.008031 loss_iou_aware: 0.535249 loss_obj: 0.927587 loss_cls: 0.024442 loss: 4.536455 eta: 2 days, 11:26:58 batch_cost: 14.6182 data_cost: 14.1191 ips: 0.6841 images/s
        [07/04 09:27:54] ppdet.engine INFO: Epoch: [7] [300/526] learning_rate: 0.000100 loss_xy: 0.326333 loss_wh: 0.362505 loss_iou: 1.984050 loss_iou_aware: 0.516304 loss_obj: 1.194836 loss_cls: 0.026523 loss: 4.335548 eta: 2 days, 11:32:07 batch_cost: 20.1691 data_cost: 19.7483 ips: 0.4958 images/s
        [07/04 09:29:54] ppdet.engine INFO: Epoch: [7] [305/526] learning_rate: 0.000100 loss_xy: 0.359196 loss_wh: 0.549035 loss_iou: 2.009509 loss_iou_aware: 0.518590 loss_obj: 1.044420 loss_cls: 0.022281 loss: 4.509370 eta: 2 days, 11:49:12 batch_cost: 23.9580 data_cost: 23.5868 ips: 0.4174 images/s
        [07/04 09:31:20] ppdet.engine INFO: Epoch: [7] [310/526] learning_rate: 0.000100 loss_xy: 0.307913 loss_wh: 0.334253 loss_iou: 1.817642 loss_iou_aware: 0.485536 loss_obj: 0.968255 loss_cls: 0.021530 loss: 4.017389 eta: 2 days, 11:43:58 batch_cost: 17.0872 data_cost: 16.6231 ips: 0.5852 images/s
        [07/04 09:32:56] ppdet.engine INFO: Epoch: [7] [315/526] learning_rate: 0.000100 loss_xy: 0.320267 loss_wh: 0.418962 loss_iou: 1.932071 loss_iou_aware: 0.513946 loss_obj: 1.005213 loss_cls: 0.031163 loss: 4.379160 eta: 2 days, 11:45:20 batch_cost: 19.1686 data_cost: 18.5957 ips: 0.5217 images/s
        [07/04 09:34:13] ppdet.engine INFO: Epoch: [7] [320/526] learning_rate: 0.000100 loss_xy: 0.345757 loss_wh: 0.446112 loss_iou: 1.939461 loss_iou_aware: 0.510864 loss_obj: 1.503500 loss_cls: 0.034992 loss: 4.843706 eta: 2 days, 11:34:55 batch_cost: 15.3495 data_cost: 14.8307 ips: 0.6515 images/s
        [07/04 09:35:34] ppdet.engine INFO: Epoch: [7] [325/526] learning_rate: 0.000100 loss_xy: 0.304388 loss_wh: 0.358780 loss_iou: 1.702022 loss_iou_aware: 0.490853 loss_obj: 1.189819 loss_cls: 0.024433 loss: 3.943646 eta: 2 days, 11:26:45 batch_cost: 16.0001 data_cost: 15.5196 ips: 0.6250 images/s
        [07/04 09:36:51] ppdet.engine INFO: Epoch: [7] [330/526] learning_rate: 0.000100 loss_xy: 0.299712 loss_wh: 0.402002 loss_iou: 1.856949 loss_iou_aware: 0.502581 loss_obj: 1.058779 loss_cls: 0.027956 loss: 4.215559 eta: 2 days, 11:17:04 batch_cost: 15.4246 data_cost: 14.9509 ips: 0.6483 images/s
        [07/04 09:38:12] ppdet.engine INFO: Epoch: [7] [335/526] learning_rate: 0.000100 loss_xy: 0.239440 loss_wh: 0.376075 loss_iou: 1.680766 loss_iou_aware: 0.508127 loss_obj: 0.761994 loss_cls: 0.020704 loss: 3.391821 eta: 2 days, 11:10:01 batch_cost: 16.2350 data_cost: 15.7449 ips: 0.6160 images/s
        [07/04 09:39:28] ppdet.engine INFO: Epoch: [7] [340/526] learning_rate: 0.000100 loss_xy: 0.340304 loss_wh: 0.410194 loss_iou: 2.171657 loss_iou_aware: 0.547689 loss_obj: 1.093696 loss_cls: 0.020441 loss: 4.606133 eta: 2 days, 10:59:33 batch_cost: 14.9926 data_cost: 14.4086 ips: 0.6670 images/s
        [07/04 09:41:06] ppdet.engine INFO: Epoch: [7] [345/526] learning_rate: 0.000100 loss_xy: 0.370058 loss_wh: 0.394630 loss_iou: 1.780612 loss_iou_aware: 0.504826 loss_obj: 1.263272 loss_cls: 0.036595 loss: 4.239148 eta: 2 days, 11:02:28 batch_cost: 19.6233 data_cost: 19.1833 ips: 0.5096 images/s
        [07/04 09:42:24] ppdet.engine INFO: Epoch: [7] [350/526] learning_rate: 0.000100 loss_xy: 0.295433 loss_wh: 0.375704 loss_iou: 1.638628 loss_iou_aware: 0.481048 loss_obj: 0.834188 loss_cls: 0.030682 loss: 3.879297 eta: 2 days, 10:53:56 batch_cost: 15.5653 data_cost: 14.9037 ips: 0.6425 images/s
        [07/04 09:43:43] ppdet.engine INFO: Epoch: [7] [355/526] learning_rate: 0.000100 loss_xy: 0.331520 loss_wh: 0.398464 loss_iou: 1.847210 loss_iou_aware: 0.484037 loss_obj: 1.020255 loss_cls: 0.026583 loss: 4.020402 eta: 2 days, 10:46:01 batch_cost: 15.7150 data_cost: 15.1785 ips: 0.6363 images/s
        [07/04 09:45:09] ppdet.engine INFO: Epoch: [7] [360/526] learning_rate: 0.000100 loss_xy: 0.392244 loss_wh: 0.509356 loss_iou: 2.207922 loss_iou_aware: 0.555670 loss_obj: 0.962765 loss_cls: 0.031678 loss: 4.449707 eta: 2 days, 10:41:59 batch_cost: 17.0837 data_cost: 16.5275 ips: 0.5854 images/s
        [07/04 09:46:30] ppdet.engine INFO: Epoch: [7] [365/526] learning_rate: 0.000100 loss_xy: 0.356287 loss_wh: 0.450955 loss_iou: 1.979756 loss_iou_aware: 0.526763 loss_obj: 1.359337 loss_cls: 0.030232 loss: 4.683675 eta: 2 days, 10:35:32 batch_cost: 16.1464 data_cost: 15.5787 ips: 0.6193 images/s
        [07/04 09:47:48] ppdet.engine INFO: Epoch: [7] [370/526] learning_rate: 0.000100 loss_xy: 0.278520 loss_wh: 0.497630 loss_iou: 2.021318 loss_iou_aware: 0.524387 loss_obj: 0.841166 loss_cls: 0.019115 loss: 4.222310 eta: 2 days, 10:27:40 batch_cost: 15.5616 data_cost: 15.0782 ips: 0.6426 images/s
        [07/04 09:49:09] ppdet.engine INFO: Epoch: [7] [375/526] learning_rate: 0.000100 loss_xy: 0.278704 loss_wh: 0.372895 loss_iou: 1.831522 loss_iou_aware: 0.504282 loss_obj: 0.992580 loss_cls: 0.034275 loss: 4.029595 eta: 2 days, 10:21:11 batch_cost: 16.0230 data_cost: 15.4811 ips: 0.6241 images/s
        [07/04 09:50:36] ppdet.engine INFO: Epoch: [7] [380/526] learning_rate: 0.000100 loss_xy: 0.338182 loss_wh: 0.434440 loss_iou: 1.876887 loss_iou_aware: 0.521846 loss_obj: 1.153687 loss_cls: 0.045839 loss: 4.690042 eta: 2 days, 10:18:26 batch_cost: 17.4334 data_cost: 16.8767 ips: 0.5736 images/s
        [07/04 09:51:49] ppdet.engine INFO: Epoch: [7] [385/526] learning_rate: 0.000100 loss_xy: 0.248263 loss_wh: 0.384693 loss_iou: 1.915816 loss_iou_aware: 0.498569 loss_obj: 1.190078 loss_cls: 0.026199 loss: 4.364306 eta: 2 days, 10:08:15 batch_cost: 14.4721 data_cost: 14.0581 ips: 0.6910 images/s
        [07/04 09:53:17] ppdet.engine INFO: Epoch: [7] [390/526] learning_rate: 0.000100 loss_xy: 0.325607 loss_wh: 0.389441 loss_iou: 1.814340 loss_iou_aware: 0.478424 loss_obj: 0.989821 loss_cls: 0.037373 loss: 4.097196 eta: 2 days, 10:05:50 batch_cost: 17.5010 data_cost: 17.0451 ips: 0.5714 images/s
        [07/04 09:54:33] ppdet.engine INFO: Epoch: [7] [395/526] learning_rate: 0.000100 loss_xy: 0.267582 loss_wh: 0.366212 loss_iou: 1.740673 loss_iou_aware: 0.474284 loss_obj: 0.783790 loss_cls: 0.031453 loss: 4.243313 eta: 2 days, 9:58:03 batch_cost: 15.3064 data_cost: 14.8691 ips: 0.6533 images/s
        [07/04 09:55:45] ppdet.engine INFO: Epoch: [7] [400/526] learning_rate: 0.000100 loss_xy: 0.331384 loss_wh: 0.403433 loss_iou: 1.862186 loss_iou_aware: 0.513111 loss_obj: 1.094553 loss_cls: 0.041456 loss: 4.334839 eta: 2 days, 9:47:55 batch_cost: 14.2775 data_cost: 13.7824 ips: 0.7004 images/s
        [07/04 09:56:57] ppdet.engine INFO: Epoch: [7] [405/526] learning_rate: 0.000100 loss_xy: 0.324505 loss_wh: 0.365570 loss_iou: 1.724869 loss_iou_aware: 0.471305 loss_obj: 1.085927 loss_cls: 0.038073 loss: 3.856627 eta: 2 days, 9:38:09 batch_cost: 14.3350 data_cost: 13.7870 ips: 0.6976 images/s
        [07/04 09:58:19] ppdet.engine INFO: Epoch: [7] [410/526] learning_rate: 0.000100 loss_xy: 0.313151 loss_wh: 0.331072 loss_iou: 1.792507 loss_iou_aware: 0.495526 loss_obj: 0.929585 loss_cls: 0.024885 loss: 3.602784 eta: 2 days, 9:33:14 batch_cost: 16.2931 data_cost: 15.8511 ips: 0.6138 images/s
        [07/04 09:59:29] ppdet.engine INFO: Epoch: [7] [415/526] learning_rate: 0.000100 loss_xy: 0.251054 loss_wh: 0.400270 loss_iou: 1.779956 loss_iou_aware: 0.498480 loss_obj: 1.173726 loss_cls: 0.022959 loss: 4.318279 eta: 2 days, 9:23:09 batch_cost: 14.0465 data_cost: 13.5949 ips: 0.7119 images/s
        [07/04 10:00:47] ppdet.engine INFO: Epoch: [7] [420/526] learning_rate: 0.000100 loss_xy: 0.287021 loss_wh: 0.426930 loss_iou: 1.749973 loss_iou_aware: 0.490117 loss_obj: 1.003443 loss_cls: 0.024965 loss: 3.954142 eta: 2 days, 9:16:27 batch_cost: 15.4250 data_cost: 14.8234 ips: 0.6483 images/s
        [07/04 10:02:02] ppdet.engine INFO: Epoch: [7] [425/526] learning_rate: 0.000100 loss_xy: 0.324225 loss_wh: 0.457792 loss_iou: 1.916178 loss_iou_aware: 0.540017 loss_obj: 1.322351 loss_cls: 0.035252 loss: 4.611326 eta: 2 days, 9:08:40 batch_cost: 14.8883 data_cost: 14.2352 ips: 0.6717 images/s
        [07/04 10:03:45] ppdet.engine INFO: Epoch: [7] [430/526] learning_rate: 0.000100 loss_xy: 0.271839 loss_wh: 0.428785 loss_iou: 1.930393 loss_iou_aware: 0.525617 loss_obj: 0.992150 loss_cls: 0.026406 loss: 3.970448 eta: 2 days, 9:14:09 batch_cost: 20.7109 data_cost: 20.2307 ips: 0.4828 images/s
        [07/04 10:05:02] ppdet.engine INFO: Epoch: [7] [435/526] learning_rate: 0.000100 loss_xy: 0.253771 loss_wh: 0.457005 loss_iou: 2.026963 loss_iou_aware: 0.526612 loss_obj: 1.207673 loss_cls: 0.021741 loss: 4.575372 eta: 2 days, 9:07:13 batch_cost: 15.2098 data_cost: 14.7445 ips: 0.6575 images/s
        [07/04 10:06:18] ppdet.engine INFO: Epoch: [7] [440/526] learning_rate: 0.000100 loss_xy: 0.273718 loss_wh: 0.371457 loss_iou: 1.711582 loss_iou_aware: 0.500316 loss_obj: 1.095438 loss_cls: 0.032815 loss: 4.214133 eta: 2 days, 9:00:08 batch_cost: 15.0809 data_cost: 14.5061 ips: 0.6631 images/s
        [07/04 10:07:34] ppdet.engine INFO: Epoch: [7] [445/526] learning_rate: 0.000100 loss_xy: 0.351868 loss_wh: 0.449821 loss_iou: 2.028580 loss_iou_aware: 0.528968 loss_obj: 0.839146 loss_cls: 0.030612 loss: 4.226998 eta: 2 days, 8:53:31 batch_cost: 15.2354 data_cost: 14.7676 ips: 0.6564 images/s
        [07/04 10:08:53] ppdet.engine INFO: Epoch: [7] [450/526] learning_rate: 0.000100 loss_xy: 0.285920 loss_wh: 0.373788 loss_iou: 1.781815 loss_iou_aware: 0.499203 loss_obj: 0.868353 loss_cls: 0.024865 loss: 4.251136 eta: 2 days, 8:47:54 batch_cost: 15.6484 data_cost: 15.1028 ips: 0.6390 images/s
        [07/04 10:10:16] ppdet.engine INFO: Epoch: [7] [455/526] learning_rate: 0.000100 loss_xy: 0.339106 loss_wh: 0.368719 loss_iou: 1.665999 loss_iou_aware: 0.474975 loss_obj: 0.784573 loss_cls: 0.029952 loss: 3.682681 eta: 2 days, 8:44:17 batch_cost: 16.5439 data_cost: 15.9818 ips: 0.6045 images/s
        [07/04 10:11:25] ppdet.engine INFO: Epoch: [7] [460/526] learning_rate: 0.000100 loss_xy: 0.251786 loss_wh: 0.378192 loss_iou: 1.864195 loss_iou_aware: 0.483405 loss_obj: 0.957796 loss_cls: 0.026653 loss: 3.787965 eta: 2 days, 8:35:08 batch_cost: 13.8853 data_cost: 13.4254 ips: 0.7202 images/s
        [07/04 10:12:40] ppdet.engine INFO: Epoch: [7] [465/526] learning_rate: 0.000100 loss_xy: 0.286353 loss_wh: 0.404571 loss_iou: 1.750444 loss_iou_aware: 0.503677 loss_obj: 1.218117 loss_cls: 0.033367 loss: 4.183507 eta: 2 days, 8:28:20 batch_cost: 14.9357 data_cost: 14.4423 ips: 0.6695 images/s
        [07/04 10:14:21] ppdet.engine INFO: Epoch: [7] [470/526] learning_rate: 0.000100 loss_xy: 0.320658 loss_wh: 0.449462 loss_iou: 1.972115 loss_iou_aware: 0.539170 loss_obj: 1.225012 loss_cls: 0.039561 loss: 4.532922 eta: 2 days, 8:32:06 batch_cost: 20.0154 data_cost: 19.4809 ips: 0.4996 images/s
        [07/04 10:15:37] ppdet.engine INFO: Epoch: [7] [475/526] learning_rate: 0.000100 loss_xy: 0.283869 loss_wh: 0.339919 loss_iou: 1.651963 loss_iou_aware: 0.474857 loss_obj: 0.907367 loss_cls: 0.030523 loss: 3.888017 eta: 2 days, 8:25:51 batch_cost: 15.1474 data_cost: 14.6123 ips: 0.6602 images/s
        [07/04 10:16:54] ppdet.engine INFO: Epoch: [7] [480/526] learning_rate: 0.000100 loss_xy: 0.277359 loss_wh: 0.431755 loss_iou: 2.062310 loss_iou_aware: 0.538239 loss_obj: 0.830686 loss_cls: 0.027108 loss: 4.164532 eta: 2 days, 8:20:12 batch_cost: 15.3964 data_cost: 14.9636 ips: 0.6495 images/s
        [07/04 10:18:11] ppdet.engine INFO: Epoch: [7] [485/526] learning_rate: 0.000100 loss_xy: 0.340683 loss_wh: 0.480706 loss_iou: 2.033728 loss_iou_aware: 0.525150 loss_obj: 0.899671 loss_cls: 0.025439 loss: 4.208438 eta: 2 days, 8:14:38 batch_cost: 15.3943 data_cost: 14.9145 ips: 0.6496 images/s
        [07/04 10:20:02] ppdet.engine INFO: Epoch: [7] [490/526] learning_rate: 0.000100 loss_xy: 0.277140 loss_wh: 0.292697 loss_iou: 1.549559 loss_iou_aware: 0.444487 loss_obj: 0.753578 loss_cls: 0.024200 loss: 3.569586 eta: 2 days, 8:22:25 batch_cost: 22.1215 data_cost: 21.6274 ips: 0.4520 images/s
        [07/04 10:21:17] ppdet.engine INFO: Epoch: [7] [495/526] learning_rate: 0.000100 loss_xy: 0.333376 loss_wh: 0.335529 loss_iou: 1.688137 loss_iou_aware: 0.490739 loss_obj: 1.095548 loss_cls: 0.030452 loss: 3.860460 eta: 2 days, 8:15:53 batch_cost: 14.8795 data_cost: 14.3070 ips: 0.6721 images/s
        [07/04 10:22:28] ppdet.engine INFO: Epoch: [7] [500/526] learning_rate: 0.000100 loss_xy: 0.272487 loss_wh: 0.337801 loss_iou: 1.844177 loss_iou_aware: 0.507398 loss_obj: 0.817930 loss_cls: 0.023127 loss: 3.775109 eta: 2 days, 8:08:08 batch_cost: 14.1955 data_cost: 13.6942 ips: 0.7044 images/s
        [07/04 10:23:58] ppdet.engine INFO: Epoch: [7] [505/526] learning_rate: 0.000100 loss_xy: 0.265068 loss_wh: 0.342254 loss_iou: 1.812132 loss_iou_aware: 0.487518 loss_obj: 0.958714 loss_cls: 0.016438 loss: 3.927074 eta: 2 days, 8:07:40 batch_cost: 17.9371 data_cost: 17.4474 ips: 0.5575 images/s
        [07/04 10:25:12] ppdet.engine INFO: Epoch: [7] [510/526] learning_rate: 0.000100 loss_xy: 0.262330 loss_wh: 0.394822 loss_iou: 1.784417 loss_iou_aware: 0.479115 loss_obj: 0.786947 loss_cls: 0.018559 loss: 3.610360 eta: 2 days, 8:00:53 batch_cost: 14.6124 data_cost: 14.1532 ips: 0.6844 images/s
        [07/04 10:26:29] ppdet.engine INFO: Epoch: [7] [515/526] learning_rate: 0.000100 loss_xy: 0.302016 loss_wh: 0.495736 loss_iou: 2.059801 loss_iou_aware: 0.523866 loss_obj: 1.135634 loss_cls: 0.022543 loss: 4.354139 eta: 2 days, 7:55:39 batch_cost: 15.3787 data_cost: 14.9810 ips: 0.6503 images/s
        [07/04 10:27:42] ppdet.engine INFO: Epoch: [7] [520/526] learning_rate: 0.000100 loss_xy: 0.258651 loss_wh: 0.380378 loss_iou: 1.677820 loss_iou_aware: 0.480484 loss_obj: 0.961978 loss_cls: 0.028227 loss: 3.891515 eta: 2 days, 7:49:07 batch_cost: 14.6321 data_cost: 14.1662 ips: 0.6834 images/s
        [07/04 10:28:54] ppdet.engine INFO: Epoch: [7] [525/526] learning_rate: 0.000100 loss_xy: 0.368762 loss_wh: 0.374008 loss_iou: 1.781887 loss_iou_aware: 0.496053 loss_obj: 0.843844 loss_cls: 0.025339 loss: 3.943705 eta: 2 days, 7:41:54 batch_cost: 14.2157 data_cost: 13.6629 ips: 0.7034 images/s
        [07/04 10:28:59] reader WARNING: fail to map sample transform [Decode_15e5d1] with error: [Errno 5] Input/output error and stack:
        Traceback (most recent call last):
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/reader.py", line 54, in __call__
            data = f(data)
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 103, in __call__
            sample[i] = self.apply(sample[i], context)
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 123, in apply
            sample['image'] = f.read()
        OSError: [Errno 5] Input/output error

        Exception in thread Thread-2:
        Traceback (most recent call last):
          File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
            self.run()
          File "/usr/lib/python3.7/threading.py", line 870, in run
            self._target(*self._args, **self._kwargs)
          File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/dataloader_iter.py", line 216, in _thread_loop
            self._thread_done_event)
          File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/fetcher.py", line 121, in fetch
            data.append(self.dataset[idx])
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/source/dataset.py", line 91, in __getitem__
            return self.transform(roidb)
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/reader.py", line 60, in __call__
            raise e
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/reader.py", line 54, in __call__
            data = f(data)
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 103, in __call__
            sample[i] = self.apply(sample[i], context)
          File "/content/drive/MyDrive/UIT/Four_Year/Term_2/DS505.M21/Code/PaddleDetection/ppdet/data/transform/operators.py", line 123, in apply
            sample['image'] = f.read()
        OSError: [Errno 5] Input/output error

        [07/04 10:29:19] ppdet.utils.checkpoint INFO: Save checkpoint: output/ppyolo_r50vd_dcn_1x_coco
        loading annotations into memory...
        Done (t=0.90s)
        creating index...
        index created!
        loading annotations into memory...
        Done (t=0.02s)
        creating index...
        index created!
        [07/04 10:36:36] ppdet.engine INFO: Eval iter: 0
        [07/04 10:50:25] ppdet.engine INFO: Eval iter: 100
        [07/04 10:53:58] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
        loading annotations into memory...
        Done (t=0.85s)
        creating index...
        index created!
        [07/04 10:54:01] ppdet.metrics.coco_utils INFO: Start evaluate...
        Loading and preparing results...
        DONE (t=0.06s)
        creating index...
        index created!
        Running per image evaluation...
        Evaluate annotation type *bbox*
        DONE (t=0.85s).
        Accumulating evaluation results...
        DONE (t=0.16s).
         Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.771
         Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 1.000
         Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.971
         Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.666
         Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.797
         Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
         Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.817
         Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.838
         Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.838
         Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.793
         Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.844
         Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
        [07/04 10:54:02] ppdet.engine INFO: Total sample number: 1000, averge FPS: 0.9547666875704138
        [07/04 10:54:02] ppdet.engine INFO: Best test bbox ap is 0.771.
        [07/04 10:54:32] ppdet.utils.checkpoint INFO: Save checkpoint: output/ppyolo_r50vd_dcn_1x_coco