search

chensnathan / YOLOF

You Only Look One-level Feature (YOLOF), CVPR2021, Detectron2
MIT License
271 stars 28 forks source link

AssertionError: bad box: x1 larger than x2 #40

Open raj-4444 opened 2 years ago

raj-4444 commented 2 years ago

[07/11 18:24:42 d2.engine.train_loop]: Starting training from iteration 0 [07/11 18:24:52 d2.utils.events]: eta: 7:53:07 iter: 19 total_loss: 1.522 loss_cls: 0.2083 loss_box_reg: 0.0001553 loss_mask: 0.6411 loss_rpn_cls: 0.6729 loss_rpn_loc: 0.02641 time: 0.4052 data_time: 0.0427 lr: 0.00019981 max_mem: 6824M [07/11 18:25:00 d2.utils.events]: eta: 7:58:34 iter: 39 total_loss: 0.826 loss_cls: 0.02481 loss_box_reg: 0.001338 loss_mask: 0.3514 loss_rpn_cls: 0.3847 loss_rpn_loc: 0.0112 time: 0.4081 data_time: 0.0091 lr: 0.00039961 max_mem: 7054M [07/11 18:25:09 d2.utils.events]: eta: 8:03:20 iter: 59 total_loss: 0.4657 loss_cls: 0.0525 loss_box_reg: 0.00528 loss_mask: 0.2328 loss_rpn_cls: 0.1254 loss_rpn_loc: 0.01294 time: 0.4081 data_time: 0.0062 lr: 0.00059941 max_mem: 7054M [07/11 18:25:17 d2.utils.events]: eta: 8:10:51 iter: 79 total_loss: 0.5412 loss_cls: 0.1219 loss_box_reg: 0.01392 loss_mask: 0.1982 loss_rpn_cls: 0.1097 loss_rpn_loc: 0.04281 time: 0.4142 data_time: 0.0080 lr: 0.00079921 max_mem: 8390M [07/11 18:25:26 d2.utils.events]: eta: 8:12:44 iter: 99 total_loss: 0.3733 loss_cls: 0.09589 loss_box_reg: 0.03402 loss_mask: 0.1765 loss_rpn_cls: 0.07032 loss_rpn_loc: 0.02893 time: 0.4211 data_time: 0.0077 lr: 0.00099901 max_mem: 10375M [07/11 18:25:35 d2.utils.events]: eta: 8:17:41 iter: 119 total_loss: 0.3755 loss_cls: 0.08576 loss_box_reg: 0.03418 loss_mask: 0.1741 loss_rpn_cls: 0.0485 loss_rpn_loc: 0.02191 time: 0.4233 data_time: 0.0083 lr: 0.0011988 max_mem: 10375M [07/11 18:25:44 d2.utils.events]: eta: 8:24:34 iter: 139 total_loss: 0.3946 loss_cls: 0.09633 loss_box_reg: 0.0391 loss_mask: 0.1771 loss_rpn_cls: 0.04919 loss_rpn_loc: 0.04056 time: 0.4265 data_time: 0.0084 lr: 0.0013986 max_mem: 10375M [07/11 18:25:53 d2.utils.events]: eta: 8:28:27 iter: 159 total_loss: 0.4053 loss_cls: 0.08783 loss_box_reg: 0.03864 loss_mask: 0.1736 loss_rpn_cls: 0.04172 loss_rpn_loc: 0.04759 time: 0.4287 data_time: 0.0093 lr: 0.0015984 max_mem: 10375M [07/11 18:26:02 d2.utils.events]: eta: 8:28:45 iter: 179 total_loss: 0.4056 loss_cls: 0.1044 loss_box_reg: 0.03753 loss_mask: 0.1796 loss_rpn_cls: 0.03264 loss_rpn_loc: 0.0305 time: 0.4302 data_time: 0.0074 lr: 0.0017982 max_mem: 10375M [07/11 18:26:11 d2.utils.events]: eta: 8:28:40 iter: 199 total_loss: 0.3904 loss_cls: 0.08258 loss_box_reg: 0.02545 loss_mask: 0.1929 loss_rpn_cls: 0.03777 loss_rpn_loc: 0.02238 time: 0.4302 data_time: 0.0071 lr: 0.001998 max_mem: 10375M [07/11 18:26:20 d2.utils.events]: eta: 8:30:04 iter: 219 total_loss: 0.3945 loss_cls: 0.09928 loss_box_reg: 0.02967 loss_mask: 0.1953 loss_rpn_cls: 0.0315 loss_rpn_loc: 0.03111 time: 0.4312 data_time: 0.0092 lr: 0.0021978 max_mem: 10375M [07/11 18:26:28 d2.utils.events]: eta: 8:29:56 iter: 239 total_loss: 0.3698 loss_cls: 0.08268 loss_box_reg: 0.02776 loss_mask: 0.1708 loss_rpn_cls: 0.0224 loss_rpn_loc: 0.04258 time: 0.4310 data_time: 0.0067 lr: 0.0023976 max_mem: 10375M [07/11 18:26:37 d2.utils.events]: eta: 8:30:41 iter: 259 total_loss: 0.3101 loss_cls: 0.05248 loss_box_reg: 0.01284 loss_mask: 0.1519 loss_rpn_cls: 0.01409 loss_rpn_loc: 0.02101 time: 0.4315 data_time: 0.0078 lr: 0.0025974 max_mem: 10375M [07/11 18:26:46 d2.utils.events]: eta: 8:29:39 iter: 279 total_loss: 0.3361 loss_cls: 0.07323 loss_box_reg: 0.01987 loss_mask: 0.1451 loss_rpn_cls: 0.02021 loss_rpn_loc: 0.0349 time: 0.4326 data_time: 0.0072 lr: 0.0027972 max_mem: 10375M [07/11 18:26:55 d2.utils.events]: eta: 8:28:24 iter: 299 total_loss: 0.2654 loss_cls: 0.06387 loss_box_reg: 0.01594 loss_mask: 0.1451 loss_rpn_cls: 0.01477 loss_rpn_loc: 0.01315 time: 0.4324 data_time: 0.0065 lr: 0.002997 max_mem: 10375M [07/11 18:27:04 d2.utils.events]: eta: 8:30:20 iter: 319 total_loss: 0.3898 loss_cls: 0.08069 loss_box_reg: 0.02259 loss_mask: 0.1694 loss_rpn_cls: 0.02082 loss_rpn_loc: 0.03517 time: 0.4336 data_time: 0.0093 lr: 0.0031968 max_mem: 10375M [07/11 18:27:13 d2.utils.events]: eta: 8:30:33 iter: 339 total_loss: 0.3345 loss_cls: 0.08011 loss_box_reg: 0.02296 loss_mask: 0.161 loss_rpn_cls: 0.0221 loss_rpn_loc: 0.04661 time: 0.4344 data_time: 0.0076 lr: 0.0033966 max_mem: 10375M [07/11 18:27:22 d2.utils.events]: eta: 8:30:21 iter: 359 total_loss: 0.2744 loss_cls: 0.0485 loss_box_reg: 0.01726 loss_mask: 0.1293 loss_rpn_cls: 0.01705 loss_rpn_loc: 0.03203 time: 0.4351 data_time: 0.0077 lr: 0.0035964 max_mem: 10375M [07/11 18:27:31 d2.utils.events]: eta: 8:30:16 iter: 379 total_loss: 0.2558 loss_cls: 0.06494 loss_box_reg: 0.01695 loss_mask: 0.1368 loss_rpn_cls: 0.008527 loss_rpn_loc: 0.01464 time: 0.4352 data_time: 0.0068 lr: 0.0037962 max_mem: 10375M [07/11 18:27:40 d2.utils.events]: eta: 8:30:04 iter: 399 total_loss: 0.246 loss_cls: 0.04827 loss_box_reg: 0.01576 loss_mask: 0.1197 loss_rpn_cls: 0.01373 loss_rpn_loc: 0.01939 time: 0.4349 data_time: 0.0070 lr: 0.003996 max_mem: 10375M [07/11 18:27:48 d2.utils.events]: eta: 8:29:17 iter: 419 total_loss: 0.2877 loss_cls: 0.06765 loss_box_reg: 0.01713 loss_mask: 0.156 loss_rpn_cls: 0.01482 loss_rpn_loc: 0.0207 time: 0.4345 data_time: 0.0082 lr: 0.0041958 max_mem: 10375M [07/11 18:27:57 d2.utils.events]: eta: 8:29:30 iter: 439 total_loss: 0.4209 loss_cls: 0.1212 loss_box_reg: 0.02121 loss_mask: 0.1896 loss_rpn_cls: 0.01865 loss_rpn_loc: 0.03847 time: 0.4343 data_time: 0.0088 lr: 0.0043956 max_mem: 10375M [07/11 18:28:06 d2.utils.events]: eta: 8:29:38 iter: 459 total_loss: 0.4073 loss_cls: 0.1058 loss_box_reg: 0.02056 loss_mask: 0.163 loss_rpn_cls: 0.02908 loss_rpn_loc: 0.04771 time: 0.4348 data_time: 0.0089 lr: 0.0045954 max_mem: 10375M [07/11 18:28:15 d2.utils.events]: eta: 8:30:15 iter: 479 total_loss: 0.3359 loss_cls: 0.08171 loss_box_reg: 0.0161 loss_mask: 0.1719 loss_rpn_cls: 0.02255 loss_rpn_loc: 0.03287 time: 0.4353 data_time: 0.0076 lr: 0.0047952 max_mem: 10375M [07/11 18:28:24 d2.utils.events]: eta: 8:31:50 iter: 499 total_loss: 0.3511 loss_cls: 0.09287 loss_box_reg: 0.02037 loss_mask: 0.1579 loss_rpn_cls: 0.009786 loss_rpn_loc: 0.03596 time: 0.4361 data_time: 0.0082 lr: 0.004995 max_mem: 10375M [07/11 18:28:33 d2.utils.events]: eta: 8:31:51 iter: 519 total_loss: 0.3224 loss_cls: 0.08946 loss_box_reg: 0.02361 loss_mask: 0.1484 loss_rpn_cls: 0.0141 loss_rpn_loc: 0.03703 time: 0.4363 data_time: 0.0084 lr: 0.0051948 max_mem: 10375M [07/11 18:28:42 d2.utils.events]: eta: 8:33:04 iter: 539 total_loss: 0.3617 loss_cls: 0.06344 loss_box_reg: 0.01625 loss_mask: 0.2091 loss_rpn_cls: 0.01894 loss_rpn_loc: 0.02714 time: 0.4365 data_time: 0.0091 lr: 0.0053946 max_mem: 10375M [07/11 18:28:51 d2.utils.events]: eta: 8:33:12 iter: 559 total_loss: 0.3383 loss_cls: 0.09253 loss_box_reg: 0.01976 loss_mask: 0.1941 loss_rpn_cls: 0.0146 loss_rpn_loc: 0.02182 time: 0.4367 data_time: 0.0078 lr: 0.0055944 max_mem: 10375M [07/11 18:28:59 d2.utils.events]: eta: 8:33:44 iter: 579 total_loss: 0.3444 loss_cls: 0.08604 loss_box_reg: 0.01819 loss_mask: 0.1638 loss_rpn_cls: 0.01426 loss_rpn_loc: 0.02466 time: 0.4369 data_time: 0.0084 lr: 0.0057942 max_mem: 10375M [07/11 18:29:08 d2.utils.events]: eta: 8:32:55 iter: 599 total_loss: 0.3361 loss_cls: 0.07452 loss_box_reg: 0.01833 loss_mask: 0.148 loss_rpn_cls: 0.02424 loss_rpn_loc: 0.03891 time: 0.4367 data_time: 0.0082 lr: 0.005994 max_mem: 10375M [07/11 18:29:17 d2.utils.events]: eta: 8:32:46 iter: 619 total_loss: 0.3567 loss_cls: 0.0879 loss_box_reg: 0.01742 loss_mask: 0.1598 loss_rpn_cls: 0.009541 loss_rpn_loc: 0.01607 time: 0.4367 data_time: 0.0081 lr: 0.0061938 max_mem: 10375M [07/11 18:29:26 d2.utils.events]: eta: 8:32:22 iter: 639 total_loss: 0.2534 loss_cls: 0.07315 loss_box_reg: 0.01847 loss_mask: 0.1428 loss_rpn_cls: 0.006772 loss_rpn_loc: 0.01518 time: 0.4364 data_time: 0.0067 lr: 0.0063936 max_mem: 10375M [07/11 18:29:34 d2.utils.events]: eta: 8:32:10 iter: 659 total_loss: 0.3182 loss_cls: 0.06682 loss_box_reg: 0.01769 loss_mask: 0.178 loss_rpn_cls: 0.007947 loss_rpn_loc: 0.02726 time: 0.4364 data_time: 0.0077 lr: 0.0065934 max_mem: 10375M [07/11 18:29:43 d2.utils.events]: eta: 8:32:20 iter: 679 total_loss: 0.3215 loss_cls: 0.07152 loss_box_reg: 0.01705 loss_mask: 0.1694 loss_rpn_cls: 0.01041 loss_rpn_loc: 0.02417 time: 0.4365 data_time: 0.0086 lr: 0.0067932 max_mem: 10375M [07/11 18:29:52 d2.utils.events]: eta: 8:32:46 iter: 699 total_loss: 0.2565 loss_cls: 0.06282 loss_box_reg: 0.02091 loss_mask: 0.1476 loss_rpn_cls: 0.006757 loss_rpn_loc: 0.01335 time: 0.4368 data_time: 0.0075 lr: 0.006993 max_mem: 10375M [07/11 18:30:01 d2.utils.events]: eta: 8:32:03 iter: 719 total_loss: 0.2769 loss_cls: 0.07035 loss_box_reg: 0.01751 loss_mask: 0.1437 loss_rpn_cls: 0.009603 loss_rpn_loc: 0.02026 time: 0.4365 data_time: 0.0072 lr: 0.0071928 max_mem: 10375M [07/11 18:30:10 d2.utils.events]: eta: 8:32:29 iter: 739 total_loss: 0.3333 loss_cls: 0.09517 loss_box_reg: 0.0208 loss_mask: 0.1501 loss_rpn_cls: 0.01084 loss_rpn_loc: 0.03651 time: 0.4370 data_time: 0.0096 lr: 0.0073926 max_mem: 10375M [07/11 18:30:19 d2.utils.events]: eta: 8:32:32 iter: 759 total_loss: 0.3375 loss_cls: 0.08592 loss_box_reg: 0.02321 loss_mask: 0.1568 loss_rpn_cls: 0.008646 loss_rpn_loc: 0.0306 time: 0.4375 data_time: 0.0076 lr: 0.0075924 max_mem: 10375M [07/11 18:30:28 d2.utils.events]: eta: 8:32:09 iter: 779 total_loss: 0.3126 loss_cls: 0.06175 loss_box_reg: 0.0145 loss_mask: 0.1849 loss_rpn_cls: 0.01163 loss_rpn_loc: 0.01285 time: 0.4372 data_time: 0.0082 lr: 0.0077922 max_mem: 10375M [07/11 18:30:37 d2.utils.events]: eta: 8:32:03 iter: 799 total_loss: 0.261 loss_cls: 0.06443 loss_box_reg: 0.01786 loss_mask: 0.1488 loss_rpn_cls: 0.01037 loss_rpn_loc: 0.02081 time: 0.4375 data_time: 0.0085 lr: 0.007992 max_mem: 10375M [07/11 18:30:45 d2.utils.events]: eta: 8:31:52 iter: 819 total_loss: 0.2574 loss_cls: 0.05869 loss_box_reg: 0.01517 loss_mask: 0.1326 loss_rpn_cls: 0.008631 loss_rpn_loc: 0.02182 time: 0.4372 data_time: 0.0073 lr: 0.0081918 max_mem: 10375M [07/11 18:30:55 d2.utils.events]: eta: 8:31:43 iter: 839 total_loss: 0.333 loss_cls: 0.08468 loss_box_reg: 0.02047 loss_mask: 0.1637 loss_rpn_cls: 0.01798 loss_rpn_loc: 0.03197 time: 0.4375 data_time: 0.0077 lr: 0.0083916 max_mem: 10375M [07/11 18:31:03 d2.utils.events]: eta: 8:31:13 iter: 859 total_loss: 0.277 loss_cls: 0.07466 loss_box_reg: 0.01636 loss_mask: 0.1659 loss_rpn_cls: 0.006309 loss_rpn_loc: 0.0188 time: 0.4370 data_time: 0.0075 lr: 0.0085914 max_mem: 10375M [07/11 18:31:12 d2.utils.events]: eta: 8:31:40 iter: 879 total_loss: 0.3506 loss_cls: 0.09326 loss_box_reg: 0.02224 loss_mask: 0.1705 loss_rpn_cls: 0.01516 loss_rpn_loc: 0.03855 time: 0.4375 data_time: 0.0082 lr: 0.0087912 max_mem: 10375M [07/11 18:31:21 d2.utils.events]: eta: 8:31:31 iter: 899 total_loss: 0.2598 loss_cls: 0.07439 loss_box_reg: 0.01754 loss_mask: 0.1518 loss_rpn_cls: 0.005731 loss_rpn_loc: 0.01866 time: 0.4375 data_time: 0.0079 lr: 0.008991 max_mem: 10375M [07/11 18:31:30 d2.utils.events]: eta: 8:31:23 iter: 919 total_loss: 0.2764 loss_cls: 0.08773 loss_box_reg: 0.01714 loss_mask: 0.1435 loss_rpn_cls: 0.006121 loss_rpn_loc: 0.01353 time: 0.4374 data_time: 0.0077 lr: 0.0091908 max_mem: 10375M [07/11 18:31:39 d2.utils.events]: eta: 8:31:23 iter: 939 total_loss: 0.264 loss_cls: 0.08377 loss_box_reg: 0.01873 loss_mask: 0.1317 loss_rpn_cls: 0.008426 loss_rpn_loc: 0.02521 time: 0.4376 data_time: 0.0067 lr: 0.0093906 max_mem: 10375M [07/11 18:31:47 d2.utils.events]: eta: 8:31:05 iter: 959 total_loss: 0.249 loss_cls: 0.04743 loss_box_reg: 0.008316 loss_mask: 0.1123 loss_rpn_cls: 0.005139 loss_rpn_loc: 0.006432 time: 0.4374 data_time: 0.0071 lr: 0.0095904 max_mem: 10375M [07/11 18:31:56 d2.utils.events]: eta: 8:31:19 iter: 979 total_loss: 0.3152 loss_cls: 0.1002 loss_box_reg: 0.01721 loss_mask: 0.1611 loss_rpn_cls: 0.01409 loss_rpn_loc: 0.02712 time: 0.4376 data_time: 0.0092 lr: 0.0097902 max_mem: 10375M [07/11 18:32:05 d2.utils.events]: eta: 8:31:14 iter: 999 total_loss: 0.3033 loss_cls: 0.1042 loss_box_reg: 0.02027 loss_mask: 0.153 loss_rpn_cls: 0.01101 loss_rpn_loc: 0.02101 time: 0.4375 data_time: 0.0085 lr: 0.00999 max_mem: 10375M [07/11 18:32:14 d2.utils.events]: eta: 8:31:48 iter: 1019 total_loss: 0.3161 loss_cls: 0.08504 loss_box_reg: 0.02013 loss_mask: 0.1508 loss_rpn_cls: 0.01067 loss_rpn_loc: 0.03305 time: 0.4376 data_time: 0.0087 lr: 0.01 max_mem: 10375M [07/11 18:32:23 d2.utils.events]: eta: 8:33:14 iter: 1039 total_loss: 0.2638 loss_cls: 0.05909 loss_box_reg: 0.01622 loss_mask: 0.1384 loss_rpn_cls: 0.008573 loss_rpn_loc: 0.0123 time: 0.4377 data_time: 0.0087 lr: 0.01 max_mem: 10375M [07/11 18:32:31 d2.utils.events]: eta: 8:33:05 iter: 1059 total_loss: 0.2521 loss_cls: 0.06265 loss_box_reg: 0.0175 loss_mask: 0.1267 loss_rpn_cls: 0.006054 loss_rpn_loc: 0.0113 time: 0.4373 data_time: 0.0076 lr: 0.01 max_mem: 10375M [07/11 18:32:40 d2.utils.events]: eta: 8:34:00 iter: 1079 total_loss: 0.3011 loss_cls: 0.09204 loss_box_reg: 0.02271 loss_mask: 0.1404 loss_rpn_cls: 0.006699 loss_rpn_loc: 0.02894 time: 0.4375 data_time: 0.0080 lr: 0.01 max_mem: 10375M [07/11 18:32:49 d2.utils.events]: eta: 8:33:59 iter: 1099 total_loss: 0.3083 loss_cls: 0.09003 loss_box_reg: 0.01742 loss_mask: 0.1526 loss_rpn_cls: 0.01039 loss_rpn_loc: 0.02548 time: 0.4377 data_time: 0.0084 lr: 0.01 max_mem: 10375M [07/11 18:32:58 d2.utils.events]: eta: 8:33:51 iter: 1119 total_loss: 0.3485 loss_cls: 0.106 loss_box_reg: 0.02342 loss_mask: 0.1609 loss_rpn_cls: 0.008596 loss_rpn_loc: 0.02881 time: 0.4379 data_time: 0.0084 lr: 0.01 max_mem: 10375M [07/11 18:33:07 d2.utils.events]: eta: 8:33:28 iter: 1139 total_loss: 0.2774 loss_cls: 0.07644 loss_box_reg: 0.02027 loss_mask: 0.1465 loss_rpn_cls: 0.01067 loss_rpn_loc: 0.03347 time: 0.4379 data_time: 0.0093 lr: 0.01 max_mem: 10375M [07/11 18:33:16 d2.utils.events]: eta: 8:32:37 iter: 1159 total_loss: 0.2716 loss_cls: 0.07568 loss_box_reg: 0.02224 loss_mask: 0.1451 loss_rpn_cls: 0.005922 loss_rpn_loc: 0.01752 time: 0.4379 data_time: 0.0085 lr: 0.01 max_mem: 10375M [07/11 18:33:24 d2.utils.events]: eta: 8:32:29 iter: 1179 total_loss: 0.23 loss_cls: 0.07258 loss_box_reg: 0.01687 loss_mask: 0.125 loss_rpn_cls: 0.004398 loss_rpn_loc: 0.01229 time: 0.4378 data_time: 0.0071 lr: 0.01 max_mem: 10375M [07/11 18:33:33 d2.utils.events]: eta: 8:32:32 iter: 1199 total_loss: 0.3609 loss_cls: 0.08158 loss_box_reg: 0.02385 loss_mask: 0.1824 loss_rpn_cls: 0.01222 loss_rpn_loc: 0.03337 time: 0.4380 data_time: 0.0081 lr: 0.01 max_mem: 10375M [07/11 18:33:42 d2.utils.events]: eta: 8:32:01 iter: 1219 total_loss: 0.2507 loss_cls: 0.0415 loss_box_reg: 0.009657 loss_mask: 0.1466 loss_rpn_cls: 0.01578 loss_rpn_loc: 0.03665 time: 0.4378 data_time: 0.0085 lr: 0.01 max_mem: 10375M [07/11 18:33:52 d2.utils.events]: eta: 8:32:51 iter: 1239 total_loss: 0.3217 loss_cls: 0.0634 loss_box_reg: 0.01922 loss_mask: 0.1806 loss_rpn_cls: 0.01585 loss_rpn_loc: 0.0405 time: 0.4382 data_time: 0.0086 lr: 0.01 max_mem: 10824M [07/11 18:34:00 d2.utils.events]: eta: 8:32:42 iter: 1259 total_loss: 0.2819 loss_cls: 0.05084 loss_box_reg: 0.0185 loss_mask: 0.1524 loss_rpn_cls: 0.01591 loss_rpn_loc: 0.02505 time: 0.4381 data_time: 0.0075 lr: 0.01 max_mem: 10824M [07/11 18:34:09 d2.utils.events]: eta: 8:32:48 iter: 1279 total_loss: 0.3119 loss_cls: 0.05793 loss_box_reg: 0.01852 loss_mask: 0.1548 loss_rpn_cls: 0.01206 loss_rpn_loc: 0.03262 time: 0.4382 data_time: 0.0074 lr: 0.01 max_mem: 10824M [07/11 18:34:18 d2.utils.events]: eta: 8:32:56 iter: 1299 total_loss: 0.2717 loss_cls: 0.06279 loss_box_reg: 0.01653 loss_mask: 0.129 loss_rpn_cls: 0.01387 loss_rpn_loc: 0.02503 time: 0.4381 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:34:27 d2.utils.events]: eta: 8:32:24 iter: 1319 total_loss: 0.3009 loss_cls: 0.08376 loss_box_reg: 0.01819 loss_mask: 0.1446 loss_rpn_cls: 0.00914 loss_rpn_loc: 0.01798 time: 0.4379 data_time: 0.0077 lr: 0.01 max_mem: 10824M [07/11 18:34:36 d2.utils.events]: eta: 8:32:26 iter: 1339 total_loss: 0.2284 loss_cls: 0.06878 loss_box_reg: 0.01575 loss_mask: 0.1159 loss_rpn_cls: 0.01002 loss_rpn_loc: 0.02297 time: 0.4381 data_time: 0.0086 lr: 0.01 max_mem: 10824M [07/11 18:34:45 d2.utils.events]: eta: 8:32:33 iter: 1359 total_loss: 0.2712 loss_cls: 0.06552 loss_box_reg: 0.01762 loss_mask: 0.1461 loss_rpn_cls: 0.008251 loss_rpn_loc: 0.02143 time: 0.4383 data_time: 0.0090 lr: 0.01 max_mem: 10824M [07/11 18:34:53 d2.utils.events]: eta: 8:32:08 iter: 1379 total_loss: 0.242 loss_cls: 0.05389 loss_box_reg: 0.01578 loss_mask: 0.1311 loss_rpn_cls: 0.004269 loss_rpn_loc: 0.007739 time: 0.4381 data_time: 0.0070 lr: 0.01 max_mem: 10824M [07/11 18:35:02 d2.utils.events]: eta: 8:32:54 iter: 1399 total_loss: 0.2266 loss_cls: 0.05687 loss_box_reg: 0.01922 loss_mask: 0.1227 loss_rpn_cls: 0.006927 loss_rpn_loc: 0.0143 time: 0.4382 data_time: 0.0069 lr: 0.01 max_mem: 10824M [07/11 18:35:11 d2.utils.events]: eta: 8:32:58 iter: 1419 total_loss: 0.305 loss_cls: 0.08086 loss_box_reg: 0.02218 loss_mask: 0.1419 loss_rpn_cls: 0.006573 loss_rpn_loc: 0.01774 time: 0.4382 data_time: 0.0083 lr: 0.01 max_mem: 10824M [07/11 18:35:20 d2.utils.events]: eta: 8:32:48 iter: 1439 total_loss: 0.3102 loss_cls: 0.08404 loss_box_reg: 0.02221 loss_mask: 0.149 loss_rpn_cls: 0.01271 loss_rpn_loc: 0.04032 time: 0.4383 data_time: 0.0093 lr: 0.01 max_mem: 10824M [07/11 18:35:29 d2.utils.events]: eta: 8:32:28 iter: 1459 total_loss: 0.2952 loss_cls: 0.06519 loss_box_reg: 0.01814 loss_mask: 0.1636 loss_rpn_cls: 0.00677 loss_rpn_loc: 0.02062 time: 0.4381 data_time: 0.0078 lr: 0.01 max_mem: 10824M [07/11 18:35:38 d2.utils.events]: eta: 8:32:31 iter: 1479 total_loss: 0.2856 loss_cls: 0.06728 loss_box_reg: 0.01703 loss_mask: 0.1394 loss_rpn_cls: 0.006174 loss_rpn_loc: 0.02054 time: 0.4385 data_time: 0.0086 lr: 0.01 max_mem: 10824M [07/11 18:35:47 d2.utils.events]: eta: 8:31:51 iter: 1499 total_loss: 0.2848 loss_cls: 0.07849 loss_box_reg: 0.02293 loss_mask: 0.15 loss_rpn_cls: 0.009778 loss_rpn_loc: 0.0196 time: 0.4387 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:35:56 d2.utils.events]: eta: 8:31:17 iter: 1519 total_loss: 0.3311 loss_cls: 0.07853 loss_box_reg: 0.01755 loss_mask: 0.1597 loss_rpn_cls: 0.01422 loss_rpn_loc: 0.02734 time: 0.4387 data_time: 0.0080 lr: 0.01 max_mem: 10824M [07/11 18:36:05 d2.utils.events]: eta: 8:32:08 iter: 1539 total_loss: 0.314 loss_cls: 0.09528 loss_box_reg: 0.02204 loss_mask: 0.1555 loss_rpn_cls: 0.008758 loss_rpn_loc: 0.02828 time: 0.4390 data_time: 0.0095 lr: 0.01 max_mem: 10824M [07/11 18:36:14 d2.utils.events]: eta: 8:32:09 iter: 1559 total_loss: 0.2458 loss_cls: 0.05934 loss_box_reg: 0.01621 loss_mask: 0.1345 loss_rpn_cls: 0.005067 loss_rpn_loc: 0.018 time: 0.4390 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:36:24 d2.utils.events]: eta: 8:32:26 iter: 1579 total_loss: 0.356 loss_cls: 0.1014 loss_box_reg: 0.02792 loss_mask: 0.1477 loss_rpn_cls: 0.007858 loss_rpn_loc: 0.03938 time: 0.4394 data_time: 0.0095 lr: 0.01 max_mem: 10824M [07/11 18:36:33 d2.utils.events]: eta: 8:33:11 iter: 1599 total_loss: 0.2719 loss_cls: 0.08995 loss_box_reg: 0.02081 loss_mask: 0.1336 loss_rpn_cls: 0.007486 loss_rpn_loc: 0.04157 time: 0.4397 data_time: 0.0088 lr: 0.01 max_mem: 10824M [07/11 18:36:42 d2.utils.events]: eta: 8:33:36 iter: 1619 total_loss: 0.2727 loss_cls: 0.06473 loss_box_reg: 0.01508 loss_mask: 0.1268 loss_rpn_cls: 0.00439 loss_rpn_loc: 0.01114 time: 0.4399 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:36:51 d2.utils.events]: eta: 8:33:56 iter: 1639 total_loss: 0.3048 loss_cls: 0.07076 loss_box_reg: 0.01792 loss_mask: 0.1598 loss_rpn_cls: 0.01129 loss_rpn_loc: 0.0204 time: 0.4399 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:37:00 d2.utils.events]: eta: 8:34:01 iter: 1659 total_loss: 0.2019 loss_cls: 0.05851 loss_box_reg: 0.0147 loss_mask: 0.1105 loss_rpn_cls: 0.006341 loss_rpn_loc: 0.01156 time: 0.4400 data_time: 0.0094 lr: 0.01 max_mem: 10824M [07/11 18:37:09 d2.utils.events]: eta: 8:33:38 iter: 1679 total_loss: 0.2913 loss_cls: 0.07293 loss_box_reg: 0.01963 loss_mask: 0.1563 loss_rpn_cls: 0.01009 loss_rpn_loc: 0.01724 time: 0.4401 data_time: 0.0089 lr: 0.01 max_mem: 10824M [07/11 18:37:18 d2.utils.events]: eta: 8:33:19 iter: 1699 total_loss: 0.242 loss_cls: 0.0543 loss_box_reg: 0.01551 loss_mask: 0.1358 loss_rpn_cls: 0.006947 loss_rpn_loc: 0.0167 time: 0.4403 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:37:27 d2.utils.events]: eta: 8:33:28 iter: 1719 total_loss: 0.2673 loss_cls: 0.06433 loss_box_reg: 0.01698 loss_mask: 0.1393 loss_rpn_cls: 0.01027 loss_rpn_loc: 0.026 time: 0.4402 data_time: 0.0089 lr: 0.01 max_mem: 10824M [07/11 18:37:36 d2.utils.events]: eta: 8:33:12 iter: 1739 total_loss: 0.2919 loss_cls: 0.07103 loss_box_reg: 0.02 loss_mask: 0.1496 loss_rpn_cls: 0.006164 loss_rpn_loc: 0.01211 time: 0.4403 data_time: 0.0083 lr: 0.01 max_mem: 10824M [07/11 18:37:45 d2.utils.events]: eta: 8:33:05 iter: 1759 total_loss: 0.2686 loss_cls: 0.04505 loss_box_reg: 0.01351 loss_mask: 0.1634 loss_rpn_cls: 0.009903 loss_rpn_loc: 0.01445 time: 0.4404 data_time: 0.0077 lr: 0.01 max_mem: 10824M [07/11 18:37:54 d2.utils.events]: eta: 8:33:57 iter: 1779 total_loss: 0.2619 loss_cls: 0.06645 loss_box_reg: 0.01745 loss_mask: 0.1382 loss_rpn_cls: 0.006083 loss_rpn_loc: 0.01326 time: 0.4404 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:38:03 d2.utils.events]: eta: 8:33:51 iter: 1799 total_loss: 0.3124 loss_cls: 0.09307 loss_box_reg: 0.02187 loss_mask: 0.1663 loss_rpn_cls: 0.005512 loss_rpn_loc: 0.024 time: 0.4406 data_time: 0.0076 lr: 0.01 max_mem: 10824M [07/11 18:38:12 d2.utils.events]: eta: 8:33:42 iter: 1819 total_loss: 0.2443 loss_cls: 0.05758 loss_box_reg: 0.01611 loss_mask: 0.1355 loss_rpn_cls: 0.004745 loss_rpn_loc: 0.01235 time: 0.4404 data_time: 0.0072 lr: 0.01 max_mem: 10824M [07/11 18:38:21 d2.utils.events]: eta: 8:33:20 iter: 1839 total_loss: 0.2389 loss_cls: 0.05698 loss_box_reg: 0.01671 loss_mask: 0.1417 loss_rpn_cls: 0.008342 loss_rpn_loc: 0.01314 time: 0.4404 data_time: 0.0079 lr: 0.01 max_mem: 10824M [07/11 18:38:30 d2.utils.events]: eta: 8:33:53 iter: 1859 total_loss: 0.2556 loss_cls: 0.07939 loss_box_reg: 0.01739 loss_mask: 0.1392 loss_rpn_cls: 0.009146 loss_rpn_loc: 0.02361 time: 0.4407 data_time: 0.0086 lr: 0.01 max_mem: 10824M [07/11 18:38:39 d2.utils.events]: eta: 8:33:16 iter: 1879 total_loss: 0.268 loss_cls: 0.06665 loss_box_reg: 0.0187 loss_mask: 0.1206 loss_rpn_cls: 0.003925 loss_rpn_loc: 0.01439 time: 0.4408 data_time: 0.0082 lr: 0.01 max_mem: 10824M [07/11 18:38:48 d2.utils.events]: eta: 8:33:08 iter: 1899 total_loss: 0.2089 loss_cls: 0.05096 loss_box_reg: 0.01195 loss_mask: 0.1319 loss_rpn_cls: 0.00131 loss_rpn_loc: 0.005368 time: 0.4407 data_time: 0.0075 lr: 0.01 max_mem: 10824M [07/11 18:38:57 d2.utils.events]: eta: 8:33:05 iter: 1919 total_loss: 0.2939 loss_cls: 0.08017 loss_box_reg: 0.02089 loss_mask: 0.1447 loss_rpn_cls: 0.01021 loss_rpn_loc: 0.02003 time: 0.4408 data_time: 0.0080 lr: 0.01 max_mem: 10824M [07/11 18:39:06 d2.utils.events]: eta: 8:33:25 iter: 1939 total_loss: 0.2455 loss_cls: 0.06749 loss_box_reg: 0.01648 loss_mask: 0.1294 loss_rpn_cls: 0.004726 loss_rpn_loc: 0.02557 time: 0.4409 data_time: 0.0082 lr: 0.01 max_mem: 10824M

WARNING: skipped 801942 bytes of output

[07/12 01:31:55 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [RandomRotation(angle=[0.0, 360.0], interp=2)] [07/12 01:31:57 d2.data.common]: Serializing 234 elements to byte tensors and concatenating them all ... [07/12 01:31:57 d2.data.common]: Serialized dataset takes 1.69 MiB WARNING [07/12 01:31:57 d2.evaluation.coco_evaluation]: COCO Evaluator instantiated using config, this is deprecated behavior. Please pass in explicit arguments instead. [07/12 01:31:57 d2.evaluation.coco_evaluation]: Trying to convert 'dataset_test' to COCO format ... [07/12 01:31:57 d2.data.datasets.coco]: Converting annotations of dataset 'dataset_test' to COCO format ...) [07/12 01:31:58 d2.data.datasets.coco]: Converting dataset dicts into COCO format [07/12 01:31:58 d2.data.datasets.coco]: Conversion finished, #images: 234, #annotations: 2780 [07/12 01:31:58 d2.data.datasets.coco]: Caching COCO format annotations at 'Shared/cfu/models/c20495e1-6249-468f-a90a-985e980ac774/inference/dataset_test_coco_format.json' ... [07/12 01:31:59 d2.evaluation.evaluator]: Start inference on 234 batches [07/12 01:32:01 d2.evaluation.evaluator]: Inference done 11/234. Dataloading: 0.0035 s/iter. Inference: 0.1297 s/iter. Eval: 0.0531 s/iter. Total: 0.1863 s/iter. ETA=0:00:41 [07/12 01:32:06 d2.evaluation.evaluator]: Inference done 34/234. Dataloading: 0.0051 s/iter. Inference: 0.1273 s/iter. Eval: 0.0807 s/iter. Total: 0.2131 s/iter. ETA=0:00:42 [07/12 01:32:11 d2.evaluation.evaluator]: Inference done 65/234. Dataloading: 0.0047 s/iter. Inference: 0.1202 s/iter. Eval: 0.0617 s/iter. Total: 0.1866 s/iter. ETA=0:00:31 [07/12 01:32:17 d2.evaluation.evaluator]: Inference done 102/234. Dataloading: 0.0045 s/iter. Inference: 0.1142 s/iter. Eval: 0.0496 s/iter. Total: 0.1684 s/iter. ETA=0:00:22 [07/12 01:32:22 d2.evaluation.evaluator]: Inference done 137/234. Dataloading: 0.0044 s/iter. Inference: 0.1145 s/iter. Eval: 0.0437 s/iter. Total: 0.1627 s/iter. ETA=0:00:15 [07/12 01:32:27 d2.evaluation.evaluator]: Inference done 172/234. Dataloading: 0.0042 s/iter. Inference: 0.1158 s/iter. Eval: 0.0388 s/iter. Total: 0.1589 s/iter. ETA=0:00:09 [07/12 01:32:32 d2.evaluation.evaluator]: Inference done 225/234. Dataloading: 0.0037 s/iter. Inference: 0.1080 s/iter. Eval: 0.0318 s/iter. Total: 0.1437 s/iter. ETA=0:00:01 [07/12 01:32:33 d2.evaluation.evaluator]: Total inference time: 0:00:32.351187 (0.141272 s / iter per device, on 1 devices) [07/12 01:32:33 d2.evaluation.evaluator]: Total inference pure compute time: 0:00:24 (0.106709 s / iter per device, on 1 devices) [07/12 01:32:33 d2.evaluation.coco_evaluation]: Preparing results for COCO format ... [07/12 01:32:33 d2.evaluation.coco_evaluation]: Saving results to Shared/cfu/models/c20495e1-6249-468f-a90a-985e980ac774/inference/coco_instances_results.json [07/12 01:32:33 d2.evaluation.coco_evaluation]: Evaluating predictions with unofficial COCO API... Loading and preparing results... DONE (t=0.00s) creating index... index created! [07/12 01:32:33 d2.evaluation.fast_eval_api]: Evaluate annotation type bbox [07/12 01:32:33 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 0.06 seconds. [07/12 01:32:33 d2.evaluation.fast_eval_api]: Accumulating evaluation results... [07/12 01:32:33 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 0.01 seconds. Average Precision (AP) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 area= all maxDets=100 ] = 0.001 Average Precision (AP) @[ IoU=0.75 area= all maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 1 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 10 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.009 Average Recall (AR) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.008 [07/12 01:32:33 d2.evaluation.coco_evaluation]: Evaluation results for bbox: AP AP50 AP75 APs APm APl
0.017 0.073 0.002 0.016 0.030 0.227
Loading and preparing results... DONE (t=0.02s) creating index... index created! [07/12 01:32:33 d2.evaluation.fast_eval_api]: Evaluate annotation type segm [07/12 01:32:33 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 0.13 seconds. [07/12 01:32:33 d2.evaluation.fast_eval_api]: Accumulating evaluation results... [07/12 01:32:33 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 0.01 seconds. Average Precision (AP) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 area= all maxDets=100 ] = 0.001 Average Precision (AP) @[ IoU=0.75 area= all maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 1 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 10 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.008 Average Recall (AR) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.008 [07/12 01:32:33 d2.evaluation.coco_evaluation]: Evaluation results for segm: AP AP50 AP75 APs APm APl
0.015 0.071 0.002 0.016 0.025 0.233

[07/12 01:32:33 d2.engine.defaults]: Evaluation results for dataset_test in csv format: [07/12 01:32:33 d2.evaluation.testing]: copypaste: Task: main [07/12 01:32:33 d2.evaluation.testing]: copypaste: MAE (counts) 0,MAPE (% counts) 11+ [07/12 01:32:33 d2.evaluation.testing]: copypaste: 0.0000,4.8696 [07/12 01:32:33 d2.evaluation.testing]: copypaste: Task: mae [07/12 01:32:33 d2.evaluation.testing]: copypaste: 11+,51+,101+,0+,1+,0 [07/12 01:32:33 d2.evaluation.testing]: copypaste: 1.5263,1.6250,4.0000,0.4872,0.6552,0.0000 [07/12 01:32:33 d2.evaluation.testing]: copypaste: Task: mape [07/12 01:32:33 d2.evaluation.testing]: copypaste: 11+,51+,101+,1+,0 [07/12 01:32:33 d2.evaluation.testing]: copypaste: 4.8696,2.1965,3.5819,6.6990,nan [07/12 01:32:33 d2.evaluation.testing]: copypaste: Task: ratio [07/12 01:32:33 d2.evaluation.testing]: copypaste: 11+,51+,101+,0+,1+,0 [07/12 01:32:33 d2.evaluation.testing]: copypaste: 24.3590,6.8376,0.8547,100.0000,74.3590,25.6410 [07/12 01:32:33 d2.evaluation.testing]: copypaste: Task: bbox [07/12 01:32:33 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl [07/12 01:32:33 d2.evaluation.testing]: copypaste: 0.0169,0.0728,0.0024,0.0163,0.0304,0.2268 [07/12 01:32:33 d2.evaluation.testing]: copypaste: Task: segm [07/12 01:32:33 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl [07/12 01:32:33 d2.evaluation.testing]: copypaste: 0.0151,0.0710,0.0024,0.0161,0.0250,0.2332 [07/12 01:32:33 d2.utils.events]: eta: 1:57:34 iter: 55999 total_loss: 0.1169 loss_cls: 0.02125 loss_box_reg: 0.008282 loss_mask: 0.08076 loss_rpn_cls: 0.0007155 loss_rpn_loc: 0.001906 time: 0.4460 data_time: 0.0075 lr: 0.01 max_mem: 11345M [07/12 01:32:42 d2.utils.events]: eta: 1:57:25 iter: 56019 total_loss: 0.1438 loss_cls: 0.02627 loss_box_reg: 0.01023 loss_mask: 0.09723 loss_rpn_cls: 0.0008739 loss_rpn_loc: 0.004887 time: 0.4460 data_time: 0.0077 lr: 0.01 max_mem: 11345M [07/12 01:32:52 d2.utils.events]: eta: 1:57:22 iter: 56039 total_loss: 0.1892 loss_cls: 0.05523 loss_box_reg: 0.01809 loss_mask: 0.1071 loss_rpn_cls: 0.001002 loss_rpn_loc: 0.009079 time: 0.4460 data_time: 0.0218 lr: 0.01 max_mem: 11345M [07/12 01:33:01 d2.utils.events]: eta: 1:57:05 iter: 56059 total_loss: 0.2366 loss_cls: 0.06919 loss_box_reg: 0.01766 loss_mask: 0.1326 loss_rpn_cls: 0.00175 loss_rpn_loc: 0.01513 time: 0.4460 data_time: 0.0091 lr: 0.01 max_mem: 11345M [07/12 01:33:11 d2.utils.events]: eta: 1:57:04 iter: 56079 total_loss: 0.2322 loss_cls: 0.06314 loss_box_reg: 0.01654 loss_mask: 0.1259 loss_rpn_cls: 0.0009802 loss_rpn_loc: 0.01876 time: 0.4460 data_time: 0.0084 lr: 0.01 max_mem: 11345M [07/12 01:33:20 d2.utils.events]: eta: 1:56:55 iter: 56099 total_loss: 0.2308 loss_cls: 0.06212 loss_box_reg: 0.01369 loss_mask: 0.1242 loss_rpn_cls: 0.0008664 loss_rpn_loc: 0.01518 time: 0.4460 data_time: 0.0087 lr: 0.01 max_mem: 11345M [07/12 01:33:28 d2.utils.events]: eta: 1:56:46 iter: 56119 total_loss: 0.2202 loss_cls: 0.05569 loss_box_reg: 0.01555 loss_mask: 0.1299 loss_rpn_cls: 0.0002159 loss_rpn_loc: 0.00653 time: 0.4460 data_time: 0.0074 lr: 0.01 max_mem: 11345M [07/12 01:33:37 d2.utils.events]: eta: 1:56:34 iter: 56139 total_loss: 0.1764 loss_cls: 0.03215 loss_box_reg: 0.01407 loss_mask: 0.1191 loss_rpn_cls: 0.0005995 loss_rpn_loc: 0.005629 time: 0.4460 data_time: 0.0079 lr: 0.01 max_mem: 11345M [07/12 01:33:46 d2.utils.events]: eta: 1:56:42 iter: 56159 total_loss: 0.1778 loss_cls: 0.03448 loss_box_reg: 0.01189 loss_mask: 0.1052 loss_rpn_cls: 0.0004555 loss_rpn_loc: 0.004947 time: 0.4460 data_time: 0.0087 lr: 0.01 max_mem: 11345M [07/12 01:33:55 d2.utils.events]: eta: 1:56:34 iter: 56179 total_loss: 0.1679 loss_cls: 0.03963 loss_box_reg: 0.01348 loss_mask: 0.1169 loss_rpn_cls: 0.001884 loss_rpn_loc: 0.009099 time: 0.4460 data_time: 0.0078 lr: 0.01 max_mem: 11345M [07/12 01:34:04 d2.utils.events]: eta: 1:56:23 iter: 56199 total_loss: 0.2262 loss_cls: 0.06002 loss_box_reg: 0.01456 loss_mask: 0.1326 loss_rpn_cls: 0.00176 loss_rpn_loc: 0.0157 time: 0.4460 data_time: 0.0083 lr: 0.01 max_mem: 11345M [07/12 01:34:13 d2.utils.events]: eta: 1:56:14 iter: 56219 total_loss: 0.2058 loss_cls: 0.05331 loss_box_reg: 0.01728 loss_mask: 0.1241 loss_rpn_cls: 0.0008553 loss_rpn_loc: 0.0199 time: 0.4460 data_time: 0.0090 lr: 0.01 max_mem: 11345M [07/12 01:34:23 d2.utils.events]: eta: 1:56:09 iter: 56239 total_loss: 0.2486 loss_cls: 0.0609 loss_box_reg: 0.01941 loss_mask: 0.1307 loss_rpn_cls: 0.001822 loss_rpn_loc: 0.01551 time: 0.4460 data_time: 0.0074 lr: 0.01 max_mem: 11345M [07/12 01:34:32 d2.utils.events]: eta: 1:56:10 iter: 56259 total_loss: 0.2412 loss_cls: 0.06293 loss_box_reg: 0.01933 loss_mask: 0.1249 loss_rpn_cls: 0.0007813 loss_rpn_loc: 0.009812 time: 0.4460 data_time: 0.0081 lr: 0.01 max_mem: 11345M [07/12 01:34:41 d2.utils.events]: eta: 1:55:53 iter: 56279 total_loss: 0.2173 loss_cls: 0.04217 loss_box_reg: 0.01525 loss_mask: 0.1229 loss_rpn_cls: 0.00235 loss_rpn_loc: 0.01429 time: 0.4460 data_time: 0.0090 lr: 0.01 max_mem: 11345M [07/12 01:34:49 d2.utils.events]: eta: 1:55:42 iter: 56299 total_loss: 0.2156 loss_cls: 0.04108 loss_box_reg: 0.01467 loss_mask: 0.1237 loss_rpn_cls: 0.001051 loss_rpn_loc: 0.01096 time: 0.4460 data_time: 0.0075 lr: 0.01 max_mem: 11345M [07/12 01:34:59 d2.utils.events]: eta: 1:55:40 iter: 56319 total_loss: 0.257 loss_cls: 0.07825 loss_box_reg: 0.01948 loss_mask: 0.1326 loss_rpn_cls: 0.003547 loss_rpn_loc: 0.01474 time: 0.4460 data_time: 0.0090 lr: 0.01 max_mem: 11345M [07/12 01:35:07 d2.utils.events]: eta: 1:55:21 iter: 56339 total_loss: 0.2018 loss_cls: 0.04359 loss_box_reg: 0.01425 loss_mask: 0.1073 loss_rpn_cls: 0.001117 loss_rpn_loc: 0.008918 time: 0.4460 data_time: 0.0090 lr: 0.01 max_mem: 11345M [07/12 01:35:16 d2.utils.events]: eta: 1:55:12 iter: 56359 total_loss: 0.1739 loss_cls: 0.03893 loss_box_reg: 0.01233 loss_mask: 0.1126 loss_rpn_cls: 0.0004813 loss_rpn_loc: 0.003779 time: 0.4460 data_time: 0.0072 lr: 0.01 max_mem: 11345M [07/12 01:35:25 d2.utils.events]: eta: 1:55:09 iter: 56379 total_loss: 0.2093 loss_cls: 0.06248 loss_box_reg: 0.01734 loss_mask: 0.1192 loss_rpn_cls: 0.001699 loss_rpn_loc: 0.01419 time: 0.4460 data_time: 0.0088 lr: 0.01 max_mem: 11345M [07/12 01:35:34 d2.utils.events]: eta: 1:55:08 iter: 56399 total_loss: 0.1778 loss_cls: 0.04614 loss_box_reg: 0.0134 loss_mask: 0.1097 loss_rpn_cls: 0.0009438 loss_rpn_loc: 0.01291 time: 0.4460 data_time: 0.0083 lr: 0.01 max_mem: 11345M [07/12 01:35:43 d2.utils.events]: eta: 1:55:03 iter: 56419 total_loss: 0.1671 loss_cls: 0.04791 loss_box_reg: 0.01153 loss_mask: 0.1033 loss_rpn_cls: 0.0006425 loss_rpn_loc: 0.00368 time: 0.4460 data_time: 0.0084 lr: 0.01 max_mem: 11345M [07/12 01:35:52 d2.utils.events]: eta: 1:54:45 iter: 56439 total_loss: 0.1764 loss_cls: 0.0421 loss_box_reg: 0.01245 loss_mask: 0.107 loss_rpn_cls: 0.0006063 loss_rpn_loc: 0.006493 time: 0.4460 data_time: 0.0089 lr: 0.01 max_mem: 11345M [07/12 01:36:01 d2.utils.events]: eta: 1:54:42 iter: 56459 total_loss: 0.1802 loss_cls: 0.04233 loss_box_reg: 0.01541 loss_mask: 0.1087 loss_rpn_cls: 0.0005481 loss_rpn_loc: 0.00798 time: 0.4460 data_time: 0.0069 lr: 0.01 max_mem: 11345M [07/12 01:36:10 d2.utils.events]: eta: 1:54:30 iter: 56479 total_loss: 0.1868 loss_cls: 0.04195 loss_box_reg: 0.01782 loss_mask: 0.1082 loss_rpn_cls: 0.001694 loss_rpn_loc: 0.01846 time: 0.4460 data_time: 0.0088 lr: 0.01 max_mem: 11345M [07/12 01:36:18 d2.utils.events]: eta: 1:54:24 iter: 56499 total_loss: 0.2178 loss_cls: 0.05501 loss_box_reg: 0.01563 loss_mask: 0.127 loss_rpn_cls: 0.0008795 loss_rpn_loc: 0.01398 time: 0.4460 data_time: 0.0089 lr: 0.01 max_mem: 11345M [07/12 01:36:27 d2.utils.events]: eta: 1:53:50 iter: 56519 total_loss: 0.1668 loss_cls: 0.03967 loss_box_reg: 0.01277 loss_mask: 0.09428 loss_rpn_cls: 0.0009289 loss_rpn_loc: 0.00562 time: 0.4460 data_time: 0.0072 lr: 0.01 max_mem: 11345M [07/12 01:36:37 d2.utils.events]: eta: 1:53:54 iter: 56539 total_loss: 0.2153 loss_cls: 0.0553 loss_box_reg: 0.01673 loss_mask: 0.1219 loss_rpn_cls: 0.0007145 loss_rpn_loc: 0.009533 time: 0.4460 data_time: 0.0093 lr: 0.01 max_mem: 11345M [07/12 01:36:46 d2.utils.events]: eta: 1:53:54 iter: 56559 total_loss: 0.2056 loss_cls: 0.05546 loss_box_reg: 0.01681 loss_mask: 0.1252 loss_rpn_cls: 0.001095 loss_rpn_loc: 0.01251 time: 0.4460 data_time: 0.0088 lr: 0.01 max_mem: 11345M [07/12 01:36:55 d2.utils.events]: eta: 1:53:37 iter: 56579 total_loss: 0.1711 loss_cls: 0.04191 loss_box_reg: 0.01343 loss_mask: 0.1044 loss_rpn_cls: 0.001209 loss_rpn_loc: 0.01152 time: 0.4460 data_time: 0.0073 lr: 0.01 max_mem: 11345M [07/12 01:37:03 d2.utils.events]: eta: 1:53:29 iter: 56599 total_loss: 0.2211 loss_cls: 0.05814 loss_box_reg: 0.01569 loss_mask: 0.129 loss_rpn_cls: 0.001717 loss_rpn_loc: 0.01976 time: 0.4460 data_time: 0.0088 lr: 0.01 max_mem: 11345M [07/12 01:37:13 d2.utils.events]: eta: 1:53:24 iter: 56619 total_loss: 0.2186 loss_cls: 0.05173 loss_box_reg: 0.01801 loss_mask: 0.1142 loss_rpn_cls: 0.001285 loss_rpn_loc: 0.0193 time: 0.4460 data_time: 0.0097 lr: 0.01 max_mem: 11345M [07/12 01:37:21 d2.utils.events]: eta: 1:53:11 iter: 56639 total_loss: 0.1613 loss_cls: 0.03338 loss_box_reg: 0.01316 loss_mask: 0.1112 loss_rpn_cls: 0.0003977 loss_rpn_loc: 0.004656 time: 0.4460 data_time: 0.0072 lr: 0.01 max_mem: 11345M [07/12 01:37:30 d2.utils.events]: eta: 1:52:44 iter: 56659 total_loss: 0.1672 loss_cls: 0.02986 loss_box_reg: 0.01245 loss_mask: 0.1029 loss_rpn_cls: 0.0006512 loss_rpn_loc: 0.008434 time: 0.4460 data_time: 0.0075 lr: 0.01 max_mem: 11345M [07/12 01:37:39 d2.utils.events]: eta: 1:52:40 iter: 56679 total_loss: 0.223 loss_cls: 0.05242 loss_box_reg: 0.01731 loss_mask: 0.1349 loss_rpn_cls: 0.001059 loss_rpn_loc: 0.01187 time: 0.4460 data_time: 0.0085 lr: 0.01 max_mem: 11345M [07/12 01:37:48 d2.utils.events]: eta: 1:52:38 iter: 56699 total_loss: 0.1547 loss_cls: 0.02856 loss_box_reg: 0.0107 loss_mask: 0.1131 loss_rpn_cls: 0.001192 loss_rpn_loc: 0.006186 time: 0.4460 data_time: 0.0075 lr: 0.01 max_mem: 11345M [07/12 01:37:56 d2.utils.events]: eta: 1:52:24 iter: 56719 total_loss: 0.1958 loss_cls: 0.04971 loss_box_reg: 0.01445 loss_mask: 0.1476 loss_rpn_cls: 0.001104 loss_rpn_loc: 0.01117 time: 0.4460 data_time: 0.0084 lr: 0.01 max_mem: 11345M [07/12 01:38:05 d2.utils.events]: eta: 1:52:15 iter: 56739 total_loss: 0.1769 loss_cls: 0.03637 loss_box_reg: 0.01332 loss_mask: 0.1161 loss_rpn_cls: 0.001293 loss_rpn_loc: 0.00866 time: 0.4460 data_time: 0.0084 lr: 0.01 max_mem: 11345M [07/12 01:38:15 d2.utils.events]: eta: 1:52:25 iter: 56759 total_loss: 0.227 loss_cls: 0.05001 loss_box_reg: 0.01671 loss_mask: 0.1225 loss_rpn_cls: 0.001918 loss_rpn_loc: 0.01686 time: 0.4460 data_time: 0.0091 lr: 0.01 max_mem: 11345M [07/12 01:38:24 d2.utils.events]: eta: 1:52:20 iter: 56779 total_loss: 0.2195 loss_cls: 0.04918 loss_box_reg: 0.01663 loss_mask: 0.1289 loss_rpn_cls: 0.002565 loss_rpn_loc: 0.01142 time: 0.4460 data_time: 0.0084 lr: 0.01 max_mem: 11345M [07/12 01:38:33 d2.utils.events]: eta: 1:52:05 iter: 56799 total_loss: 0.2651 loss_cls: 0.08386 loss_box_reg: 0.01853 loss_mask: 0.143 loss_rpn_cls: 0.004439 loss_rpn_loc: 0.0167 time: 0.4460 data_time: 0.0089 lr: 0.01 max_mem: 11345M [07/12 01:38:42 d2.utils.events]: eta: 1:51:59 iter: 56819 total_loss: 0.2422 loss_cls: 0.05831 loss_box_reg: 0.01871 loss_mask: 0.1438 loss_rpn_cls: 0.002861 loss_rpn_loc: 0.007384 time: 0.4460 data_time: 0.0077 lr: 0.01 max_mem: 11345M [07/12 01:38:51 d2.utils.events]: eta: 1:51:50 iter: 56839 total_loss: 0.2292 loss_cls: 0.0434 loss_box_reg: 0.01323 loss_mask: 0.1351 loss_rpn_cls: 0.0023 loss_rpn_loc: 0.01812 time: 0.4460 data_time: 0.0088 lr: 0.01 max_mem: 11345M [07/12 01:38:59 d2.utils.events]: eta: 1:51:50 iter: 56859 total_loss: 0.242 loss_cls: 0.0578 loss_box_reg: 0.01444 loss_mask: 0.1273 loss_rpn_cls: 0.001829 loss_rpn_loc: 0.008506 time: 0.4460 data_time: 0.0078 lr: 0.01 max_mem: 11345M [07/12 01:39:08 d2.utils.events]: eta: 1:51:33 iter: 56879 total_loss: 0.2622 loss_cls: 0.06702 loss_box_reg: 0.01816 loss_mask: 0.1327 loss_rpn_cls: 0.0005544 loss_rpn_loc: 0.01144 time: 0.4460 data_time: 0.0081 lr: 0.01 max_mem: 11345M [07/12 01:39:17 d2.utils.events]: eta: 1:51:27 iter: 56899 total_loss: 0.2804 loss_cls: 0.0782 loss_box_reg: 0.02008 loss_mask: 0.1333 loss_rpn_cls: 0.002271 loss_rpn_loc: 0.01798 time: 0.4460 data_time: 0.0095 lr: 0.01 max_mem: 11345M [07/12 01:39:26 d2.utils.events]: eta: 1:51:11 iter: 56919 total_loss: 0.2591 loss_cls: 0.05679 loss_box_reg: 0.01635 loss_mask: 0.1276 loss_rpn_cls: 0.0007462 loss_rpn_loc: 0.009331 time: 0.4460 data_time: 0.0074 lr: 0.01 max_mem: 11345M [07/12 01:39:35 d2.utils.events]: eta: 1:51:12 iter: 56939 total_loss: 0.2809 loss_cls: 0.06402 loss_box_reg: 0.02063 loss_mask: 0.1455 loss_rpn_cls: 0.002003 loss_rpn_loc: 0.02014 time: 0.4460 data_time: 0.0084 lr: 0.01 max_mem: 11345M [07/12 01:39:44 d2.utils.events]: eta: 1:51:03 iter: 56959 total_loss: 0.2491 loss_cls: 0.06104 loss_box_reg: 0.01602 loss_mask: 0.1299 loss_rpn_cls: 0.001515 loss_rpn_loc: 0.0185 time: 0.4460 data_time: 0.0087 lr: 0.01 max_mem: 11345M [07/12 01:39:52 d2.utils.events]: eta: 1:50:48 iter: 56979 total_loss: 0.2124 loss_cls: 0.04628 loss_box_reg: 0.01474 loss_mask: 0.1145 loss_rpn_cls: 0.0009609 loss_rpn_loc: 0.006586 time: 0.4460 data_time: 0.0081 lr: 0.01 max_mem: 11345M [07/12 01:40:02 d2.utils.events]: eta: 1:50:46 iter: 56999 total_loss: 0.2877 loss_cls: 0.07403 loss_box_reg: 0.01935 loss_mask: 0.1316 loss_rpn_cls: 0.001429 loss_rpn_loc: 0.01994 time: 0.4460 data_time: 0.0089 lr: 0.01 max_mem: 11345M [07/12 01:40:10 d2.utils.events]: eta: 1:50:46 iter: 57019 total_loss: 0.1763 loss_cls: 0.03502 loss_box_reg: 0.01379 loss_mask: 0.1164 loss_rpn_cls: 0.000419 loss_rpn_loc: 0.006528 time: 0.4460 data_time: 0.0073 lr: 0.01 max_mem: 11345M [07/12 01:40:20 d2.utils.events]: eta: 1:50:33 iter: 57039 total_loss: 0.2803 loss_cls: 0.08492 loss_box_reg: 0.02088 loss_mask: 0.1424 loss_rpn_cls: 0.003215 loss_rpn_loc: 0.01601 time: 0.4460 data_time: 0.0092 lr: 0.01 max_mem: 11345M [07/12 01:40:28 d2.utils.events]: eta: 1:50:16 iter: 57059 total_loss: 0.1912 loss_cls: 0.04257 loss_box_reg: 0.01312 loss_mask: 0.1217 loss_rpn_cls: 0.001922 loss_rpn_loc: 0.00909 time: 0.4460 data_time: 0.0079 lr: 0.01 max_mem: 11345M [07/12 01:40:38 d2.utils.events]: eta: 1:50:10 iter: 57079 total_loss: 0.2533 loss_cls: 0.07634 loss_box_reg: 0.01825 loss_mask: 0.1199 loss_rpn_cls: 0.002404 loss_rpn_loc: 0.01514 time: 0.4460 data_time: 0.0090 lr: 0.01 max_mem: 11345M [07/12 01:40:46 d2.utils.events]: eta: 1:49:50 iter: 57099 total_loss: 0.4486 loss_cls: 0.08131 loss_box_reg: 0.0095 loss_mask: 0.2016 loss_rpn_cls: 0.03777 loss_rpn_loc: 0.01597 time: 0.4460 data_time: 0.0077 lr: 0.01 max_mem: 11345M ERROR [07/12 01:40:51 d2.engine.train_loop]: Exception during training: Traceback (most recent call last): File "/databricks/python/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 149, in train self.run_step() File "/databricks/python/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 494, in run_step self._trainer.run_step() File "/databricks/python/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 273, in run_step loss_dict = self.model(data) File "/databricks/python/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, *kwargs) File "/databricks/python/lib/python3.8/site-packages/detectron2/modeling/metaarch/rcnn.py", line 163, in forward , detector_losses = self.roi_heads(images, features, proposals, gt_instances) File "/databricks/python/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(input, **kwargs) File "/databricks/python/lib/python3.8/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 739, in forward losses = self._forward_box(features, proposals) File "/databricks/python/lib/python3.8/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 804, in _forward_box losses = self.box_predictor.losses(predictions, proposals) File "/databricks/python/lib/python3.8/site-packages/detectron2/modeling/roi_heads/fast_rcnn.py", line 323, in losses "loss_box_reg": self.box_reg_loss( File "/databricks/python/lib/python3.8/site-packages/detectron2/modeling/roi_heads/fast_rcnn.py", line 358, in box_reg_loss loss_box_reg = giou_loss(fg_pred_boxes, gt_boxes[fg_inds], reduction="sum") File "/databricks/python/lib/python3.8/site-packages/fvcore/nn/giou_loss.py", line 32, in giou_loss assert (x2 >= x1).all(), "bad box: x1 larger than x2" AssertionError: bad box: x1 larger than x2 [07/12 01:40:51 d2.engine.hooks]: Overall training speed: 57109 iterations in 7:04:29 (0.4460 s / it) [07/12 01:40:51 d2.engine.hooks]: Total training time: 7:16:06 (0:11:36 on hooks) [07/12 01:40:51 d2.utils.events]: eta: 1:49:42 iter: 57111 total_loss: 0.8432 loss_cls: 0.07376 loss_box_reg: 0.00708 loss_mask: 0.2679 loss_rpn_cls: 0.3349 loss_rpn_loc: 0.04453 time: 0.4460 data_time: 0.0079 lr: 0.01 max_mem: 11345M

lartiguebe commented 1 year ago

Were you able to figure this bug out ? We have the same problem.