Open mandyxiaomeng opened 6 months ago
Hi @mandyxiaomeng, you could add a --show-dir
with your targeted directory. BTW, the visualization has a threshold.
Hi @mandyxiaomeng, you could add a
--show-dir
with your targeted directory. BTW, the visualization has a threshold.
Hi @wondervictor , Thank you for the reply. I found out it is not a problem of visualization but a problem of training. My trained model cannot detect anything.
The problem is:
This is exactly like #146 , #126 and #185.
I have tried different methods mentioned in these three topics, but still cannot solve the problem. I think my metainfo
and load_from
are correct? Do you have any suggestions? Thank you!
Below is the detailed info I got during training.
05/08 17:16:01 - mmengine - INFO - Epoch(train) [1][ 50/655] base_lr: 2.0000e-05 lr: 4.9873e-07 eta: 0:13:08 time: 1.3025 data_time: 0.8345 memory: 34153 grad_norm: nan loss: 609.7719 loss_cls: 609.7719 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:17:01 - mmengine - INFO - Epoch(train) [1][100/655] base_lr: 2.0000e-05 lr: 1.0076e-06 eta: 0:11:34 time: 1.1991 data_time: 0.9747 memory: 10640 grad_norm: 27732.7017 loss: 442.9217 loss_cls: 442.9217 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:17:58 - mmengine - INFO - Epoch(train) [1][150/655] base_lr: 2.0000e-05 lr: 1.5165e-06 eta: 0:10:11 time: 1.1300 data_time: 0.9082 memory: 10640 grad_norm: 16402.5593 loss: 255.1138 loss_cls: 255.1138 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:18:52 - mmengine - INFO - Epoch(train) [1][200/655] base_lr: 2.0000e-05 lr: 2.0254e-06 eta: 0:08:57 time: 1.0905 data_time: 0.8680 memory: 10640 grad_norm: 11682.1961 loss: 128.2933 loss_cls: 128.2933 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:19:48 - mmengine - INFO - Epoch(train) [1][250/655] base_lr: 2.0000e-05 lr: 2.5344e-06 eta: 0:07:52 time: 1.1172 data_time: 0.8876 memory: 10640 grad_norm: 6005.4033 loss: 60.0497 loss_cls: 60.0497 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:20:45 - mmengine - INFO - Epoch(train) [1][300/655] base_lr: 2.0000e-05 lr: 3.0433e-06 eta: 0:06:52 time: 1.1387 data_time: 0.9109 memory: 10640 grad_norm: 2919.8520 loss: 24.3001 loss_cls: 24.3001 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:21:44 - mmengine - INFO - Epoch(train) [1][350/655] base_lr: 2.0000e-05 lr: 3.5522e-06 eta: 0:05:55 time: 1.1786 data_time: 0.9514 memory: 10640 grad_norm: 797.1709 loss: 7.0634 loss_cls: 7.0634 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:22:43 - mmengine - INFO - Epoch(train) [1][400/655] base_lr: 2.0000e-05 lr: 4.0611e-06 eta: 0:04:57 time: 1.1869 data_time: 0.9600 memory: 10640 grad_norm: 553.4262 loss: 2.1420 loss_cls: 2.1420 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:23:34 - mmengine - INFO - Epoch(train) [1][450/655] base_lr: 2.0000e-05 lr: 4.5700e-06 eta: 0:03:55 time: 1.0127 data_time: 0.7891 memory: 10640 grad_norm: 260.8789 loss: 0.5537 loss_cls: 0.5537 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:24:27 - mmengine - INFO - Epoch(train) [1][500/655] base_lr: 2.0000e-05 lr: 5.0789e-06 eta: 0:02:56 time: 1.0512 data_time: 0.8300 memory: 10640 grad_norm: 47.5409 loss: 0.0902 loss_cls: 0.0902 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:25:26 - mmengine - INFO - Epoch(train) [1][550/655] base_lr: 2.0000e-05 lr: 5.5878e-06 eta: 0:02:00 time: 1.1815 data_time: 0.9594 memory: 10640 grad_norm: 9.8181 loss: 0.0138 loss_cls: 0.0138 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:26:22 - mmengine - INFO - Epoch(train) [1][600/655] base_lr: 2.0000e-05 lr: 6.0967e-06 eta: 0:01:02 time: 1.1224 data_time: 0.8993 memory: 10640 grad_norm: 22.4686 loss: 0.0083 loss_cls: 0.0083 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:27:13 - mmengine - INFO - Epoch(train) [1][650/655] base_lr: 2.0000e-05 lr: 6.6056e-06 eta: 0:00:05 time: 1.0293 data_time: 0.8067 memory: 10640 grad_norm: 2.7762 loss: 0.0034 loss_cls: 0.0034 loss_bbox: 0.0000 loss_dfl: 0.0000
05/08 17:27:15 - mmengine - INFO - Exp name: yolo_world_l_usecase3_random2_20240508_171439
05/08 17:27:15 - mmengine - INFO - Saving checkpoint at 1 epochs
05/08 17:27:18 - mmengine - WARNING - `save_param_scheduler` is True but `self.param_schedulers` is None, so skip saving parameter schedulers
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
05/08 17:27:23 - mmengine - INFO - Epoch(val) [1][ 50/2621] eta: 0:02:07 time: 0.0496 data_time: 0.0039 memory: 10640
05/08 17:27:25 - mmengine - INFO - Epoch(val) [1][ 100/2621] eta: 0:01:43 time: 0.0327 data_time: 0.0003 memory: 1638
05/08 17:27:26 - mmengine - INFO - Epoch(val) [1][ 150/2621] eta: 0:01:34 time: 0.0322 data_time: 0.0003 memory: 1638
05/08 17:27:28 - mmengine - INFO - Epoch(val) [1][ 200/2621] eta: 0:01:28 time: 0.0321 data_time: 0.0003 memory: 1638
05/08 17:27:30 - mmengine - INFO - Epoch(val) [1][ 250/2621] eta: 0:01:24 time: 0.0321 data_time: 0.0003 memory: 1638
05/08 17:27:31 - mmengine - INFO - Epoch(val) [1][ 300/2621] eta: 0:01:21 time: 0.0314 data_time: 0.0003 memory: 1638
05/08 17:27:33 - mmengine - INFO - Epoch(val) [1][ 350/2621] eta: 0:01:18 time: 0.0323 data_time: 0.0003 memory: 1638
05/08 17:27:34 - mmengine - INFO - Epoch(val) [1][ 400/2621] eta: 0:01:16 time: 0.0329 data_time: 0.0003 memory: 1638
05/08 17:27:36 - mmengine - INFO - Epoch(val) [1][ 450/2621] eta: 0:01:15 time: 0.0359 data_time: 0.0004 memory: 1638
05/08 17:27:38 - mmengine - INFO - Epoch(val) [1][ 500/2621] eta: 0:01:13 time: 0.0339 data_time: 0.0004 memory: 1638
05/08 17:27:40 - mmengine - INFO - Epoch(val) [1][ 550/2621] eta: 0:01:11 time: 0.0334 data_time: 0.0003 memory: 1638
05/08 17:27:41 - mmengine - INFO - Epoch(val) [1][ 600/2621] eta: 0:01:09 time: 0.0333 data_time: 0.0004 memory: 1638
05/08 17:27:43 - mmengine - INFO - Epoch(val) [1][ 650/2621] eta: 0:01:07 time: 0.0328 data_time: 0.0003 memory: 1638
05/08 17:27:45 - mmengine - INFO - Epoch(val) [1][ 700/2621] eta: 0:01:05 time: 0.0354 data_time: 0.0004 memory: 1638
05/08 17:27:46 - mmengine - INFO - Epoch(val) [1][ 750/2621] eta: 0:01:04 time: 0.0335 data_time: 0.0003 memory: 1638
05/08 17:27:48 - mmengine - INFO - Epoch(val) [1][ 800/2621] eta: 0:01:02 time: 0.0337 data_time: 0.0004 memory: 1638
05/08 17:27:50 - mmengine - INFO - Epoch(val) [1][ 850/2621] eta: 0:01:00 time: 0.0354 data_time: 0.0004 memory: 1638
05/08 17:27:52 - mmengine - INFO - Epoch(val) [1][ 900/2621] eta: 0:00:58 time: 0.0345 data_time: 0.0003 memory: 1638
05/08 17:27:53 - mmengine - INFO - Epoch(val) [1][ 950/2621] eta: 0:00:57 time: 0.0350 data_time: 0.0003 memory: 1638
05/08 17:27:55 - mmengine - INFO - Epoch(val) [1][1000/2621] eta: 0:00:55 time: 0.0337 data_time: 0.0004 memory: 1638
05/08 17:27:57 - mmengine - INFO - Epoch(val) [1][1050/2621] eta: 0:00:53 time: 0.0357 data_time: 0.0004 memory: 1638
05/08 17:27:59 - mmengine - INFO - Epoch(val) [1][1100/2621] eta: 0:00:52 time: 0.0340 data_time: 0.0003 memory: 1638
05/08 17:28:00 - mmengine - INFO - Epoch(val) [1][1150/2621] eta: 0:00:50 time: 0.0350 data_time: 0.0003 memory: 1638
05/08 17:28:02 - mmengine - INFO - Epoch(val) [1][1200/2621] eta: 0:00:48 time: 0.0351 data_time: 0.0003 memory: 1638
05/08 17:28:04 - mmengine - INFO - Epoch(val) [1][1250/2621] eta: 0:00:47 time: 0.0353 data_time: 0.0003 memory: 1638
05/08 17:28:05 - mmengine - INFO - Epoch(val) [1][1300/2621] eta: 0:00:45 time: 0.0325 data_time: 0.0003 memory: 1638
05/08 17:28:07 - mmengine - INFO - Epoch(val) [1][1350/2621] eta: 0:00:43 time: 0.0345 data_time: 0.0003 memory: 1638
05/08 17:28:09 - mmengine - INFO - Epoch(val) [1][1400/2621] eta: 0:00:42 time: 0.0359 data_time: 0.0004 memory: 1638
05/08 17:28:11 - mmengine - INFO - Epoch(val) [1][1450/2621] eta: 0:00:40 time: 0.0364 data_time: 0.0003 memory: 1638
05/08 17:28:13 - mmengine - INFO - Epoch(val) [1][1500/2621] eta: 0:00:38 time: 0.0361 data_time: 0.0003 memory: 1638
05/08 17:28:14 - mmengine - INFO - Epoch(val) [1][1550/2621] eta: 0:00:36 time: 0.0339 data_time: 0.0003 memory: 1638
05/08 17:28:16 - mmengine - INFO - Epoch(val) [1][1600/2621] eta: 0:00:35 time: 0.0351 data_time: 0.0003 memory: 1638
05/08 17:28:18 - mmengine - INFO - Epoch(val) [1][1650/2621] eta: 0:00:33 time: 0.0340 data_time: 0.0003 memory: 1638
05/08 17:28:19 - mmengine - INFO - Epoch(val) [1][1700/2621] eta: 0:00:31 time: 0.0349 data_time: 0.0003 memory: 1638
05/08 17:28:21 - mmengine - INFO - Epoch(val) [1][1750/2621] eta: 0:00:30 time: 0.0335 data_time: 0.0003 memory: 1638
05/08 17:28:23 - mmengine - INFO - Epoch(val) [1][1800/2621] eta: 0:00:28 time: 0.0336 data_time: 0.0003 memory: 1638
05/08 17:28:25 - mmengine - INFO - Epoch(val) [1][1850/2621] eta: 0:00:26 time: 0.0347 data_time: 0.0003 memory: 1638
05/08 17:28:26 - mmengine - INFO - Epoch(val) [1][1900/2621] eta: 0:00:24 time: 0.0345 data_time: 0.0004 memory: 1638
05/08 17:28:28 - mmengine - INFO - Epoch(val) [1][1950/2621] eta: 0:00:23 time: 0.0333 data_time: 0.0003 memory: 1638
05/08 17:28:30 - mmengine - INFO - Epoch(val) [1][2000/2621] eta: 0:00:21 time: 0.0340 data_time: 0.0003 memory: 1638
05/08 17:28:31 - mmengine - INFO - Epoch(val) [1][2050/2621] eta: 0:00:19 time: 0.0343 data_time: 0.0003 memory: 1638
05/08 17:28:33 - mmengine - INFO - Epoch(val) [1][2100/2621] eta: 0:00:17 time: 0.0329 data_time: 0.0003 memory: 1638
05/08 17:28:35 - mmengine - INFO - Epoch(val) [1][2150/2621] eta: 0:00:16 time: 0.0312 data_time: 0.0003 memory: 1638
05/08 17:28:36 - mmengine - INFO - Epoch(val) [1][2200/2621] eta: 0:00:14 time: 0.0318 data_time: 0.0003 memory: 1638
05/08 17:28:38 - mmengine - INFO - Epoch(val) [1][2250/2621] eta: 0:00:12 time: 0.0332 data_time: 0.0003 memory: 1638
05/08 17:28:40 - mmengine - INFO - Epoch(val) [1][2300/2621] eta: 0:00:10 time: 0.0345 data_time: 0.0003 memory: 1638
05/08 17:28:41 - mmengine - INFO - Epoch(val) [1][2350/2621] eta: 0:00:09 time: 0.0343 data_time: 0.0003 memory: 1638
05/08 17:28:43 - mmengine - INFO - Epoch(val) [1][2400/2621] eta: 0:00:07 time: 0.0323 data_time: 0.0003 memory: 1638
05/08 17:28:45 - mmengine - INFO - Epoch(val) [1][2450/2621] eta: 0:00:05 time: 0.0334 data_time: 0.0003 memory: 1638
05/08 17:28:46 - mmengine - INFO - Epoch(val) [1][2500/2621] eta: 0:00:04 time: 0.0340 data_time: 0.0003 memory: 1638
05/08 17:28:48 - mmengine - INFO - Epoch(val) [1][2550/2621] eta: 0:00:02 time: 0.0339 data_time: 0.0003 memory: 1638
05/08 17:28:50 - mmengine - INFO - Epoch(val) [1][2600/2621] eta: 0:00:00 time: 0.0340 data_time: 0.0003 memory: 1638
05/08 17:28:50 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
05/08 17:28:50 - mmengine - ERROR - /proj/simtoreal_mandy/users/x_xizhu/Project/Paper1/mmdetection-3.0.0/mmdet/evaluation/metrics/coco_metric.py - compute_metrics - 461 - The testing results of the whole dataset is empty.
05/08 17:28:50 - mmengine - INFO - Epoch(val) [1][2621/2621] data_time: 0.0004 time: 0.0342
05/08 17:28:50 - mmengine - WARNING - Since `metrics` is an empty dict, the behavior to save the best checkpoint will be skipped in this evaluation.
I have the same problem. It's the same. Have you solved it?
您好,
我用了以下config在我自己的数据上微调:
我可以实现微调,试着训练了5个epoch得到了epoch_5.pth 然后我想用test.sh看这个模型的测试结果,用了以下命令:
可是show 显示出来的图片没有bounding box, 连GroundTruth 的bounding box也没有。但是是有测试结果的。 请问这是为什么呢? 我的数据集结构: --data_usecase3_yoloworld --train --val --annotations --val.json --train.json 用mmyolo的analysis_tools, browse_coco_json.py是能显示出GT的bounding box的。后面我用image_demo.py测试也是能有bounding box的。 所以我不太清楚是哪里错了。谢谢!