Problems with the total loss.

[wwylvswy]

Dear author, I've recently started learning about and researching Open World Object Detection (OWOD) and have been running your project. Due to the limited GPU resources I have—only two 1080ti GPUs (total memory 24GB)—I've made some adjustments to the two parameters in the base.yaml file."

IMS_PER_BATCH: 4
BASE_LR: 0.00001

This is the log information when running S-OWOD/t1.

iter: 19  total_loss: 257.4  loss_ce: 35.42 ...
......
iter: 2919  total_loss: 13.65  loss_ce: 0.8571...
......
iter: 5279  total_loss: 11.92  loss_ce: 0.7209 
......
iter: 25579  total_loss: 8.654  loss_ce: 0.4517
......
iter: 39979  total_loss: 8.137  loss_ce: 0.4111...

The total_loss value keeps fluctuating between 7 and 9, starting from 25579 and reaching 39979. I'm unsure if this is normal behavior. I've observed a similar phenomenon in subsequent tasks (including M-OWOD and IOD), where total_loss can quickly decrease to around 8 but fails to decrease further."

[feifeiobama]

I checked some S-OWODB and M-OWODB logs. The total loss on S-OWODB/t1 is close to your range, but does not fluctuate that much. The loss on other tasks continues to decrease during training and often reaches 6.0~6.5 at the end of the task.

I think the current behavior (the loss fluctuating between 7 and 9) is probably related to the batch size settings, especially in that the prediction orthogonality loss is unstable at small batch sizes.

[wwylvswy]

Thank you for your reply. I have some other questions I would like to ask you for advice.

Is the PTH file in this link the model with the optimal parameters?

What is the relationship between 15_5_0, 15_5_1, and 15_5_ft? I tried testing with ours_iod_15_5_747.pth on 15_5_fit and compared it with my own results (using two 1080ti GPUs, IMS_PER_BATCH=4).

The results using ours_iod_15_5_747.pth are basically consistent with the results presented in the paper.

aeroplane has 946 predictions.
bicycle has 654 predictions.
bird has 1664 predictions.
boat has 2047 predictions.
 bottle has 2696 predictions.
bus has 633 predictions.
car has 6238 predictions.
cat has 893 predictions.
chair has 5992 predictions.
cow has 1183 predictions.
diningtable has 675 predictions.
dog has 900 predictions.
horse has 632 predictions.
motorbike has 924 predictions.
person has 19755 predictions.
pottedplant has 3279 predictions.
sheep has 1464 predictions.
sofa has 2259 predictions.
train has 583 predictions.
tvmonitor has 1363 predictions.
unknown has 2160 predictions.
Wilderness Impact: {0.1: {50: 0.0}, 0.2: {50: 0.0}, 0.3: {50: 0.0}, 0.4: {50: 0.0}, 0.5: {50: 0.0}, 0.6: {50: 0.0}, 0.7: {50: 0.0}, 0.8: {50: 0.0}, 0.9: {50: 0.0}}
avg_precision: {0.1: {50: 0.0}, 0.2: {50: 0.0}, 0.3: {50: 0.0}, 0.4: {50: 0.0}, 0.5: {50: 0.0}, 0.6: {50: 0.0}, 0.7: {50: 0.0}, 0.8: {50: 0.0}, 0.9: {50: 0.0}}
Absolute OSE (total_num_unk_det_as_known): {50: 0.0}
total_num_unk 0
['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor', 'unknown']
AP50: ['82.8', '80.7', '79.1', '55.4', '65.4', '81.4', '89.9', '86.0', '52.3', '84.1', '62.9', '77.9', '82.6', '81.5', '84.2', '46.8', '71.2', '68.4', '82.2', '78.3', 'nan']
Precisions50: ['27.9', '46.2', '27.3', '10.8', '14.8', '30.7', '19.7', '37.9', '11.2', '20.8', '26.3', '48.4', '50.8', '33.5', '22.5', '11.3', '15.1', '10.7', '44.2', '20.8', '0.0']
Recall50: ['91.2', '87.5', '93.0', '79.8', '81.7', '88.7', '96.3', '93.6', '80.4', '94.7', '81.1', '87.5', '87.9', '92.0', '94.5', '75.6', '88.4', '92.9', '89.7', '89.3', 'nan']
Prev class AP50: 76.41166619940797
Prev class Precisions50: 28.595067568923188
Prev class Recall50: 88.67130393624441
Current class AP50: 69.38153217513073
Current class Precisions50: 20.41646409759274
Current class Recall50: 87.1887615394541
Known AP50: 74.65413269333865
Known Precisions50: 26.55041670109058
Known Recall50: 88.30066833704684

However, the results of my own trained model are very poor.

aeroplane has 782 predictions.
bicycle has 909 predictions.
bird has 2332 predictions.
boat has 2210 predictions.
bottle has 1188 predictions.
bus has 479 predictions.
car has 7380 predictions.
cat has 1430 predictions.
chair has 13827 predictions.
cow has 518 predictions.
diningtable has 1957 predictions.
dog has 1838 predictions.
horse has 514 predictions.
motorbike has 1051 predictions.
person has 23230 predictions.
pottedplant has 9104 predictions.
sheep has 618 predictions.
sofa has 1665 predictions.
train has 823 predictions.
tvmonitor has 800 predictions.
unknown has 2397 predictions.
Wilderness Impact: {0.1: {50: 0.0}, 0.2: {50: 0.0}, 0.3: {50: 0.0}, 0.4: {50: 0.0}, 0.5: {50: 0.0}, 0.6: {50: 0.0}, 0.7: {50: 0.0}, 0.8: {50: 0.0}, 0.9: {50: 0.0}}
avg_precision: {0.1: {50: 0.0}, 0.2: {50: 0.0}, 0.3: {50: 0.0}, 0.4: {50: 0.0}, 0.5: {50: 0.0}, 0.6: {50: 0.0}, 0.7: {50: 0.0}, 0.8: {50: 0.0}, 0.9: {50: 0.0}}
Absolute OSE (total_num_unk_det_as_known): {50: 0.0}
total_num_unk 0
['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor', 'unknown']
AP50: ['58.8', '63.5', '52.4', '36.6', '34.0', '64.6', '81.8', '59.4', '28.3', '41.7', '47.0', '51.4', '62.7', '63.7', '72.2', '23.4', '35.7', '45.3', '61.4', '56.9', 'nan']
Precisions50: ['27.8', '30.3', '16.1', '8.5', '22.7', '35.7', '16.0', '21.6', '4.3', '32.1', '8.9', '23.5', '49.5', '24.7', '18.2', '3.9', '25.5', '12.2', '28.9', '28.4', '0.0']
Recall50: ['76.1', '80.1', '79.7', '69.2', '56.5', '79.3', '94.3', '86.0', '73.8', '62.7', '81.1', '87.7', '71.6', '79.1', '91.0', '74.2', '62.8', '80.3', '83.7', '71.8', 'nan']
Prev class AP50: 54.530283225026174
Prev class Precisions50: 22.66535874294839
Prev class Recall50: 77.89128109265106
Current class AP50: 44.52513481893416
Current class Precisions50: 19.80134815452225
Current class Recall50: 74.55050041423388
Known AP50: 52.028996123503156
Known Precisions50: 21.949356095841857
Known Recall50: 77.05608592304677

Could this also be caused by the batch size being set too small?

[feifeiobama]

1. Yes, the final results in our paper are obtained using these checkpoints.
2. 15_5_0, 15_5_1, and 15_5_ft correspond to different training stages. See https://github.com/feifeiobama/OrthogonalDet/issues/3#issuecomment-2213445961.
3. It is very likely. See my replies to a similar issue: https://github.com/feifeiobama/OrthogonalDet/issues/2#issuecomment-2191792254.