total_loss: 0.6 Bad results

[ghost]

@youngwanLEE

• I trained both centermask_V_39_eSE_FPN_ms_3x and centermask_lite_V_19_slim_dw_eSE_FPN_ms_4x
• After ~70,000 iterations, the results were bad, and the total_loss doesn't stabilize below 0.6

How to reproduce:

• Install detectron2
• git clone https://github.com/youngwanLEE/centermask2.git
• cd centermask2
• copy all the files from here, and place them in centermask2 main folder.
• run python train.py
• to inference test, run inference.py

Both v19 and v39 get bad results, and won't decline further below ~total_loss: 0.6

Samples

[youngwanLEE]

@deepseek

How many GPUs did you use?

It seems that your results are trained on the custom dataset.

We can guess what the problem is.

1. dataset

   • Is the number of samples enough?
   • Is there a balance between classes?

2. training environment

   • appropriate hyper-parameters setting according to the number of GPUs : e.g., batch size and learning rate, learning schedule, etc.

It is recommended that you should search for your optimal training setting by referring to the above mentions.

[ghost]

@youngwanLEE

• I uploaded the files to reproduce my training and inferencing steps. download them all and place them in centermask2 main folder. Then run train.py
• I am training on 1 gpu only.
• I tested on multiple configurations and datasets, still the results were bad.

[ghost]

Centermask2 still needs further improvement

[Adam1904]

Did you find a solution? I'm also using 1 GPU.