zwbx / SHIFT-Continuous_Test_Time_Adaptation

21 stars 0 forks source link

Performance issue of the source model #13

Closed Cascol-Chen closed 9 months ago

Cascol-Chen commented 10 months ago

Runing bash run_source.sh can generate exactly the same json files in the res folder. However, the evaluation.txt reports significant better result as following:

Evaluation on all_shift_type
+---------------+-------+-------+
| Class         | IoU   | Acc   |
+---------------+-------+-------+
| building      | 48.84 | 70.3  |
| fence         | 32.35 | 44.06 |
| pedestrian    | 35.42 | 41.32 |
| pole          | 41.65 | 56.21 |
| road line     | 70.91 | 80.87 |
| road          | 93.64 | 97.61 |
| sidewalk      | 68.52 | 71.38 |
| vegetation    | 69.3  | 77.89 |
| vehicle       | 69.14 | 97.69 |
| wall          | 32.56 | 47.22 |
| traffic sign  | 46.83 | 52.88 |
| sky           | 75.18 | 80.03 |
| traffic light | 26.65 | 30.79 |
| terrain       | 50.09 | 69.04 |
| *Average*     | 54.36 | 65.52 |
+---------------+-------+-------+
Evaluation on daytime_to_night
+---------------+-------+-------+
| Class         | IoU   | Acc   |
+---------------+-------+-------+
| building      | 88.55 | 94.11 |
| fence         | 37.64 | 46.89 |
| pedestrian    | 50.6  | 57.86 |
| pole          | 44.06 | 50.39 |
| road line     | 77.6  | 85.51 |
| road          | 95.19 | 97.52 |
| sidewalk      | 85.32 | 89.11 |
| vegetation    | 71.81 | 82.48 |
| vehicle       | 85.51 | 97.01 |
| wall          | 0.0   | 0.0   |
| traffic sign  | 71.66 | 79.24 |
| sky           | 87.21 | 89.84 |
| traffic light | 23.57 | 27.1  |
| terrain       | 41.49 | 56.2  |
| *Average*     | 61.44 | 68.09 |
+---------------+-------+-------+
Evaluation on clear_to_rainy
+---------------+-------+-------+
| Class         | IoU   | Acc   |
+---------------+-------+-------+
| building      | 13.62 | 69.37 |
| fence         | 48.84 | 79.64 |
| pedestrian    | 29.83 | 36.44 |
| pole          | 45.36 | 65.09 |
| road line     | 61.22 | 73.66 |
| road          | 96.76 | 97.85 |
| sidewalk      | 86.66 | 89.56 |
| vegetation    | 71.75 | 80.78 |
| vehicle       | 38.77 | 98.69 |
| wall          | 65.12 | 94.44 |
| traffic sign  | 8.95  | 13.05 |
| sky           | 56.78 | 57.05 |
| traffic light | 62.53 | 72.65 |
| terrain       | 62.31 | 79.44 |
| *Average*     | 53.46 | 71.98 |
+---------------+-------+-------+
Evaluation on clear_to_foggy
+---------------+-------+-------+
| Class         | IoU   | Acc   |
+---------------+-------+-------+
| building      | 0.14  | 0.74  |
| fence         | 0.0   | 0.0   |
| pedestrian    | 1.05  | 1.43  |
| pole          | 27.01 | 55.92 |
| road line     | 70.19 | 81.39 |
| road          | 82.74 | 97.41 |
| sidewalk      | 0.0   | 0.0   |
| vegetation    | 56.88 | 58.36 |
| vehicle       | 80.8  | 97.75 |
| traffic sign  | 72.92 | 79.81 |
| sky           | 75.89 | 96.54 |
| traffic light | 0.0   | 0.0   |
| terrain       | 51.45 | 86.77 |
| *Average*     | 37.08 | 46.87 |
+---------------+-------+-------+