Incoherent logged val/iou value despite coherent confusion matrix and per-class IoUs when training model in DDP

IGNF / myria3d

Myria3D: Aerial Lidar HD Semantic Segmentation with Deep Learning

https://ignf.github.io/myria3d/

BSD 3-Clause "New" or "Revised" License

151 stars 20 forks source link

Incoherent logged val/iou value despite coherent confusion matrix and per-class IoUs when training model in DDP #122

Open CharlesGaydon opened 2 months ago

CharlesGaydon commented 2 months ago

Observed in DDP mode with 2x3 GPUs (not tested on single node training).

This is surprising, because the per-class IoUs are computed with a bespoke method from the confusion matrix of the MultiClassJaccardIndex object that computes mean IoU

https://github.com/IGNF/myria3d/blob/d52810eb6024c6b4dbb7816ff3be83fb1dd732b0/myria3d/models/model.py#L190

Mean IoU:

Per-class:

Might be linked to https://github.com/IGNF/myria3d/issues/108

CharlesGaydon commented 2 months ago

Note: maybe https://github.com/IGNF/myria3d/pull/124 has corrected this since the computation of the IoU is now performed using a different function. Feel free to close if you do not observe this behavior again.