search

gathierry / FastFlow

Apache License 2.0
132 stars 24 forks source link

AUROC reach peak in several epochs and then drop #4

Open ZwkAlex opened 2 years ago

ZwkAlex commented 2 years ago

auroc seems reach the peak in several epoch (5-10 epoch), then drop and never ascent after the peak, any reson for that? thanks so much.

gathierry commented 2 years ago

I have similar observations as well and I don't have a clear answer about that. Therefore, I put both results from best epoch and last epoch. I'll leave this issue open to see if there are better answers. And I might change the title to be more explicit.

founderlin commented 2 years ago

I had the same issue. Did anyone compare this version with the one in anomalib? The fastflow model in anomalib has a more stable and higher AUROC. Is there any tricks in anomalib?

I have similar observations as well and I don't have a clear answer about that. Therefore, I put both results from best epoch and last epoch. I'll leave this issue open to see if there are better answers. And I might change the title to be more explicit.

cytotoxicity8 commented 2 years ago

It might be because of early stopping. Model architectures are almost identical.

WiillyWonka commented 4 months ago

I faced the same issue. I think the problem is in the log likelihood calculation for anomaly map construction. In this line https://github.com/gathierry/FastFlow/blob/2cf1f2f4c562a7f13cfb1959e3afe5df2f2d2565/fastflow.py#L148 the log likelihood must be calculated as -torch.mean(output**2, dim=1, keepdim=True) * 0.5 + log_jac_dets. This is consistent with the formula for variable substitution in the distribution. Also, it helped me. Now the learning process has become stable and doesn't drop.