search

gstoica27 / ZipIt

A framework for merging models solving different tasks with different initializations into one multi-task model without any additional training
MIT License
286 stars 25 forks source link

pdb at line 51 in the file metric_calculators.py #19

Open nsaadati opened 1 year ago

nsaadati commented 1 year ago

hi can you please expline why you put this condition at line 51 in the file metric_calculators.py, I'm getting this error a lot and im not sure what is the resaon for that def finalize(self, numel, eps=1e-4): self.outer /= numel self.mean /= numel self.std /= numel cov = self.outer - torch.outer(self.mean, self.mean) if torch.isnan(cov).any(): breakpoint() if (torch.diagonal(cov) < 0).sum(): pdb.set_trace() return cov

ansharora7 commented 1 year ago

Hi, I am facing the same problem. Did you find the solution?

gstoica27 commented 11 months ago

Hi all,

Are you still facing this issue? If so, is it with models from our experiment settings? Or from new ones that you have implemented?

adampan0527 commented 1 month ago

Hi all,

Are you still facing this issue? If so, is it with models from our experiment settings? Or from new ones that you have implemented?

Hi, I met same issue, the model is from your experiment settings(resnet20), nothing changed, but found cov has 1 element <0, can you tell me how can covariance<0 happen? And what's the problem may be. I trained another pair of models, and this time everything goes well, is this problem happens randomly?

gstoica27 commented 1 month ago

What's the value of the element that is < 0? Is it near zero? I'm wondering if it's because of numerical precision.

adampan0527 commented 3 weeks ago

What's the value of the element that is < 0? Is it near zero? I'm wondering if it's because of numerical precision.

Really close, about 1e-6 or -7, I agree with numerical precision problem, thx a lot