delve-team / delve

PyTorch model training and layer saturation monitor
https://delve-docs.readthedocs.io
MIT License
79 stars 13 forks source link

TypeError with pytest #32

Closed JustinShenk closed 3 years ago

JustinShenk commented 3 years ago

Running py.test,

https://github.com/delve-team/delve/blob/6a2b594eb9ce43c38c9f94be8781ea8c57610de2/delve/writers.py#L469 returns a TypeError:

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''.

Contents of df.values[0]:

array([list([(tensor(-1.5482e-08, dtype=torch.float64), tensor(-1.3955e-09, dtype=torch.float64)), (tensor(-2.5362e-08, dtype=torch.float64), tensor(-2.5179e-09, dtype=torch.float64)), (tensor(-3.1511e-08, dtype=torch.float64), tensor(-3.6894e-09, dtype=torch.float64)), (tensor(-3.5553e-08, dtype=torch.float64), tensor(-4.1750e-09, dtype=torch.float64)), (tensor(-3.8271e-08, dtype=torch.float64), tensor(-4.4061e-09, dtype=torch.float64)), (tensor(-3.7972e-08, dtype=torch.float64), tensor(-2.7664e-09, dtype=torch.float64)), (tensor(-3.7489e-08, dtype=torch.float64), tensor(-1.7852e-09, dtype=torch.float64)), (tensor(-3.7178e-08, dtype=torch.float64), tensor(-1.3027e-09, dtype=torch.float64))])],
      dtype=object)
MLRichter commented 3 years ago

Hi, posted an update. I kept the try catch block, since I want this tool to be as non-blocking as possible for the training. However the stack trace should now be logged correctly. Also, the tested scenario is now caught over if-else, similar to the code you posted.

JustinShenk commented 3 years ago

Closed by #41