search

marco-rudolph / AST

This is the code to the WACV 2023 paper "Asymmetric Student-Teacher Networks for Industrial Anomaly Detection" by Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn and Bastian Wandt.
72 stars 10 forks source link

RuntimeWarning: invalid value encountered in float_scalars #15

Closed Jay-zzcoder closed 10 months ago

Jay-zzcoder commented 10 months ago

Thank you for excellent work! But when I run this code, I meet some bugs: When I train teacher model, I meet this bug: RuntimeWarning: invalid value encountered in float_scalars. The train loss is NaN. I didn't change the code.

marco-rudolph commented 10 months ago

Thanks :) Hard to say what happened there as nobody had reported something like this before... Have you installed the given requirements as specified? On which data was the model trained? Does the error occur from the first iteration or comes after some epochs/iterations? Can you inspect from which layer the problem occurs? Is the loss or some weight diverging before?

Jay-zzcoder commented 10 months ago

Thanks :) Hard to say what happened there as nobody had reported something like this before... Have you installed the given requirements as specified? On which data was the model trained? Does the error occur from the first iteration or comes after some epochs/iterations? Can you inspect from which layer the problem occurs? Is the loss or some weight diverging before?

Sorry~ Maybe I made mistakes when I preprocessed 3D data. Now I can run this code with no bugs.

jcjlin commented 9 months ago

Thanks :) Hard to say what happened there as nobody had reported something like this before... Have you installed the given requirements as specified? On which data was the model trained? Does the error occur from the first iteration or comes after some epochs/iterations? Can you inspect from which layer the problem occurs? Is the loss or some weight diverging before?

Sorry~ Maybe I made mistakes when I preprocessed 3D data. Now I can run this code with no bugs.

Hey @Jay-zzcoder would you mind to share which part did you made mistakes while preprocessing 3D data? I am facing similar issues. Thanks.