search

openai / guided-diffusion

MIT License
6.03k stars 803 forks source link

evaluation errors #94

Open zen-d opened 1 year ago

zen-d commented 1 year ago

When doing the evaluation, the following error accurs:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.                                                              
(0) INVALID_ARGUMENT: activation input is not finite. : Tensor had NaN values                                                                          [[{{node 2905231348_876199450/conv_2/CheckNumerics}}]]                                                                                 
         [[strided_slice_2/_5]]                                                                                                                 
  (1) INVALID_ARGUMENT: activation input is not finite. : Tensor had NaN values                                       
         [[{{node 2905231348_876199450/conv_2/CheckNumerics}}]]                                                                                 
0 successful operations.                                                
0 derived errors ignored.

How to fix this problem? Thanks.

Daizy97 commented 10 months ago

I meet the same problem. Do you fix it?

shalini-maiti commented 8 months ago

I have the same issue.

I'm using the Imagenet dataset -

Reference batch: VIRTUAL_imagenet256_labeled.npz Sample batch: admnet_imagenet256.npz

Error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: activation input is not finite. : Tensor had NaN values [[node 2559280021_1452120685/conv_2/CheckNumerics (defined at ..) [[strided_slice_2/_5]]

(1) Invalid argument: activation input is not finite. : Tensor had NaN values [[node 2559280021_1452120685/conv_2/CheckNumerics (defined at ..)

DuoLi1999 commented 8 months ago

Have you fix? I have the same issue.

I have the same issue.

I'm using the Imagenet dataset -

Reference batch: VIRTUAL_imagenet256_labeled.npz Sample batch: admnet_imagenet256.npz

Error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: activation input is not finite. : Tensor had NaN values [[node 2559280021_1452120685/conv_2/CheckNumerics (defined at ..) [[strided_slice_2/_5]]

(1) Invalid argument: activation input is not finite. : Tensor had NaN values [[node 2559280021_1452120685/conv_2/CheckNumerics (defined at ..)

zhangdan8962 commented 5 months ago

Anyone solved the problem?

LIzhiqian-cassie commented 5 months ago

same problem

Zyriix commented 3 months ago

I solved this by change A100 to V100. This may related with architecture of GPU.

PhilHippo commented 2 months ago

I have the same issue

ChongjianGE commented 1 month ago

Does anyone solve this?