search

ftbabi / LayersNet_ICCV2023

Apache License 2.0
38 stars 4 forks source link

NaN Values Detected in Testing #4

Open kai-lan opened 3 months ago

kai-lan commented 3 months ago

I used the network provided here: https://drive.google.com/file/d/1okojvkl7AzLXm2aqrlKZaB2yK6FaRM6I/view.

I generated data using

python tools/preprocessing_data.py configs/layersnet/base/ep1.py --work_dir output --dataset test --type dynamic

and 

python tools/preprocessing_data.py configs/layersnet/base/ep1.py --work_dir output --dataset test --type static

Then I tested using

python tools/test.py configs/layersnet/base/ep1.py data/ckpt.pth --show-dir output --show-options rollout=396

However, it has the assertion error

File "SomeDIR/LayersNet_ICCV2023/layersnet/models/utils/norm_func.py", line 54, in attn_rotation
    assert torch.where(torch.isnan(self_attn_normed))[0].size(0) == 0
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Why did I get NaN values?

ftbabi commented 2 months ago

Hi Kai,

You may try the "test.py" config first:

python tools/test.py configs/layersnet/base/test.py ...

BTW, did you generate the training data also? The inputs for the model are normalized by the mean and std calculated on the training set and stored in "stat.h5". The "stat.h5" contains meaningful information only when generating the data from training sequences. Please refer to "LayersNet_ICCV2023/layersnet/datasets/layers_dynamic_dataset.py" at L193.

Best, Yidi