search

nansencenter / sea_ice_type_cnn_training

Deep learning of satellite data: Use the data from satellites for machine learning (deep learning) purposes
GNU General Public License v3.0
5 stars 4 forks source link

Nan values in .npz files #16

Closed Alissa13777 closed 3 years ago

Alissa13777 commented 3 years ago

When running the training on the .npz files I have got the loss function equal to nan. When debugging, I found that there a nan values in the .npz files (in the matrixes for 'nersc_sar_primary' and 'nersc_sar_secondary'). I have run the python train_model.py -o /fold2 -bs 4 -p 0.8 -see -sft on the .npz files extracted from the following .nc files:

To find the nan values, I have run a Python script that stocks the different values of each file and then did a CTRL+F. In the Method 2 of the python script (https://github.com/Alissa13777/Internship_NERSC_CNN_IceTypes), one can see the file in which there are nan values in the shell.

opsdep commented 3 years ago

@Alissa13777 solved with #18

opsdep commented 3 years ago

Not completely solved by #18.

Alissa13777 commented 3 years ago

Solved the issue with a filtering script. (nan values also in btemp_69h and btemp_69v files). Putting the script on personal github.

Alissa13777 commented 3 years ago

Mask not working properly in archive.py in get_the_mask_of_amsr2_data(self, amsr_labels, fil) function (line 297). There are nan values after the mask is calculated and applied.

opsdep commented 3 years ago

@Alissa13777 Have you run the code with what is inside the test branch? I have fixed both two bugs previous week in that PR.

opsdep commented 3 years ago

If not, run the building with the new code. Could you please run this script in a seperate file and screenshot the resutl of your execution?

import os
import numpy as np

file_list = os.listdir("output")
for file_ in file_list:
    if file_.endswith('.npz'):
        v=np.load(os.path.join("output",file_))
        for variable in v.files:
            print(">>>>>>>>>>>variable:", variable)
            print(">>>>>>>>>>>file:", file_)
            assert None not in v[variable]

I have run it and everything is correct as shown below: image

opsdep commented 3 years ago

We have tests for making sure that the function works properly. Have at these two tests: 1. https://github.com/nansencenter/satellite-data-pipeline/blob/b1364be823b4e2cd25aadaf6ad871c7a03656c95/asip_v2/tests/test_archive.py#L305

2. https://github.com/nansencenter/satellite-data-pipeline/blob/b1364be823b4e2cd25aadaf6ad871c7a03656c95/asip_v2/tests/test_archive.py#L316

akorosov commented 3 years ago

Confirmed as solved by Alissa