$HOME/data/dataset/CMIP5.input* file not found ?

[JingliangGao]

Hi, In order to training with CMIP5( CNN/csh/nino34.cmip.csh ), I have run the command csh nino34.cmip.csh, but still come out a problem : " No such file or directory : '$HOME/data/dataset/CMIP5.input.36mon.*.nc' " Should I still need to change somewhere to run this script successfuly? Anyway, thank you for generously sharing your code and I look forward to your reply.

[JingliangGao]

@jeonghwan723

[jeonghwan723]

Dear, @JingliangGao I'm so sorry for the late reply.

The dataset is too large (1.1GB) for upload to github. You can download dataset here: https://drive.google.com/open?id=1cbeA3pGF9Ls-U805uFIwa21DhzMRD2eK

The dataset consists of the following: Training set: ‘CMIP5.input.36mon.1861_2001.nc’, ‘CMIP5.label.12mon.1863_2003.nc’ Training set for transfer learning (i.e. fine tuning): ‘SODA.input.36mon.1871_1970.nc’, ‘SODA.label.12mon.1873_1972.nc’ Validation set: ‘GODAS.input.36mon.1980_2015.nc’, ‘GODAS.label.12mon.1982_2017.nc’

The additional information:

The z dimension of input defined as consecutive 36 months from January (-2yr) to December (0yr). The python code (nino34.training.sample) automatically extracts three months to match target season and lead month (schematics: https://drive.google.com/open?id=1Bdln6vbMlUxco-fcsJQJsfOKbaDujVJs). Input and label are set to retrieve netCDF4 files and there should be no missing value (recommended to insert zeros instead of missing value). Validation result is set to be saved as a 4-byte binary format without header information.

Jeong-Hwan

[kirands16]

Hi, This is really a nice paper i gone thorugh and curious to obtain same result with ur sample data. thanks for sharing. I am trying to training with CMIP5( CNN/csh/nino34.cmip.csh ), I have run the command csh nino34.cmip.csh, but still come out a problem : in code python nino34.train_cmip.py, valid, and ensemble python code is called . i am getting error as File "nino34.train_cmip.py", line 6, in from netCDF4 import Dataset my module available is netcdf/4.0 but it looked for netCDF4 and unable to find nino34.train_cmip.py, in which i have to change. Please help me for running the model the code i did it on centos 7 system miniconda . looking forward for your kind reply. Thanks in advance.

[jeonghwan723]

Dear, @kirands16

Thanks for your warm word. Did you activate your miniconda environment before running the code? (e.g. "conda activate your_env_name" or "activate your_env_name")

Or, you can check it by opening the python console.

Good luck :)

[kirands16]

Thanks for your quick response. I tried it.

Traceback (most recent call last): File "", line 1, in ImportError: No module named netcdf4

module av netcdf/4.0

---------------------------------------------- /gpfs1/home/Libs/GNU/MODULE/modules-3.2.10/Modules/INTEL ---------------------------------------------- netcdf/4.0 netcdf/4.0.1 where can i find nino34.train_cmip.py

[kirands16]

module load GNU-conda/miniconda python2.7 import tensorflow

[kirands16]

following error i got now [kiran@iitmlogin5 csh]$ ./nino34.cmip.csh nino34_2month_1 Traceback (most recent call last): File "nino34.train_cmip.py", line 32, in inp1 = Dataset('/home/jhkim/data/dataset/CMIP5.input.36mon.1861_2001.nc','r') File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.init File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success IOError: [Errno 2] No such file or directory: '/home/jhkim/data/dataset/CMIP5.input.36mon.1861_2001.nc' sed: -e expression #1, char 19: unknown option to s' Traceback (most recent call last): File "nino34.train_cmip.py", line 32, in <module> inp1 = Dataset('/home/jhkim/data/dataset/CMIP5.input.36mon.1861_2001.nc','r') File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.__init__ File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success IOError: [Errno 2] No such file or directory: '/home/jhkim/data/dataset/CMIP5.input.36mon.1861_2001.nc' sed: -e expression #1, char 19: unknown option tos' Traceback (most recent call last): File "nino34.train_cmip.py", line 32, in inp1 = Dataset('/home/jhkim/data/dataset/CMIP5.input.36mon.1861_2001.nc','r') File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.init File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success IOError: [Errno 2] No such file or directory: '/home/jhkim/data/dataset/CMIP5.input.36mon.1861_2001.nc' sed: -e expression #1, char 19: unknown option to `s'

[jeonghwan723]

Dear, @kirands16

I'm sorry for my mistake. There was some mistake line 32, in [CNN/sample/nino34.train_cmip.sample]. Also, this code does not contain a data set (this code is just a sample).

So, I attaching modified version (following below link) with data set. https://drive.google.com/open?id=1mGxNSvrW6fewdcT2qLIDNYDuFArWLOnK

And, this code will not work in TensorFlow version 2.0 or later.

[kirands16]

Dear @jeonghwan723 Thanks Sir, for your kind reply...I tried it it works. There is no error only input file path is incorrect in nino34.train_cmip.py code, otherwise the script will work nino34.cmip.csh. I just want python code for nino34.train_cmip.py, nino34.valid.py, nino34.ensmean.py, nino34.heatmap.py,nino34.train_transfer.py,nino_type.train_cmip.py,nino_type.valid.py, nino_type.ensmean.py and nino_type.heatmap.py which are called in .csh script.

Thanks againt sir.

[jeonghwan723]

Dear @kirands16

You can find the python code that called in .csh script in the directory [CNN/sample/]. ~.sample files are the python code you are looking for (i.e. nino34.tran_cmip.sample, nino34.valid.sample, nino34.ensmean.sample, ...).

Also, you can find this information in the line 54~78 in the nino34.cmip.csh.

Have a nice day.

[kirands16]

Thank you Sir,

[kirands16]

Dear @jeonghwan723 , will you please share the dataset for DL_ENSO/figure/fig*py
Thanks alot....

[jatinkumar762]

Dear, @JingliangGao I'm so sorry for the late reply.

The dataset is too large (1.1GB) for upload to github. You can download dataset here: https://drive.google.com/open?id=1cbeA3pGF9Ls-U805uFIwa21DhzMRD2eK

The dataset consists of the following: Training set: ‘CMIP5.input.36mon.1861_2001.nc’, ‘CMIP5.label.12mon.1863_2003.nc’ Training set for transfer learning (i.e. fine tuning): ‘SODA.input.36mon.1871_1970.nc’, ‘SODA.label.12mon.1873_1972.nc’ Validation set: ‘GODAS.input.36mon.1980_2015.nc’, ‘GODAS.label.12mon.1982_2017.nc’

The additional information:

The z dimension of input defined as consecutive 36 months from January (-2yr) to December (0yr). The python code (nino34.training.sample) automatically extracts three months to match target season and lead month (schematics: https://drive.google.com/open?id=1Bdln6vbMlUxco-fcsJQJsfOKbaDujVJs). Input and label are set to retrieve netCDF4 files and there should be no missing value (recommended to insert zeros instead of missing value). Validation result is set to be saved as a 4-byte binary format without header information.

Jeong-Hwan

Hi Sir, your google drive links are not working. Could you please update it? With Regards & Thanks, Jatin Kumar

[jeonghwan723]

Oh, that's my mistake. You can download dataset here: https://drive.google.com/file/d/1O-JkOh0Az_NrojietqPCZb809S-jCDps/view?usp=sharing

[jatinkumar762]

Oh, that's my mistake. You can download dataset here: https://drive.google.com/file/d/1O-JkOh0Az_NrojietqPCZb809S-jCDps/view?usp=sharing

Thanks a lot, Sir for your reply. I already downloaded dataset from the link given in readme.md file. I was asking about the second link in which you mentioned schematics.

I have one another doubt also...during training time which 3 months we extract from "sst" and "t300" in nino_type.train_cmip.sample file. In python3 version, it's not automatically mapping. During training, there is one another error comes in model in the same file nino_type.train_cmip.sample that y size is not compatible.

Sir, please guide in these issues. I will always be grateful to you for your support.

[jeonghwan723]

Dear, @jatinkumar762

I'm so sorry, that's are dataset for Nino3.4 forecast. I missed the dataset for El Nino type forecast. We conducted only 12-months DJF El Nino type forecast, so the seasons of input and label were fixed to NDJ and DJF, respectively.

Please use 'CMIP5.input.type.NDJ.1861_2001.nc' and 'CMIP5.label.one_hot.DJF.1863_2003.nc' for training, 'GODAS.input.type.NDJ.1990_2015.nc' and 'GODAS.label.one_hot.DJF.1982_2017.nc' for validation.

Please download the dataset through the following link: https://drive.google.com/file/d/11MdK0_KHjEQQ7IKsfmwFBzgJgkAgP6rV/view?usp=sharing

[jatinkumar762]

Dear Sir, Could you please tell which 13 years of data present in file GODAS.input.type.NDJ.1980_2015.nc for Nino type validation? Because the data dimension in the file is (13, 3, 24, 72). In the output we are getting 0,1 or 2. Could you please tell which number corresponds to which type?

[jeonghwan723]

Dear @jatinkumar762,

This is because 13 El Niño events occurred between 1981 and 2016. Please refer the extended data table 4 of Ham et al., 2019 paper (https://www.nature.com/articles/s41586-019-1559-7/tables/4).

El Nino type was classified according to the critetia defined in "Zhang, Z., Ren, B. & Zheng, J. A unified complex index to characterize two types of ENSO simultaneously. Sci. Rep. 9, 8373 (2019)" and then encoded it as one-hot vector. 0, 1, and 2 are denotes EP type, CP type and Mixed type respectively.

[icodeworld]

Dear Sir, Could you please tell how correlation lines of different models except CNN in fig.2 generates? Thank you very much?

[jeonghwan723]

Dear @icodeworld , Please see line 131-140 of figure/figure2.py .

[icodeworld]

Sir, thank you very much!

[icodeworld]

Could you tell me how the Nino3 index and Nino4 index are calculated in your paper?  So that I can reproduce the UCEI, r, and θ  in this paper, and then reproduce the ENSO types in the column 'OBS' of Extended Data Table 4. Reference 24 did not provide the code, and the ENSO types it calculated seems not identical to the 'OBS' in Extended Data Table 4 in your paper.

[jeonghwan723]

Dear @icodeworld,

I attached dataset for El Nino type forecast. Please download the dataset through the following link: https://drive.google.com/file/d/1Ht6__G4bFWguZTJ3nKc3XuEY1KKLMIIN/view?usp=sharing

[xiaoguoshidai-byte]

Dear Sir, Could you please tell me why could not broadcast input array from shape (877,3,24,72) into shape (1178,3,24,72) when i was training nino_type.train_cmip.py and how can i solve it. Thank you very much! ![Uploading 1.png…]()

[Song-jinshui]

Dear @icodeworld,

I attached dataset for El Nino type forecast. Please download the dataset through the following link: https://drive.google.com/file/d/1Ht6__G4bFWguZTJ3nKc3XuEY1KKLMIIN/view?usp=sharing

Thank you,Sir! I've been looking for these data sets for three days.