Open durack1 opened 9 years ago
Same issue, historical experiment:
Wrote file: /work/durack1/Shared/150313_data_density/historical/cmip5.MPI-ESM-P.historical.r1i1p1.an.ocn.Omon.density.ver-1.nc
FileCount: 45
outPath: /work/durack1/Shared/150313_data_density/historical
outfile: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.density.ver-v20120510.latestX.WARN2.xml.nc
so: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.so.ver-v20120510.latestX.WARN2.xml
thetao: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.thetao.ver-v20120510.latestX.WARN2.xml
areacello: cmip5.MRI-CGCM3.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-v20110831.latestX.WARN2.xml
grdsize: 6756480
==> model: MRI-CGCM3 (grid size: 6756480 )
==> time interval: 0 1871
==> size of time chunk, number of time chunks (memory optimization) : 12 156
--> time chunk (bounds) = 1 / 156 ( 0 11 ) MRI-CGCM3
Traceback (most recent call last):
File ".//drive_density.py", line 262, in <module>
densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin
nomask = npy.equal(vmask_3D[0],0) ; # Returns boolean
IndexError: invalid index to scalar variable.
ok I will look into this.
On 13/4/15 19:38, Paul J. Durack wrote:
Same issue, historical experiment:
| Wrote file: /work/durack1/Shared/150313_data_density/historical/cmip5.MPI-ESM-P.historical.r1i1p1.an.ocn.Omon.density.ver-1.nc FileCount: 45 outPath: /work/durack1/Shared/150313_data_density/historical outfile: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.density.ver-v20120510.latestX.WARN2.xml.nc so: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.so.ver-v20120510.latestX.WARN2.xml thetao: cmip5.MRI-CGCM3.historical.r1i1p1.mo.ocn.Omon.thetao.ver-v20120510.latestX.WARN2.xml areacello: cmip5.MRI-CGCM3.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-v20110831.latestX.WARN2.xml grdsize: 6756480 ==> model: MRI-CGCM3 (grid size: 6756480 ) ==> time interval: 0 1871 ==> size of time chunk, number of time chunks (memory optimization) : 12 156
--> time chunk (bounds) = 1 / 156 ( 0 11 ) MRI-CGCM3 Traceback (most recent call last): File ".//drive_density.py", line 262, in
densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all') File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin nomask = npy.equal(vmask_3D[0],0) ; # Returns boolean IndexError: invalid index to scalar variable. | â Reply to this email directly or view it on GitHub https://github.com/eguil/Density_bining/issues/40#issuecomment-92441688.
Eric Guilyardi IPSL/LOCEAN - Dir. Rech. CNRS Tour 45, 4eme, piece 406 UPMC, case 100 4 place Jussieu, F-75252 Paris Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi NCAS Climate Meteorology Department University of Reading Reading RG6 6BB - UK Tel: +44 (0)118 378 8315
http://ncas-climate.nerc.ac.uk/~ericg
@eguil these problems are most likely the inputs - the files include the .WARN2.
identifier - so just leave this up to me.. I'm just restarting for the moment, will have to go and revisit once I get to the end of the available files.. We're still running out of memory on MIROC4h and MPI.. But again will just keep pushing this until I get to the end before digging deeper..
@durack1 ok fine
On 13/4/15 23:14, Paul J. Durack wrote:
@eguil https://github.com/eguil these problems are most likely the inputs - the files include the |.WARN2.| identifier - so just leave this up to me.. I'm just restarting for the moment, will have to go and revisit once I get to the end of the available files.. We're still running out of memory on MIROC4h and MPI.. But again will just keep pushing this until I get to the end before digging deeper..
â Reply to this email directly or view it on GitHub https://github.com/eguil/Density_bining/issues/40#issuecomment-92500812.
Eric Guilyardi IPSL/LOCEAN - Dir. Rech. CNRS Tour 45, 4eme, piece 406 UPMC, case 100 4 place Jussieu, F-75252 Paris Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi NCAS Climate Meteorology Department University of Reading Reading RG6 6BB - UK Tel: +44 (0)118 378 8315
http://ncas-climate.nerc.ac.uk/~ericg
And another: historicalNat cmip5.CESM1-CAM5.historicalNat.r1i1p1.an.ocn.Omon.density.ver-v20130909.nc
--> time chunk (bounds) = 88 / 156 ( 1044 1055 ) CESM1-CAM5
*THETAO mean: +281.82; min: +270.23; max: +306.29
*THETAO mean: +08.67; min: -02.92; max: +33.14
Total volume in z coordinates source grid (ref = 1.33 e+18) : 1.32186397697e+18
Mean Temp./Salinity in z coordinates source grid : 3.49228724533 34.7171399393
Bining CPU analysis tc/tcdel = 87 12
average cpu1 = 0.542500000001
average cpu2 = 2.275
average cpu3 = 0.321666666667
average cpu4 = 27.07
average cpu40 = 0.0908333333349
average cpu5 = 0.598333333332
CPU read T/S = 7.07
CPU comp. rho = 13.21
Integral persistent values: 1.24750987849e+18 1.19516e+18 3.01982395738e+17
% 95.8034683786 25.2671582723
area global/atl/pac/ind 3.44248e+14 9.7772e+13 1.73523e+14 7.29536e+13
T , S glob 3.44037747224 33.0646674819
T , S atl 3.86833315611 31.8864366555
T , S pac 3.09515285816 33.3947889029
T , S ind 3.68796375976 33.8585274262
Persistence CPU analysis t/nyrtc = 0 1
cpu1 = 21.74
cpu2 = 0.0
cpu3 = 0.0
cpu4 = 0.0
cpu5 = 0.0
cpu6 = 0.0
cpu7 = 0.0
CPU of chunk inits = 21.91
CPU of density bining = 370.78
CPU of masking and var def = 8.13
CPU of annual mean compute = 4.83
CPU of interpolation = 5.96
CPU of zonal mean = 2.65
CPU of persistence compute = 21.74
CPU of chunk = 436.0
Max memory use 17.77614 GB
--> time chunk (bounds) = 89 / 156 ( 1056 1067 ) CESM1-CAM5
*THETAO mean: +281.82; min: +270.04; max: +305.50
*THETAO mean: +08.67; min: -03.11; max: +32.35
Total volume in z coordinates source grid (ref = 1.33 e+18) : 1.32186397697e+18
Mean Temp./Salinity in z coordinates source grid : 3.49275783464 34.7171368331
Bining CPU analysis tc/tcdel = 88 12
average cpu1 = 0.176666666667
average cpu2 = 2.17083333333
average cpu3 = 0.286666666666
average cpu4 = 20.3491666667
average cpu40 = 0.0716666666667
average cpu5 = 0.449166666665
CPU read T/S = 3.81
CPU comp. rho = 14.36
Traceback (most recent call last):
File ".//drive_density.py", line 262, in <module>
densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
File "/export/durack1/git/Density_bining/binDensity.py", line 969, in densityBin
maskp = npy.ma.reshape(maskp, (N_s+1, latN*lonN))
File "/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/numpy/ma/core.py", line 6508, in reshape
return a.reshape(new_shape, order=order)
ValueError: total size of new array must be unchanged
These two issues are likely due to non-z vertical coordinates..
And another (this issue is also the case with r2i1p1
through r5i1p1
):
FileCount: 41
outPath: /work/durack1/Shared/150313_data_density/historical
outfile: cmip5.MIROC5.historical.r1i1p1.mo.ocn.Omon.density.ver-1.nc
so: cmip5.MIROC5.historical.r1i1p1.mo.ocn.Omon.so.ver-1.latestX.xml
thetao: cmip5.MIROC5.historical.r1i1p1.mo.ocn.Omon.thetao.ver-1.latestX.xml
areacello: cmip5.MIROC5.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-1.latestX.xml
grdsize: 2867200
==> model: MIROC5 (grid size: 2867200 )
==> time interval: 0 1955
==> size of time chunk, number of time chunks (memory optimization) : 12 163
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1241: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
if (srcMaskValues != None):
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1248: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
if (dstMaskValues != None):
--> time chunk (bounds) = 1 / 163 ( 0 11 ) MIROC5
Traceback (most recent call last):
File ".//drive_density.py", line 262, in <module>
densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin
nomask = npy.equal(vmask_3D[0],0) ; # Returns boolean
IndexError: invalid index to scalar variable.
And another:
FileCount: 46
outPath: /work/durack1/Shared/150313_data_density/historical
outfile: cmip5.MRI-ESM1.historical.r1i1p1.mo.ocn.Omon.density.ver-v20130307.latestX.WARN2.xml.nc
so: cmip5.MRI-ESM1.historical.r1i1p1.mo.ocn.Omon.so.ver-v20130307.latestX.WARN2.xml
thetao: cmip5.MRI-ESM1.historical.r1i1p1.mo.ocn.Omon.thetao.ver-v20130307.latestX.WARN2.xml
areacello: cmip5.MRI-ESM1.rcp85.r0i0p0.fx.ocn.fx.areacello.ver-v20140210.latestX.xml
grdsize: 6756480
==> model: MRI-ESM1 (grid size: 6756480 )
==> time interval: 0 1859
==> size of time chunk, number of time chunks (memory optimization) : 12 155
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1241: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
if (srcMaskValues != None):
/usr/local/uvcdat/2014-11-24/lib/python2.7/site-packages/ESMP/src/ESMP_API.py:1248: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
if (dstMaskValues != None):
--> time chunk (bounds) = 1 / 155 ( 0 11 ) MRI-ESM1
Traceback (most recent call last):
File ".//drive_density.py", line 262, in <module>
densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
File "/export/durack1/git/Density_bining/binDensity.py", line 595, in densityBin
nomask = npy.equal(vmask_3D[0],0) ; # Returns boolean
IndexError: invalid index to scalar variable.
And another:
FileCount: 152
outPath: /work/durack1/Shared/150313_data_density/historical
outfile: cmip5.MIROC4h.historical.r2i1p1.mo.ocn.Omon.density.ver-1.nc
so: cmip5.MIROC4h.historical.r2i1p1.mo.ocn.Omon.so.ver-1.latestX.xml
thetao: cmip5.MIROC4h.historical.r2i1p1.mo.ocn.Omon.thetao.ver-1.latestX.xml
areacello: cmip5.MIROC4h.rcp45.r0i0p0.fx.ocn.fx.areacello.ver-1.latestX.xml
grdsize: 56033280
==> model: MIROC4h (grid size: 56033280 )
==> time interval: 0 671
==> size of time chunk, number of time chunks (memory optimization) : 12 56
--> time chunk (bounds) = 1 / 56 ( 0 11 ) MIROC4h
trigger mask fix - EC-EARTH/MIROC4h
350524236
*THETAO mean: +281.47; min: +271.11; max: +309.97
*THETAO mean: +08.33; min: -02.04; max: +36.82
Total volume in z coordinates source grid (ref = 1.33 e+18) : 1.36475536317e+18
Mean Temp./Salinity in z coordinates source grid : 4.03157081096 34.7199550623
Bining CPU analysis tc/tcdel = 0 12
average cpu1 = 1.42166666667
average cpu2 = 21.6533333333
average cpu3 = 2.38583333333
average cpu4 = 236.875
average cpu40 = 1.09749999999
average cpu5 = 4.55416666667
CPU read T/S = 283.05
CPU comp. rho = 248.47
0.0 0.0 0.0
0.0 0.0 0.0
Total volume in rho coordinates source grid (ref = 1.33 e+18) : 1.31614338386e+18
Mean Temp./Salinity in rho coordinates source grid : 4.06755384743 34.8811525226
Traceback (most recent call last):
File ".//drive_density.py", line 262, in <module>
densityBin(model[3],model[1],model[5],outfileDensity,debug=True,timeint='all')
File "/export/durack1/git/Density_bining/binDensity.py", line 961, in densityBin
idxvm = 1-mv.masked_values(thickBin[inim:finm,:,:,:], valmask).mask
File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/cdms2/MV2.py", line 867, in masked_values
maresult = numpy.ma.masked_values(_makeMaskedArg(data), value, rtol=rtol, atol=atol, copy=copy)
File "/usr/local/uvcdat/2.2.0/lib/python2.7/site-packages/numpy/ma/core.py", line 2193, in masked_values
condition = umath.less_equal(mabs(xnew - value), atol + rtol * mabs(value))
MemoryError
Revisit as specific test cases in #33 - test with new code and new ESMPy
and