Closed apatlpo closed 5 years ago
This error indicates that Python is running out of memory. Dask (which we use with mfdataset) can help with that, but it doesn't always solve the issue.
How is the data arranged in each of the input files? What version of dask are you using? This might be an issue with dask.array not fusing calls to __getitem__
and loading entire files into memory.
cc @mrocklin @jcrist
Thanks for your answer. You'll find below an output from ncdump in order to answer the part about data arrangement (I hope ...) Otherwise, we have:
dask 0.8.2 py27_0 defaults
xarray 0.7.2 py27_0 defaults
netcdf4 1.1.1 np18py27_0 defaults
Looking forward for any suggestions. cheers aurelien
netcdf NATL60-MJM155_y2008m01d09.5d_gridT {
dimensions:
x = 5422 ;
y = 3454 ;
deptht = 300 ;
time_counter = UNLIMITED ; // (1 currently)
time_bounds = 2 ;
variables:
float nav_lat(y, x) ;
nav_lat:axis = "Y" ;
nav_lat:standard_name = "latitude" ;
nav_lat:long_name = "Latitude" ;
nav_lat:units = "degrees_north" ;
nav_lat:nav_model = "grid_T" ;
nav_lat:_Storage = "chunked" ;
nav_lat:_ChunkSizes = 12, 5422 ;
float nav_lon(y, x) ;
nav_lon:axis = "X" ;
nav_lon:standard_name = "longitude" ;
nav_lon:long_name = "Longitude" ;
nav_lon:units = "degrees_east" ;
nav_lon:nav_model = "grid_T" ;
nav_lon:_Storage = "chunked" ;
nav_lon:_ChunkSizes = 12, 5422 ;
float deptht(deptht) ;
deptht:axis = "Z" ;
deptht:long_name = "Vertical T levels" ;
deptht:units = "m" ;
deptht:positive = "down" ;
deptht:_Storage = "chunked" ;
deptht:_ChunkSizes = 300 ;
float votemper(time_counter, deptht, y, x) ;
votemper:long_name = "temperature" ;
votemper:units = "degC" ;
votemper:online_operation = "average" ;
votemper:interval_operation = "40s" ;
votemper:interval_write = "5d" ;
votemper:_FillValue = 0.f ;
votemper:missing_value = 0.f ;
votemper:coordinates = "time_centered deptht nav_lon nav_lat" ;
votemper:_Storage = "chunked" ;
votemper:_ChunkSizes = 1, 1, 12, 5422 ;
votemper:_DeflateLevel = 1 ;
double time_centered(time_counter) ;
time_centered:standard_name = "time" ;
time_centered:long_name = "Time axis" ;
time_centered:title = "Time" ;
time_centered:calendar = "gregorian" ;
time_centered:units = "seconds since 1958-01-01 00:00:00" ;
time_centered:time_origin = "1958-01-01 00:00:00" ;
time_centered:bounds = "time_centered_bounds" ;
time_centered:_Storage = "chunked" ;
time_centered:_ChunkSizes = 1 ;
double time_centered_bounds(time_counter, time_bounds) ;
time_centered_bounds:_Storage = "chunked" ;
time_centered_bounds:_ChunkSizes = 1, 2 ;
double time_counter(time_counter) ;
time_counter:axis = "T" ;
time_counter:standard_name = "time" ;
time_counter:long_name = "Time axis" ;
time_counter:title = "Time" ;
time_counter:calendar = "gregorian" ;
time_counter:units = "seconds since 1958-01-01 00:00:00" ;
time_counter:time_origin = "1958-01-01 00:00:00" ;
time_counter:bounds = "time_counter_bounds" ;
time_counter:_Storage = "chunked" ;
time_counter:_ChunkSizes = 1 ;
double time_counter_bounds(time_counter, time_bounds) ;
time_counter_bounds:_Storage = "chunked" ;
time_counter_bounds:_ChunkSizes = 1, 2 ;
float vosaline(time_counter, deptht, y, x) ;
vosaline:long_name = "salinity" ;
vosaline:units = "psu" ;
vosaline:online_operation = "average" ;
vosaline:interval_operation = "40s" ;
vosaline:interval_write = "5d" ;
vosaline:_FillValue = 0.f ;
vosaline:missing_value = 0.f ;
vosaline:coordinates = "time_centered deptht nav_lon nav_lat" ;
vosaline:_Storage = "chunked" ;
vosaline:_ChunkSizes = 1, 1, 12, 5422 ;
vosaline:_DeflateLevel = 1 ;
float sossheig(time_counter, y, x) ;
sossheig:long_name = "sea surface height" ;
sossheig:units = "m" ;
sossheig:online_operation = "average" ;
sossheig:interval_operation = "40s" ;
sossheig:interval_write = "5d" ;
sossheig:_FillValue = 0.f ;
sossheig:missing_value = 0.f ;
sossheig:coordinates = "time_centered nav_lon nav_lat" ;
sossheig:_Storage = "chunked" ;
sossheig:_ChunkSizes = 1, 12, 5422 ;
// global attributes:
:description = "ocean T grid variables" ;
:conventions = "CF-1.1" ;
:production = "An IPSL model" ;
:start_date = 20040101 ;
:output_frequency = "5d" ;
:CONFIG = "NATL60" ;
:CASE = "MJM155" ;
:_Format = "netCDF-4" ;
}
@apatlpo Along what axis do your multiple files differ?
Along time_counter
In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity If this issue remains relevant, please comment here; otherwise it will be marked as closed automatically
Hi all,
We are trying to specify chunks after opening an mfdataset but it does not work. This works fine with datasets. Is this behavior expected? Are we doing anything wrong?
The output is:
The code hangs for a while and then spits:
Cheers
Aurelien