Open durack1 opened 6 years ago
If you use a FileVariable, it should get a faster response, for CDMS will make the call directly in 'C'
from: (line 91 or so..)
s=f.getslab(var,tim[b],tim[e])
to:
myvar=f[var] #FileVariable handle
s=myvar._obj_[:] #call C directly (return numpy array)
...
sorted=MV.numpy.sort(s,0) # call numpy directly (no grid)
Note that I got rid of your tim[b], tim[e] which are in "world" coordinates. here I select the whole array since you have 365 days in each file.
I was impressed by your use of "setdimattribute()" which does not seem to be documented.
"I was impressed by your use of "setdimattribute()" which does not seem to be documented."
This some ancient cdat code. Legacy stuff.
"Note that I got rid of your tim[b], tim[e] which are in "world" coordinates. here I select the whole array since you have 365 days in each file."
This may not be the best of examples as it also does seasonal max. So reading the whole array works for the annual, I don't think it works farther into the script for seasons. The reason it is not such a good example is that it fails for DJF in parallel since it does not have the December.
The way that I run this script serially is to create an xml file with cdscan. Then we just pop into the array and read whatever is needed. In parallel, this would be a disaster as every processor reads every netcdf file in the xml.
From: Michael Wehner
Date: Wednesday, December 20, 2017 at 4:35 PM
To: "Durack, Paul J."
Cc: "Doutriaux, Charles", Harinarayan Krishnan, "Nadeau, Denis"
Subject: Re: Slow "big data" reads in cdms2?
Paul
I also have these old examples. We would have to dig out the input files.
https://github.com/UV-CDAT/uvcdat/wiki/Embarrassingly-Parallel-Examples-Run-In-Serial
And this is dated, but reflects what I do still. sbatch instead of aprun, but otherwise the same.
https://github.com/UV-CDAT/uvcdat/wiki/How-to-run-UV-CDAT-in-parallel-at-NERSC
@mfwehner On 16 cores, you program took 16second
to run
and my change took 6s
to run.
You can find it in attachment.
make_extrema_longrun_parallel_denis.py.txt
I also used var[b:e+1,:]
since you had already computed it fortim[b]
and tim[e]
I think the point here is that the whole getslab needs to be rethought of. It seems to be slow (probably open/close file too many times). Also it can't handle modern indexing used by numpy (c = a[b>0]
)
@doutriaux1 @dnadeau4 on the "rethought" comment, these are also relevant UV-CDAT/uvcdat#1288, UV-CDAT/cdms#55, UV-CDAT/cdms#163
@doutriaux1 Your thought on this is particularly relevant if we are opening up a xml file made by cdscan, as all files are being opened by all processors.
Slow reads using
mpi4py
have been reported for use on the NERSC edison machine, full details are belowpinging @dnadeau4 @HarinarayanKrishnan @doutriaux1 And the offending script: make_extrema_longrun_parallel.py.txt