search

eguil / Density_bining

Density bining code
2 stars 5 forks source link

Interpolate onto WOA grid and make zonal mean per basin #15

Closed eguil closed 10 years ago

eguil commented 10 years ago

Yep that error to me looks like it’s not happy with the grid..

I selected a 1x1 degree grid, the WOA one as I figured it’s the most readily used. The ESMF regridder that you’re already using will do the horizontal interpolation, and then this can be interpolated to standard levels. I’ve used the linearInterpolation function within cdutil to do that in the past and it works on a 3D matrix..

P

From: Eric Guilyardi Eric.Guilyardi@locean-ipsl.upmc.fr Organization: LOCEAN/IPSL and NCAS-Climate Date: Monday, June 9, 2014 at 4:45 PM To: "Paul J. Durack" durack1@llnl.gov Cc: "Doutriaux, Charles" doutriaux1@llnl.gov, "Gleckler, Peter J." gleckler1@llnl.gov Subject: Re: Zonal mean on ocean grid - can;t find Regridder

Paul, all,

ok, here is what I get:

/usr/local/uvcdat/latest/lib/python2.7/site-packages/regrid2/horizontal.py:293: Warning: While this will work for now, please note that the Regridder class has been renamed Horizontal, the name 'Regridder' will be deprecated in future version. Please edit your code accordingly warnings.warn("While this will work for now, please note that the Regridder class has been renamed Horizontal, the name 'Regridder' will be deprecated in future version. Please edit your code accordingly",Warning) Traceback (most recent call last): File "densit_bin.py", line 226, in regridF = cdm.mvCdmsRegrid.regrid2.Regridder(ingrid,outgrid) File "/usr/local/uvcdat/latest/lib/python2.7/site-packages/regrid2/horizontal.py", line 294, in init Horizontal.init(self,ingrid,outgrid) File "/usr/local/uvcdat/latest/lib/python2.7/site-packages/regrid2/horizontal.py", line 52, in init self.inorder = ingrid.getOrder() AttributeError: 'TransientCurveGrid' object has no attribute 'getOrder'

How to do you interpolate 3D ocean data onto a common grid ? And which grid did you choose ? Thanks for your help.

Cheers, Eric

durack1 commented 10 years ago

Did you need me to do something here - is the new mask data causing grief?

eguil commented 10 years ago

Hi Paul, I put this in to remember to try to optimize the code (annual mean, regrid and zonal mean). I just ran GFDL-CM2p1 180 years and it took 150 000 sec ! The regrid is the most expensive part. If we (Charles?) could find a way to store the weights instead of recomputing them each time, it would save a lot of CPU. Cheers, Eric

On 9/8/14 00:14, Paul J. Durack wrote:

Did you need me to do something here - is the new mask data causing grief?

— Reply to this email directly or view it on GitHub https://github.com/eguil/Density_bining/issues/15#issuecomment-51663883.

Eric Guilyardi IPSL/LOCEAN - Dir. Rech. CNRS Tour 45, 4eme, piece 406 UPMC, case 100 4 place Jussieu, F-75252 Paris Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi NCAS Climate Meteorology Department University of Reading Reading RG6 6BB - UK Tel: +44 (0)118 378 8315

             http://ncas-climate.nerc.ac.uk/~ericg

Currently visiting Lawrence Berkeley and Livermore National Laboratories, California

durack1 commented 10 years ago

I'll have a chat to Charles on Tuesday about this.. I was under the impression that you can provide weights to the function call, thereby removing another task it recomputes each time..

eguil commented 10 years ago

ok great. It would help a lot ! On 10/8/14 23:30, Paul J. Durack wrote:

I'll have a chat to Charles on Tuesday about this.. I was under the impression that you can provide weights to the function call, thereby removing another task it recomputes each time..

— Reply to this email directly or view it on GitHub https://github.com/eguil/Density_bining/issues/15#issuecomment-51727996.

Eric Guilyardi IPSL/LOCEAN - Dir. Rech. CNRS Tour 45, 4eme, piece 406 UPMC, case 100 4 place Jussieu, F-75252 Paris Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi NCAS Climate Meteorology Department University of Reading Reading RG6 6BB - UK Tel: +44 (0)118 378 8315

             http://ncas-climate.nerc.ac.uk/~ericg

Currently visiting Lawrence Berkeley and Livermore National Laboratories, California

durack1 commented 10 years ago

Ok so here we go, some tips: https://ice.txcorp.com/trac/modave/wiki/esmfConservative

In theory if we create an empty dictionary variable called diag and then call the regridder on one timestep, the diag dictionary should be filled with a bunch of variables (srcAreas, srcAreaFractions etc) which we can then provide to the regridder for the remaining timesteps.. I am yet to test this, and will have to do so in the coming days..

Some other useful links: https://ice.txcorp.com/trac/modave/wiki/regrid http://www.earthsystemmodeling.org/esmf_releases/last/regridding_status.html

eguil commented 10 years ago

Ok. Let me know how it goes. I've started coding the 'persistence' we discussed wrt the bowl and first tests look good.

Eric

On 12 août 2014, at 20:51, "Paul J. Durack" notifications@github.com wrote:

Ok so here we go, some tips: https://ice.txcorp.com/trac/modave/wiki/esmfConservative

In theory if we create an empty dictionary variable called diag and then call the regridder on one timestep, the diag dictionary should be filled with a bunch of variables (srcAreas, srcAreaFractions etc) which we can then provide to the regridder for the remaining timesteps.. I am yet to test this, and will have to do so in the coming days..

Some other useful links: https://ice.txcorp.com/trac/modave/wiki/regrid http://www.earthsystemmodeling.org/esmf_releases/last/regridding_status.html

— Reply to this email directly or view it on GitHub.

eguil commented 10 years ago

OK, I had a go at cdmsRegrid and it works ! The trick (which took some time to figure out...) was to transform the nan values it creates into valmask. CPU is now down by 90% !

Interpolation init (regrid)

ESMP.ESMP_Initialize() regridObj = CdmsRegrid(ingrid, outgrid, depth_bin.dtype, missing = valmask, regridMethod = 'linear', regridTool = 'esmf')

Next on the list: persistence per basin and total persistent volume per year.

durack1 commented 10 years ago

Wow, ok this sounds like you're cooking with gas! And yeah the NaN values in numpy vs what cdms2 is expecting is a bit of a quirk with cdms2.. It would be nice if it just dealt with NaNs natively..

eguil commented 10 years ago

now we are back to the annual mean using cdu.YEAR costing 2/3rd of the code... Something more to change (I'll continue investigating) I am currently running without the output of the monthly bin on native grid (huge file for 150 yrs...) and exploring the zonal mean and the persistence (great to see where the ocean is ventilated). Besides historical of IPSL-CM5A-LR and GFDL-CM2p1, any other models I should look at ? (I tried CCSM4 but there is a bug...) Btw, why does GFDL-CM2p1 has 180 years ? I was also thinking of running on a 500 yrs picontrol to get a feel for the variability (this will help D&A for historical runs).

On 18/8/14 18:06, Paul J. Durack wrote:

Wow, ok this sounds like you're cooking with gas! And yeah the NaN values in numpy vs what cdms2 is expecting is a bit of a quirk with cdms2.. It would be nice if it just dealt with NaNs natively..

— Reply to this email directly or view it on GitHub https://github.com/eguil/Density_bining/issues/15#issuecomment-52514124.

Eric Guilyardi IPSL/LOCEAN - Dir. Rech. CNRS Tour 45, 4eme, piece 406 UPMC, case 100 4 place Jussieu, F-75252 Paris Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi NCAS Climate Meteorology Department University of Reading Reading RG6 6BB - UK Tel: +44 (0)118 378 8315

             http://ncas-climate.nerc.ac.uk/~ericg

Currently visiting Lawrence Berkeley and Livermore National Laboratories, California

durack1 commented 10 years ago

It'd be interesting to compare the data computed using the monthly means vs that computed from the annual mean input data - if these are providing the same information, then why should we duplicate the annual mean step?

It seems to me that we're ready to run this thing on the whole suite.. And then discover what falls over.. Have you committed all your changes to your master branch?

eguil commented 10 years ago

ok, I've also managed to avoid cdu.YEAR by a combination of cdu.averager and cdm.createVariable (to add the grid that was missing). I am now testing the computation of the shallowest persistent depth/temp/salt and their zonal means per basin and we should be good for integrating in your wrapper. One last thing I have not done yet is to use the areas you computed for the WOA grid - I am still using the online def I made. More soon, On 21/8/14 23:38, Paul J. Durack wrote:

It'd be interesting to compare the data computed using the monthly means vs that computed from the annual mean input data - if these are providing the same information, then why should we duplicate the annual mean step?

It seems to me that we're ready to run this thing on the whole suite.. And then discover what falls over.. Have you committed all your changes to your master branch?

— Reply to this email directly or view it on GitHub https://github.com/eguil/Density_bining/issues/15#issuecomment-52988577.

Eric Guilyardi IPSL/LOCEAN - Dir. Rech. CNRS Tour 45, 4eme, piece 406 UPMC, case 100 4 place Jussieu, F-75252 Paris Tel: +33 (0)1 44 27 70 76 Prof. Eric Guilyardi NCAS Climate Meteorology Department University of Reading Reading RG6 6BB - UK Tel: +44 (0)118 378 8315

             http://ncas-climate.nerc.ac.uk/~ericg

Currently visiting Lawrence Berkeley and Livermore National Laboratories, California

eguil commented 10 years ago

Code more or less ready. Still could do with some optimization of loops but ready to roll !