search

eguil / Density_bining

Density bining code
2 stars 5 forks source link

Convert velocity input to optional (obs have no velocity input) #74

Closed durack1 closed 5 years ago

durack1 commented 5 years ago

EN4 and other obs datasets have no corresponding velocity time series. Consequently current code won’t process obs.

eguil commented 5 years ago

Option added and input arguments slightly revisited:

densityBin(fileT,fileS,fileFx,fileV='none',outFile='out.nc',debug=True,timeint='all',\
mthout=False,gridfT='none',gridfS='none',gridfV='none'):

if fileV='none' the MSF calculation is ignored and V read is not required.

eguil commented 5 years ago

Tested and ok

durack1 commented 5 years ago

@eguil I'm still hitting problems with this:

(cdat80py2) duro@ocean:[Density_bining]:[issue71_durack1_UpdateObsUpdateESMPCalls*]:[11638]> python drive_obs.py
Processing:  0
areacello:  ~Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.so.ver-1.latestX.xml
so:         ~Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.thetao.ver-1.latestX.xml
thetao:     ~Shared/190213_data_density/obs.JAMSTEC.historical.r0i0p0.mo.ocn.Omon.so.ver-1.latestX.xml
Warning: ID reference not found: bounds=bounds_latitude
Warning: ID reference not found: bounds=bounds_longitude
Warning: ID reference not found: bounds=bounds_pressure
Warning: ID reference not found: bounds=bounds_time
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Traceback (most recent call last):
  File "drive_obs.py", line 87, in <module>
    densityBin(thetao,so,areacello,'none',outfileDensity,debug=True,timeint='all')
  File "~git/Density_bining/binDensity.py", line 358, in densityBin
    thetao_h    = ft('thetao', time = slice(1,10)) ; # remove handle for non cmor files
  File "~anaconda2/envs/cdat80py2/lib/python2.7/site-packages/cdms2/cudsinterface.py", line 33, in __call__
    raise CDMSError("No such variable, " + id)
cdms2.error.CDMSError: No such variable, thetao
eguil commented 5 years ago

@durack1 can you put the files somewhere on crunchy ? It is probably due to non CMOR format. Is the temperature variable name thetao ?

durack1 commented 5 years ago

@eguil it still seems to be an issue:

(cdat80py2) durack1@oceanonly:[Density_bining]:[issue71_durack1_UpdateObsUpdateESMPCalls*]:[11506]> python drive_obs.py
Processing:  0
areacello:  /work/durack1/Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.areacello.ver-1.latestX.nc
so:         /work/durack1/Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.so.ver-1.latestX.xml
thetao:     /work/durack1/Shared/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.thetao.ver-1.latestX.xml
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Warning: ID reference not found: bounds=depth_bnds
Warning: ID reference not found: bounds=bounds_lat
Warning: ID reference not found: bounds=bounds_lon
Warning: ID reference not found: bounds=time_bnds
Traceback (most recent call last):
  File "drive_obs.py", line 89, in <module>
    densityBin(thetao,so,areacello,'none',outfileDensity,debug=True,timeint='all')
  File "/export/durack1/git/Density_bining/binDensity.py", line 364, in densityBin
    vo_h        = fv('vo'    , time = slice(1,10)) ; #
  File "/export/durack1/anaconda2/envs/cdat80py2/lib/python2.7/site-packages/cdms2/cudsinterface.py", line 33, in __call__
    raise CDMSError("No such variable, " + id)
cdms2.error.CDMSError: No such variable, vo

I'll email the paths to data

eguil commented 5 years ago

is the call you make to density bin correct (arguments have been re-organised) ?

durack1 commented 5 years ago

@eguil looks right to me:

densityBin(thetao,so,areacello,'none',outfileDensity,debug=True,timeint='all')

https://github.com/eguil/Density_bining/blob/master/binDensity.py#L244-L266

def densityBin(fileT,fileS,fileFx,fileV='none',outFile='out.nc',debug=True,timeint='all',mthout=False,gridfT='none',gridfS='none',gridfV='none'):
    '''
    The densityBin() function takes file and variable arguments and creates
    density persistence fields which are written to a specified outfile
    Author:    Eric Guilyardi : Eric.Guilyardi@locean-ipsl.upmc.fr
    Co-author: Paul J. Durack : pauldurack@llnl.gov : @durack1.
    Created on Sun Sep 14 21:13:30 2014
    Inputs:
    ------
    - fileT(time,lev,lat,lon)   - 4D potential temperature array
    - fileS(time,lev,lat,lon)   - 4D salinity array
    - fileFx(lat,lon)           - 2D array containing the cell area values
    -> options:
    - fileV(time,lev,lat,lon)   - 4D meridional velocity array
    - outFile(str)              - output file with full path specified.
    - debug <optional>          - boolean value
    - timeint <optional>        - specify temporal step for binning <init_idx>,<ncount>
    - mthout <optional>         - write annual data (False) or all monthly data (True)
    - gridfT <optional>         - file to get T grid info from
    - gridfS <optional>         - file to get S grid info from
    - gridfV <optional>         - file to get V grid info from
...
eguil commented 5 years ago

ok I managed to run my latest version of binDensity (i.e. without your last changes) on the first year of EN4 and it is running. Here is my test script:

from binDensity import densityBin

obsThetao = '/data/ericglod/Density_binning/190213_data_density/180723_EN4.2.1.g10_monthly_TAndS_190001-190912_5-5350m.nc'
obsSo = '/data/ericglod/Density_binning/190213_data_density/180723_EN4.2.1.g10_monthly_TAndS_190001-190912_5-5350m.nc'
obsAreacello = '/data/ericglod/Density_binning/190213_data_density/obs.EN4.historical.r0i0p0.mo.ocn.Omon.areacello.ver-1.latestX.nc'

outfileDensity = '/home/ericglod/Density_bining/test/EN4.mon.ocean.Omon.density.nc'

densityBin(obsThetao,obsSo,obsAreacello,'none',outFile=outfileDensity,timeint='1,12')

Not sure what is going on on your side. Maybe we should do a call with screen sharing.

durack1 commented 5 years ago

@eguil what the the version of CDAT that you're running this against? A conda env?

eguil commented 5 years ago

an old PMP installation version

durack1 commented 5 years ago

Got any more details? If it's conda (probably not), you can peruse through /work/eguilyardi/anaconda2/envs/ - if it's CICLAD (looks like it), I'm not sure..

eguil commented 5 years ago

I am running on ciclad and it is not a conda install.

durack1 commented 5 years ago

Probably time to make the change #69, download anaconda

conda create -n cdat81py2 -c cdat/label/v81 -c conda-forge cdat "python<3"
ysilvy commented 5 years ago

Eric I don’t know if this can help but we can actually easily create conda environments on Ciclad, I’ve been testing it with simple stuff and installing new packages which seems to work, should we test that option ? That way we should be able to install the new version of cdat

Le 7 mars 2019 à 21:49, Paul J. Durack notifications@github.com a écrit :

Probably time to make the change, download anaconda https://www.anaconda.com/distribution/#download-section conda create -n cdat81py2 -c cdat/label/v81 -c conda-forge cdat "python<3" — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/eguil/Density_bining/issues/74#issuecomment-470688515, or mute the thread https://github.com/notifications/unsubscribe-auth/AWkPCkK_2hBNuoejAXVLNwkwE5rMCvImks5vUXtcgaJpZM4a9Dcp.

durack1 commented 5 years ago

@ysilvy it's a good move to update yourselves and the way you work. Creating conda environments means that you can have versions side by side, which is a real advantage, particularly as new/beta releases become available where you can validate older libraries/code works

eguil commented 5 years ago

Ok I managed to sync again binDensity on Crunchy. I will try there.

durack1 commented 5 years ago

@eguil let me know

eguil commented 5 years ago

Ok let's try that (I will transfer all the EN4 data). I can now sync git on crunchy but the python is now v3 and binDensity is v2. Where is the cdat that would work for python2 ?

eguil commented 5 years ago

the program is running on ciclad (old set up) but I have issues with the negative masking values in En4. Is it easy to change this in the files or should I work around in the code ?

durack1 commented 5 years ago

@eguil make sure you included the "python<3" from the https://github.com/eguil/Density_bining/issues/74#issuecomment-470688515, otherwise it will default to py3

eguil commented 5 years ago

I know this is confusing but my previous comment was for my tests on crunchy where I did not install CDAT.

durack1 commented 5 years ago

@eguil I'm having trouble following the thread of this discussion above, facetime/skype?

eguil commented 5 years ago

closing this