search

barronh / pseudonetcdf

PseudoNetCDF like NetCDF except for many scientific format backends
GNU Lesser General Public License v3.0
77 stars 35 forks source link

Issue on conversion of ncfile to uamiv #60

Closed ArvindhSomanathan closed 6 years ago

ArvindhSomanathan commented 6 years ago

Hey there Barron!

I am trying to convert an emission inventory file (csv) to CAMx interpret-able format (uamiv). I have used Anaconda2 to execute the conversion in 2 separate environments; one on python 3.5. version and other on python 3.6. version. But, the execution was unsuccessful on both environments.

Source of error in the csv_to_uamiv_conversion.py script; @ line 175: 

173         uamivpath = "camx.ei.LOC.1km."+date1.strftime('%Y%m%d') + ".bin"
174         inncf = dt(ncffile)
175         ncf2uamiv(inncf, uamivpath)
176         date1 = date1 + day
177 #'''

Source of error in the ~\Write.py ; @ line 180:

180     spc_names = np.array(getattr(ncffile, 'VAR-LIST'), dtype = '>c').reshape(-1, 16)[:, :10].copy()

The error:

netCDF4_netCDF4.pyx in netCDF4._netCDF4.Dataset.getattr()

netCDF4_netCDF4.pyx in netCDF4._netCDF4.Dataset.getncattr()

netCDF4_netCDF4.pyx in netCDF4._netCDF4._get_att()

netCDF4_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

AttributeError: NetCDF: Attribute not found.

The problem seems to arise when I call ncfile as a dataset (from NetCDF): 

174         inncf = dt(ncffile)
175         ncf2uamiv(inncf, uamivpath)

Should I change my py script or could there be a modified write.py script?

Thanks in advance, we really appreciate your help and work on PseudonetCDF! Hope this issue helps solve similar issues for other CAMx-PseudonetCDF users as well!

barronh commented 6 years ago

It's hard to be certain from the information provided. My best guess is that you have not set a required property (VAR-LIST). This seems to suggest you've gotten past some of the other properties. For others who don't know what you've already done, I am going to explain in more detail.

This task can be conceptualized as four steps:

  1. Read in emission data
  2. Create a NetCDF-like file with correct meta-data
  3. Populate the data
  4. Save the data out.

It appears that the first step is not completed. The script below details those four steps in an idealized case where the CSV has cell and hour indices as columns.

Step 2 is definitely the hardest. In the below example, I've made some assumptions about the grid for the required properties. 

import PseudoNetCDF as pnc
import pandas as pd
import numpy as np

# Step 1
# assuming a five column csv
# t,i,j are 0-based indices
# t,i,j,varkey,value
# 0,0,0,NO,1.
# 1,1,1,NO,1.
# ...
# 23,4,3,NO,1.

inpath = 'somewhere.csv'
emisdata = pd.read_csv(inpath)

varkeys = emisdata.varkey.unique()

# Step 2
outpath = 'emis.lo'
outfile = pnc.PseudoNetCDFFile()
outfile.createDimension('TSTEP', 24)
outfile.createDimension('VAR', len(varkeys))
outfile.createDimension('DATE-TIME', 2)
outfile.createDimension('LAY', 1)
outfile.createDimension('ROW', 3)
outfile.createDimension('COL', 5)
properties = dict(
    NAME = 'EMISSIONS ',
    NOTE = 'SOMETHING DETAILS'.ljust(60),
    ITZON = 0,
    SDATE = 2018269,
    STIME = 0,
    TSTEP = 10000,
    IUTM = 0,
    CPROJ = 2, # aka, iproj
    GDTYP = 2, # aka, iproj
    PLON = -97,
    PLAT = 40.,
    TLAT1 = 33.,
    TLAT2 = 45.,
    ISTAG = 0,
    XORIG = -2736000.,
    YORIG = -2088000.,
    XCELL = 36000.,
    YCELL = 36000.,
)
properties['VAR-LIST'] = ''.join([varkey.ljust(16) for varkey in varkeys])
properties['P_ALP'] = properties['TLAT1']
properties['P_BET'] = properties['TLAT2']
properties['P_GAM'] = properties['PLON']
properties['XCENT'] = properties['PLON']
properties['YCENT'] = properties['PLAT']
for pk, pv in properties.items():
    setattr(outfile, pk, pv)

tflag = outfile.createVariable('TFLAG', 'i', ('TSTEP', 'VAR', 'DATE-TIME'))
tflag.units = '<YYYYJJJ,HHMMSS>'
tflag[:, :, 0] = outfile.SDATE
tflag[:, :, 1] = np.arange(24)[:, None]
etflag = outfile.createVariable('ETFLAG', 'i', ('TSTEP', 'VAR', 'DATE-TIME'))
etflag.units = '<YYYYJJJ,HHMMSS>'
etflag[:, :, 0] = tflag[:, :, 0]
etflag[-1, :, 0] += 1
etflag[:, :, 1] = tflag[:, :, 1] + 10000
etflag[-1, :, 1] = 0

# Step 3

# sum data in cell-hours
i, j = outfile.ll2ij(emisdata['lon'].values, emisdata['lat'].values)
emisdata['i'] = emisdata['lon'].astype('i') * 0 + i
emisdata['j'] = emisdata['lat'].astype('i') * 0 + j

bycell = emisdata.groupby(['t', 'i', 'j', 'varkey'], as_index=False).sum()

# For each emitted species, make a variable and populate it.
for varkey in varkeys:
    vardata = bycell.query('varkey == "{}"'.format(varkey))
    evar = outfile.createVariable(varkey, 'f', ('TSTEP', 'LAY', 'ROW', 'COL'))
    evar.units = 'moles/h'
    evar.long_name = varkey.ljust(16)
    evar.var_desc = varkey.ljust(80)
    evar[:] = 0.
    t = vardata['t'].values
    i = vardata['i'].values
    j = vardata['j'].values
    k = j * 0
    evar[t, k, j, i] = vardata['val'].values

# Step 4
outfile.save(outpath, format='uamiv')

The properties required are some basic IOAPI properties and some UAMIV properties. The meaning of the UAMIV fields are in the CAMx manual. The meaning of the IOAPI fields are in the IOAPI GRIDDESC documentation (https://www.cmascenter.org/ioapi/documentation/all_versions/html/GRIDDESC.html).

UAMIV

IOAPI