search

barronh / pseudonetcdf

PseudoNetCDF like NetCDF except for many scientific format backends
GNU Lesser General Public License v3.0
75 stars 35 forks source link

Error in conversion #20

Closed crchanda98 closed 7 years ago

crchanda98 commented 7 years ago

ncpath = 'test.uamiv.nc' uamivpath = 'test.uamiv.nc.uamiv' from netCDF4 import Dataset from PseudoNetCDF.camxfiles.uamiv.Write import ncf2uamiv inncf = Dataset(ncpath) ncf2uamiv(inncf, uamivpath)

while executing this code, the error is following:

[~/.../emission_data/meic/barronh_py]: python write_emission.py Traceback (most recent call last): File "write_emission.py", line 6, in ncf2uamiv(inncf, uamivpath) File "/home/arijit/miniconda2/lib/python2.7/site-packages/PseudoNetCDF/camxfiles/uamiv/Write.py", line 156, in ncf2uamiv time_hdr['iedate'] += time_hdr['etime'] // 24 TypeError: Cannot cast ufunc add output from dtype('float64') to dtype('>i4') with casting rule 'same_kind'

barronh commented 7 years ago

Edit the Write.py file

''' time_hdr['iedate'] += time_hdr['etime'] // 24 '''

To

''' time_hdr['iedate'] += (time_hdr['etime'] // 24).astype('>i') '''

Then try again. Let me know if it works and I'll update the code. There may be a comparable line for ibdate

crchanda98 commented 7 years ago

Hi Barron,

I was trying to compile with python 2.7. That was raising the version problem. I have switched to Python3.

Just wanted to know if there is EMISSION writer . I am looking for writing gridded emission with python.

On Fri, May 5, 2017 at 1:57 AM, barronh notifications@github.com wrote:

Edit the Write.py file

''' time_hdr['iedate'] += time_hdr['etime'] // 24 '''

To

''' time_hdr['iedate'] += (time_hdr['etime'] // 24).astype('>i') '''

Then try again. Let me know if it works and I'll update the code. There may be a comparable line for ibdate

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#issuecomment-299299499, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjrxJii3taftBrPeEsuS6afCahxtuks5r2jS6gaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

crchanda98 commented 7 years ago

Hi,

I could figure the emission writing through uamiv. There was a small bug in the writer.

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 5 10002 0. 10002 1. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -3406000.00000 25000.00000 25000.00000 130 140 1 2 0. 10002 0.00 10002 1.00 10002 1.00 10002 2.00 10002 2.00 10002 3.00 10002 3.00 10002 4.00 10002 4.00 10002 5.00 10002 5.00 10002 6.00 10002 6.00 10002 7.00 10002 7.00 10002 8.00 10002 8.00 10002 9.00 10002 9.00 10002 10.00 10002 10.00 10002 11.00 10002 11.00 10002 12.00 10002 12.00 10002 13.00 10002 13.00 10002 14.00 10002 14.00 10002 15.00 10002 15.00 10002 16.00 10002 16.00 10002 17.00 10002 17.00 10002 18.00 10002 18.00 10002 19.00 10002 19.00 10002 20.00 10002 20.00 10002 21.00 10002 21.00 10002 22.00 10002 22.00 10002 23.00 10002 23.00 10003 0.00

There is a difference in time header and simulation time split. Header is 1002 0 to 1002 1. But actually it is 1002 0 to 1002 24 or 1003 0. After modifying the writer, it is coming as:

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 5 10002 0. 10002 24. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -3406000.00000 25000.00000 25000.00000 130 140 1 2 0. 10002 0.00 10002 1.00 10002 1.00 10002 2.00 10002 2.00 10002 3.00 10002 3.00 10002 4.00 10002 4.00 10002 5.00 10002 5.00 10002 6.00 10002 6.00 10002 7.00 10002 7.00 10002 8.00 10002 8.00 10002 9.00 10002 9.00 10002 10.00 10002 10.00 10002 11.00 10002 11.00 10002 12.00 10002 12.00 10002 13.00 10002 13.00 10002 14.00 10002 14.00 10002 15.00 10002 15.00 10002 16.00 10002 16.00 10002 17.00 10002 17.00 10002 18.00 10002 18.00 10002 19.00 10002 19.00 10002 20.00 10002 20.00 10002 21.00 10002 21.00 10002 22.00 10002 22.00 10002 23.00 10002 23.00 10002 24.00

On Fri, May 5, 2017 at 10:01 AM, Arijit Chanda crchanda98@gmail.com wrote:

Hi Barron,

I was trying to compile with python 2.7. That was raising the version problem. I have switched to Python3.

Just wanted to know if there is EMISSION writer . I am looking for writing gridded emission with python.

On Fri, May 5, 2017 at 1:57 AM, barronh notifications@github.com wrote:

Edit the Write.py file

''' time_hdr['iedate'] += time_hdr['etime'] // 24 '''

To

''' time_hdr['iedate'] += (time_hdr['etime'] // 24).astype('>i') '''

Then try again. Let me know if it works and I'll update the code. There may be a comparable line for ibdate

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#issuecomment-299299499, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjrxJii3taftBrPeEsuS6afCahxtuks5r2jS6gaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

crchanda98 commented 7 years ago

Hi,

I could figure the emission writing through uamiv. There was a small bug in the writer.

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 5 10002 0. 10002 1. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -3406000.00000 25000.00000 25000.00000 130 140 1 2 0. 10002 0.00 10002 1.00 10002 1.00 10002 2.00 10002 2.00 10002 3.00 10002 3.00 10002 4.00 10002 4.00 10002 5.00 10002 5.00 10002 6.00 10002 6.00 10002 7.00 10002 7.00 10002 8.00 10002 8.00 10002 9.00 10002 9.00 10002 10.00 10002 10.00 10002 11.00 10002 11.00 10002 12.00 10002 12.00 10002 13.00 10002 13.00 10002 14.00 10002 14.00 10002 15.00 10002 15.00 10002 16.00 10002 16.00 10002 17.00 10002 17.00 10002 18.00 10002 18.00 10002 19.00 10002 19.00 10002 20.00 10002 20.00 10002 21.00 10002 21.00 10002 22.00 10002 22.00 10002 23.00 10002 23.00 10003 0.00

There is a difference in time header and simulation time split. Header is 1002 0 to 1002 1. But actually it is 1002 0 to 1002 24 or 1003 0. After modifying the writer, it is coming as:

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 5 10002 0. 10002 24. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -3406000.00000 25000.00000 25000.00000 130 140 1 2 0. 10002 0.00 10002 1.00 10002 1.00 10002 2.00 10002 2.00 10002 3.00 10002 3.00 10002 4.00 10002 4.00 10002 5.00 10002 5.00 10002 6.00 10002 6.00 10002 7.00 10002 7.00 10002 8.00 10002 8.00 10002 9.00 10002 9.00 10002 10.00 10002 10.00 10002 11.00 10002 11.00 10002 12.00 10002 12.00 10002 13.00 10002 13.00 10002 14.00 10002 14.00 10002 15.00 10002 15.00 10002 16.00 10002 16.00 10002 17.00 10002 17.00 10002 18.00 10002 18.00 10002 19.00 10002 19.00 10002 20.00 10002 20.00 10002 21.00 10002 21.00 10002 22.00 10002 22.00 10002 23.00 10002 23.00 10002 24.00

On Fri, May 5, 2017 at 10:01 AM, Arijit Chanda crchanda98@gmail.com wrote:

Hi Barron,

I was trying to compile with python 2.7. That was raising the version problem. I have switched to Python3.

Just wanted to know if there is EMISSION writer . I am looking for writing gridded emission with python.

On Fri, May 5, 2017 at 1:57 AM, barronh notifications@github.com wrote:

Edit the Write.py file

''' time_hdr['iedate'] += time_hdr['etime'] // 24 '''

To

''' time_hdr['iedate'] += (time_hdr['etime'] // 24).astype('>i') '''

Then try again. Let me know if it works and I'll update the code. There may be a comparable line for ibdate

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#issuecomment-299299499, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjrxJii3taftBrPeEsuS6afCahxtuks5r2jS6gaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

barronh commented 7 years ago

Your first point about the end time for the file header does need to be updated for sure.

The lack of a 24 hour time is a conscious choice. In many utilities, there is no such thing as 10002 24 because it is synonymous with 10003 0. 

>date -ud "2011-01-01 24:00:00" +%Y%m%d%H%M%S
date: invalid date ‘2011-01-01 24:00:00’

I believe that CAMx supports both in emissions files, so I have historical chosen to maintain the +1 day 0 o'clock representation. This is more compatible with all the other time functions that are used.

FYI - you can also make a copy of an existing file and simply update the emissions. That preserves the meta data precisely.

cp old.uamiv test.uamiv
python -ic "
from PseudoNetCDF import PNC
ncffile = PNC('test.nc').ifiles[0]
testfile = PNC('--format=uamiv,mode=r+', 'test.uamiv').ifiles[0]
for outk, outv in testfile.variables.items():
    inv = ncffile.variables[outk]
    outv[:] = inv[:]
testfile.close()
"
barronh commented 7 years ago

I propose

diff --git a/src/PseudoNetCDF/camxfiles/uamiv/Write.py b/src/PseudoNetCDF/camxfiles/uamiv/Write.py
index f000585..8e53406 100644
--- a/src/PseudoNetCDF/camxfiles/uamiv/Write.py
+++ b/src/PseudoNetCDF/camxfiles/uamiv/Write.py
@@ -148,18 +148,19 @@ def ncf2uamiv(ncffile, outpath):
     time_hdr['SPAD'] = 16
     time_hdr['EPAD'] = 16
     date, time = ncffile.variables['TFLAG'][:, 0].T
+    tincr = np.diff(time)[0]
     time = time.astype('>f') / 10000.
     date = date%(date//100000*100000)
     time_hdr['ibdate'] = date
     time_hdr['btime'] = time
     time_hdr['iedate'] = date
-    time_hdr['etime'] = time + 1.
+    time_hdr['etime'] = time + tincr
     time_hdr['iedate'] += (time_hdr['etime'] // 24).astype('i')
     time_hdr['etime'] -= (time_hdr['etime'] // 24) * 24
     emiss_hdr['ibdate'] = time_hdr['ibdate'][0]
     emiss_hdr['btime'] = time_hdr['btime'][0]
-    emiss_hdr['iedate'] = time_hdr['iedate'][0]
-    emiss_hdr['etime'] = time_hdr['etime'][0]
+    emiss_hdr['iedate'] = time_hdr['iedate'][-1]
+    emiss_hdr['etime'] = time_hdr['etime'][-1]
     emiss_hdr['SPAD'] = _emiss_hdr_fmt.itemsize - 8
     emiss_hdr['EPAD'] = _emiss_hdr_fmt.itemsize - 8

Does that solve the etime problem in the file header?

crchanda98 commented 7 years ago

Hi,

Sorry for late reply. I was in weekend vacation.

Your first modification is ending with some error:

(snakes) [~/.../emission_data/meic/SPARC99_2010]: python make_barronh_input.py Traceback (most recent call last): File "make_barronh_input.py", line 202, in ncf2uamiv(inncf, uamivpath) File "/home/arijit/miniconda2/envs/snakes/lib/python3.6/site-packages/PseudoNetCDF-3.0-py3.6.egg/PseudoNetCDF/camxfiles/uamiv/Write.py", line 158, in ncf2uamiv time_hdr['iedate'] += time_hdr['etime'] // 24 TypeError: Cannot cast ufunc add output from dtype('float64') to dtype('>i4') with casting rule 'same_kind'

Your second modification is preparing the following output:

python make_barronh_input.py Convert UAM-IV format CAMx files from binary to ascii The following file types are recognized: AVERAGE AIRQUALITY BOUNDARY EMISSIONS PTSOURCE TEMPERATUR WIND

The file type will be determined automatically

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10419 15. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0

    1. 0. 10002 0.00 10418 16.00 10002 1.00 10418 17.00 10002 2.00 10418 18.00 10002 3.00 10418 19.00 10002 4.00 10418 20.00 10002 5.00 10418 21.00 10002 6.00 10418 22.00 10002 7.00 10418 23.00 10002 8.00 10419 0.00 10002 9.00 10419 1.00 10002 10.00 10419 2.00 10002 11.00 10419 3.00 10002 12.00 10419 4.00 10002 13.00 10419 5.00 10002 14.00 10419 6.00 10002 15.00 10419 7.00 10002 16.00 10419 8.00 10002 17.00 10419 9.00 10002 18.00 10419 10.00 10002 19.00 10419 11.00 10002 20.00 10419 12.00 10002 21.00 10419 13.00 10002 22.00 10419 14.00 10002 23.00 10419 15.00

I have tried to use ETFLAG which enable us avoiding time calculation in Write.py code. The ETFLAG in nc file is following:[[[ 2010003. 10000.]] [[ 2010003. 20000.]] [[ 2010003. 30000.]] [[ 2010003. 40000.]] [[ 2010003. 50000.]] [[ 2010003. 60000.]] [[ 2010003. 70000.]] [[ 2010003. 80000.]] [[ 2010003. 90000.]] [[ 2010003. 100000.]] [[ 2010003. 110000.]] [[ 2010003. 120000.]] [[ 2010003. 130000.]] [[ 2010003. 140000.]] [[ 2010003. 150000.]] [[ 2010003. 160000.]] [[ 2010003. 170000.]] [[ 2010003. 180000.]] [[ 2010003. 190000.]] [[ 2010003. 200000.]] [[ 2010003. 210000.]] [[ 2010003. 220000.]] [[ 2010003. 230000.]] [[ 2010003. 0.]]]

The result is following:

(snakes) [~/.../emission_data/meic/SPARC99_2010]: python make_barronh_input.py Convert UAM-IV format CAMx files from binary to ascii The following file types are recognized: AVERAGE AIRQUALITY BOUNDARY EMISSIONS PTSOURCE TEMPERATUR WIND

The file type will be determined automatically

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10003 0. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0

    1. 0. 10002 0.00 10003 1.00 10002 1.00 10003 2.00 10002 2.00 10003 3.00 10002 3.00 10003 4.00 10002 4.00 10003 5.00 10002 5.00 10003 6.00 10002 6.00 10003 7.00 10002 7.00 10003 8.00 10002 8.00 10003 9.00 10002 9.00 10003 10.00 10002 10.00 10003 11.00 10002 11.00 10003 12.00 10002 12.00 10003 13.00 10002 13.00 10003 14.00 10002 14.00 10003 15.00 10002 15.00 10003 16.00 10002 16.00 10003 17.00 10002 17.00 10003 18.00 10002 18.00 10003 19.00 10002 19.00 10003 20.00 10002 20.00 10003 21.00 10002 21.00 10003 22.00 10002 22.00 10003 23.00 10002 23.00 10003 0.00

I hope this solves the concern of 10002 240000 and makes it 10003 000000. The code is following:

date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
time_s = time.astype('>f') / 10000.
date_s = date%(date//100000*100000)
time_e = time_e.astype('>f') / 10000.
date_e = date_e%(date_e//100000*100000)
time_hdr['ibdate'] = date_s
time_hdr['btime'] = time_s
time_hdr['iedate'] = date_e
time_hdr['etime'] = time_e
emiss_hdr['ibdate'] = date_s[0]
emiss_hdr['btime'] = time_s[0]
emiss_hdr['iedate'] = date_e[-1]
emiss_hdr['etime'] = time_e[-1]
emiss_hdr['SPAD'] = _emiss_hdr_fmt.itemsize - 8
emiss_hdr['EPAD'] = _emiss_hdr_fmt.itemsize - 8

On Fri, May 5, 2017 at 8:25 PM, barronh notifications@github.com wrote:

I propose

diff --git a/src/PseudoNetCDF/camxfiles/uamiv/Write.py b/src/PseudoNetCDF/camxfiles/uamiv/Write.py index f000585..8e53406 100644 --- a/src/PseudoNetCDF/camxfiles/uamiv/Write.py +++ b/src/PseudoNetCDF/camxfiles/uamiv/Write.py @@ -148,18 +148,19 @@ def ncf2uamiv(ncffile, outpath): time_hdr['SPAD'] = 16 time_hdr['EPAD'] = 16 date, time = ncffile.variables['TFLAG'][:, 0].T

  • tincr = np.diff(time)[0] time = time.astype('>f') / 10000. date = date%(date//100000*100000) time_hdr['ibdate'] = date time_hdr['btime'] = time time_hdr['iedate'] = date
  • time_hdr['etime'] = time + 1.
  • time_hdr['etime'] = time + tincr time_hdr['iedate'] += (time_hdr['etime'] // 24).astype('i') time_hdr['etime'] -= (time_hdr['etime'] // 24) * 24 emiss_hdr['ibdate'] = time_hdr['ibdate'][0] emiss_hdr['btime'] = time_hdr['btime'][0]
  • emiss_hdr['iedate'] = time_hdr['iedate'][0]
  • emiss_hdr['etime'] = time_hdr['etime'][0]
  • emiss_hdr['iedate'] = time_hdr['iedate'][-1]
  • emiss_hdr['etime'] = time_hdr['etime'][-1] emiss_hdr['SPAD'] = _emiss_hdr_fmt.itemsize - 8 emiss_hdr['EPAD'] = _emiss_hdr_fmt.itemsize - 8

Does that solve the etime problem in the file header?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#issuecomment-299487258, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjsL7EZuk6mUtnhxDExmHyo4CU-gkks5r2zhtgaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

crchanda98 commented 7 years ago

Hi,

Please ignore the previous mail.

Sorry for late reply. I was in weekend vacation.

Your first modification is ending with some error:

(snakes) [~/.../emission_data/meic/SPARC99_2010]: python make_barronh_input.py Traceback (most recent call last): File "make_barronh_input.py", line 202, in ncf2uamiv(inncf, uamivpath) File "/home/arijit/miniconda2/envs/snakes/lib/python3.6/site- packages/PseudoNetCDF-3.0-py3.6.egg/PseudoNetCDF/camxfiles/uamiv/Write.py", line 158, in ncf2uamiv time_hdr['iedate'] += time_hdr['etime'] // 24 TypeError: Cannot cast ufunc add output from dtype('float64') to dtype('>i4') with casting rule 'same_kind'

Your second modification is preparing the following output:

python make_barronh_input.py Convert UAM-IV format CAMx files from binary to ascii The following file types are recognized: AVERAGE AIRQUALITY BOUNDARY EMISSIONS PTSOURCE TEMPERATUR WIND

The file type will be determined automatically

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10419 15. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0

    1. 0. 10002 0.00 10418 16.00 10002 1.00 10418 17.00 10002 2.00 10418 18.00 10002 3.00 10418 19.00 10002 4.00 10418 20.00 10002 5.00 10418 21.00 10002 6.00 10418 22.00 10002 7.00 10418 23.00 10002 8.00 10419 0.00 10002 9.00 10419 1.00 10002 10.00 10419 2.00 10002 11.00 10419 3.00 10002 12.00 10419 4.00 10002 13.00 10419 5.00 10002 14.00 10419 6.00 10002 15.00 10419 7.00 10002 16.00 10419 8.00 10002 17.00 10419 9.00 10002 18.00 10419 10.00 10002 19.00 10419 11.00 10002 20.00 10419 12.00 10002 21.00 10419 13.00 10002 22.00 10419 14.00 10002 23.00 10419 15.00

I have tried to use ETFLAG which enable us avoiding time calculation in Write.py code. The ETFLAG in nc file is following:[[[ 2010002. 10000.]] [[ 2010002. 20000.]] [[ 2010002. 30000.]] [[ 2010002. 40000.]] [[ 2010002. 50000.]] [[ 2010002. 60000.]] [[ 2010002. 70000.]] [[ 2010002. 80000.]] [[ 2010002. 90000.]] [[ 2010002. 100000.]] [[ 2010002. 110000.]] [[ 2010002. 120000.]] [[ 2010002. 130000.]] [[ 2010002. 140000.]] [[ 2010002. 150000.]] [[ 2010002. 160000.]] [[ 2010002. 170000.]] [[ 2010002. 180000.]] [[ 2010002. 190000.]] [[ 2010002. 200000.]] [[ 2010002. 210000.]] [[ 2010002. 220000.]] [[ 2010002. 230000.]] [[ 2010003. 0.]]] The result is following:

Convert UAM-IV format CAMx files from binary to ascii The following file types are recognized: AVERAGE AIRQUALITY BOUNDARY EMISSIONS PTSOURCE TEMPERATUR WIND

The file type will be determined automatically

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10003 0. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0

    1. 0. 10002 0.00 10002 1.00 10002 1.00 10002 2.00 10002 2.00 10002 3.00 10002 3.00 10002 4.00 10002 4.00 10002 5.00 10002 5.00 10002 6.00 10002 6.00 10002 7.00 10002 7.00 10002 8.00 10002 8.00 10002 9.00 10002 9.00 10002 10.00 10002 10.00 10002 11.00 10002 11.00 10002 12.00 10002 12.00 10002 13.00 10002 13.00 10002 14.00 10002 14.00 10002 15.00 10002 15.00 10002 16.00 10002 16.00 10002 17.00 10002 17.00 10002 18.00 10002 18.00 10002 19.00 10002 19.00 10002 20.00 10002 20.00 10002 21.00 10002 21.00 10002 22.00 10002 22.00 10002 23.00 10002 23.00 10003 0.00

I hope this solves the concern of 10002 240000 and makes it 10003 000000. The code is following:

date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
time_s = time.astype('>f') / 10000.
date_s = date%(date//100000*100000)
time_e = time_e.astype('>f') / 10000.
date_e = date_e%(date_e//100000*100000)
time_hdr['ibdate'] = date_s
time_hdr['btime'] = time_s
time_hdr['iedate'] = date_e
time_hdr['etime'] = time_e
emiss_hdr['ibdate'] = date_s[0]
emiss_hdr['btime'] = time_s[0]
emiss_hdr['iedate'] = date_e[-1]
emiss_hdr['etime'] = time_e[-1]

On Mon, May 8, 2017 at 2:57 PM, Arijit Chanda crchanda98@gmail.com wrote:

Hi,

Sorry for late reply. I was in weekend vacation.

Your first modification is ending with some error:

(snakes) [~/.../emission_data/meic/SPARC99_2010]: python make_barronh_input.py Traceback (most recent call last): File "make_barronh_input.py", line 202, in ncf2uamiv(inncf, uamivpath) File "/home/arijit/miniconda2/envs/snakes/lib/python3.6/site- packages/PseudoNetCDF-3.0-py3.6.egg/PseudoNetCDF/camxfiles/uamiv/Write.py", line 158, in ncf2uamiv time_hdr['iedate'] += time_hdr['etime'] // 24 TypeError: Cannot cast ufunc add output from dtype('float64') to dtype('>i4') with casting rule 'same_kind'

Your second modification is preparing the following output:

python make_barronh_input.py Convert UAM-IV format CAMx files from binary to ascii The following file types are recognized: AVERAGE AIRQUALITY BOUNDARY EMISSIONS PTSOURCE TEMPERATUR WIND

The file type will be determined automatically

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10419 15. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0

    1. 0. 10002 0.00 10418 16.00 10002 1.00 10418 17.00 10002 2.00 10418 18.00 10002 3.00 10418 19.00 10002 4.00 10418 20.00 10002 5.00 10418 21.00 10002 6.00 10418 22.00 10002 7.00 10418 23.00 10002 8.00 10419 0.00 10002 9.00 10419 1.00 10002 10.00 10419 2.00 10002 11.00 10419 3.00 10002 12.00 10419 4.00 10002 13.00 10419 5.00 10002 14.00 10419 6.00 10002 15.00 10419 7.00 10002 16.00 10419 8.00 10002 17.00 10419 9.00 10002 18.00 10419 10.00 10002 19.00 10419 11.00 10002 20.00 10419 12.00 10002 21.00 10419 13.00 10002 22.00 10419 14.00 10002 23.00 10419 15.00

I have tried to use ETFLAG which enable us avoiding time calculation in Write.py code. The ETFLAG in nc file is following:[[[ 2010003. 10000.]] [[ 2010003. 20000.]] [[ 2010003. 30000.]] [[ 2010003. 40000.]] [[ 2010003. 50000.]] [[ 2010003. 60000.]] [[ 2010003. 70000.]] [[ 2010003. 80000.]] [[ 2010003. 90000.]] [[ 2010003. 100000.]] [[ 2010003. 110000.]] [[ 2010003. 120000.]] [[ 2010003. 130000.]] [[ 2010003. 140000.]] [[ 2010003. 150000.]] [[ 2010003. 160000.]] [[ 2010003. 170000.]] [[ 2010003. 180000.]] [[ 2010003. 190000.]] [[ 2010003. 200000.]] [[ 2010003. 210000.]] [[ 2010003. 220000.]] [[ 2010003. 230000.]] [[ 2010003. 0.]]]

The result is following:

(snakes) [~/.../emission_data/meic/SPARC99_2010]: python make_barronh_input.py Convert UAM-IV format CAMx files from binary to ascii The following file types are recognized: AVERAGE AIRQUALITY BOUNDARY EMISSIONS PTSOURCE TEMPERATUR WIND

The file type will be determined automatically

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10003 0. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0

    1. 0. 10002 0.00 10003 1.00 10002 1.00 10003 2.00 10002 2.00 10003 3.00 10002 3.00 10003 4.00 10002 4.00 10003 5.00 10002 5.00 10003 6.00 10002 6.00 10003 7.00 10002 7.00 10003 8.00 10002 8.00 10003 9.00 10002 9.00 10003 10.00 10002 10.00 10003 11.00 10002 11.00 10003 12.00 10002 12.00 10003 13.00 10002 13.00 10003 14.00 10002 14.00 10003 15.00 10002 15.00 10003 16.00 10002 16.00 10003 17.00 10002 17.00 10003 18.00 10002 18.00 10003 19.00 10002 19.00 10003 20.00 10002 20.00 10003 21.00 10002 21.00 10003 22.00 10002 22.00 10003 23.00 10002 23.00 10003 0.00

I hope this solves the concern of 10002 240000 and makes it 10003 000000. The code is following:

date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
time_s = time.astype('>f') / 10000.
date_s = date%(date//100000*100000)
time_e = time_e.astype('>f') / 10000.
date_e = date_e%(date_e//100000*100000)
time_hdr['ibdate'] = date_s
time_hdr['btime'] = time_s
time_hdr['iedate'] = date_e
time_hdr['etime'] = time_e
emiss_hdr['ibdate'] = date_s[0]
emiss_hdr['btime'] = time_s[0]
emiss_hdr['iedate'] = date_e[-1]
emiss_hdr['etime'] = time_e[-1]
emiss_hdr['SPAD'] = _emiss_hdr_fmt.itemsize - 8
emiss_hdr['EPAD'] = _emiss_hdr_fmt.itemsize - 8

On Fri, May 5, 2017 at 8:25 PM, barronh notifications@github.com wrote:

I propose

diff --git a/src/PseudoNetCDF/camxfiles/uamiv/Write.py b/src/PseudoNetCDF/camxfiles/uamiv/Write.py index f000585..8e53406 100644 --- a/src/PseudoNetCDF/camxfiles/uamiv/Write.py +++ b/src/PseudoNetCDF/camxfiles/uamiv/Write.py @@ -148,18 +148,19 @@ def ncf2uamiv(ncffile, outpath): time_hdr['SPAD'] = 16 time_hdr['EPAD'] = 16 date, time = ncffile.variables['TFLAG'][:, 0].T

  • tincr = np.diff(time)[0] time = time.astype('>f') / 10000. date = date%(date//100000*100000) time_hdr['ibdate'] = date time_hdr['btime'] = time time_hdr['iedate'] = date
  • time_hdr['etime'] = time + 1.
  • time_hdr['etime'] = time + tincr time_hdr['iedate'] += (time_hdr['etime'] // 24).astype('i') time_hdr['etime'] -= (time_hdr['etime'] // 24) * 24 emiss_hdr['ibdate'] = time_hdr['ibdate'][0] emiss_hdr['btime'] = time_hdr['btime'][0]
  • emiss_hdr['iedate'] = time_hdr['iedate'][0]
  • emiss_hdr['etime'] = time_hdr['etime'][0]
  • emiss_hdr['iedate'] = time_hdr['iedate'][-1]
  • emiss_hdr['etime'] = time_hdr['etime'][-1] emiss_hdr['SPAD'] = _emiss_hdr_fmt.itemsize - 8 emiss_hdr['EPAD'] = _emiss_hdr_fmt.itemsize - 8

Does that solve the etime problem in the file header?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#issuecomment-299487258, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjsL7EZuk6mUtnhxDExmHyo4CU-gkks5r2zhtgaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

barronh commented 7 years ago

Thanks for the proposed update. The ETFLAG is not always available in the IOAPI format, which this writer relies on. I took your proposed change and tried to make it a little more flexible with respect to the ETFLAG variable. I confirmed that it works with a 1-length time that is missing the ETFLAG variable. If you can confirm it works, I'll push it to the code base.

+    date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
+    time_s = time_s.astype('>f') / 10000.
+    date_s = date_s%(date_s//100000*100000)
+    if 'ETFLAG' in ncffile.variables.keys():
+        date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
+        time_e = time_e.astype('>f') / 10000.
+        date_e = date_e%(date_e//100000*100000)
+    else:
+        if hasattr(ncffile, 'TSTEP')
+            tincr = ncffile.TSTEP / 10000
+        else:
+            tincr = np.diff(time_s)[0]
+        date_e = date_s.copy()
+        time_e = time_s.copy() + tincr
+        date_e += (time_e // 24).astype('i')
+        time_e -= (time_e // 24) * 24
crchanda98 commented 7 years ago

A small modification is working fine. The intermediate else is erroneous. The result is following:

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10003 22. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0. 10002 0.00 10002 23.00 10002 1.00 10003 0.00 10002 2.00 10003 1.00 10002 3.00 10003 2.00 10002 4.00 10003 3.00 10002 5.00 10003 4.00 10002 6.00 10003 5.00 10002 7.00 10003 6.00 10002 8.00 10003 7.00 10002 9.00 10003 8.00 10002 10.00 10003 9.00 10002 11.00 10003 10.00 10002 12.00 10003 11.00 10002 13.00 10003 12.00 10002 14.00 10003 13.00 10002 15.00 10003 14.00 10002 16.00 10003 15.00 10002 17.00 10003 16.00 10002 18.00 10003 17.00 10002 19.00 10003 18.00 10002 20.00 10003 19.00 10002 21.00 10003 20.00 10002 22.00 10003 21.00 10002 23.00 10003 22.00

If we don't consider the intermediate else and calculate tincr from TFLAG alone, its perfect. the modification is following:

date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
time_s = time_s.astype('>f') / 10000.
date_s = date_s%(date_s//100000*100000)
if 'ETFLAG' in ncffile.variables.keys():
    date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
    time_e = time_e.astype('>f') / 10000.
    date_e = date_e%(date_e//100000*100000)
else:
    tincr = np.diff(time_s)[0]
    date_e = date_s.copy()
    time_e = time_s.copy() + tincr
    date_e += (time_e // 24).astype('i')
    time_e -= (time_e // 24) * 24

This works perfect without ETFLAG or TSTEP. This can be updated and this will work perfectly as the time difference is uniform.

On Mon, May 8, 2017 at 10:53 PM, barronh notifications@github.com wrote:

Thanks for the proposed update. The ETFLAG is not always available in the IOAPI format, which this writer relies on. I took your proposed change and tried to make it a little more flexible with respect to the ETFLAG variable. I confirmed that it works with a 1-length time that is missing the ETFLAG variable. If you can confirm it works, I'll push it to the code base.

  • date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
  • time_s = time_s.astype('>f') / 10000.
  • date_s = date_s%(date_s//100000*100000)
  • if 'ETFLAG' in ncffile.variables.keys():
  • date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
  • time_e = time_e.astype('>f') / 10000.
  • date_e = date_e%(date_e//100000*100000)
  • else:
  • if hasattr(ncffile, 'TSTEP')
  • tincr = ncffile.TSTEP / 10000
  • else:
  • tincr = np.diff(time_s)[0]
  • date_e = date_s.copy()
  • time_e = time_s.copy() + tincr
  • date_e += (time_e // 24).astype('i')
  • time_e -= (time_e // 24) * 24

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#issuecomment-299932370, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjj5AXTUMyW7_jNAh_zhr6A1rRU7Zks5r30-OgaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

crchanda98 commented 7 years ago

Hi,

Did that work?

On Tue, May 9, 2017 at 8:54 PM, Arijit Chanda crchanda98@gmail.com wrote:

A small modification is working fine. The intermediate else is erroneous. The result is following:

Input file name: Opened file: test.nc.uamiv

Output file name: Opened file: test.nc.uamiv.txt

EMISSIONS CAMx 5.40 Test Problem 1 1 10002 0. 10003 22. file type is EMISSIONS 0.00000 0.00000 0 -1625000.00000 -1750000.00000 25000.00000 25000.00000 130 140 1 2 0. 10002 0.00 10002 23.00 10002 1.00 10003 0.00 10002 2.00 10003 1.00 10002 3.00 10003 2.00 10002 4.00 10003 3.00 10002 5.00 10003 4.00 10002 6.00 10003 5.00 10002 7.00 10003 6.00 10002 8.00 10003 7.00 10002 9.00 10003 8.00 10002 10.00 10003 9.00 10002 11.00 10003 10.00 10002 12.00 10003 11.00 10002 13.00 10003 12.00 10002 14.00 10003 13.00 10002 15.00 10003 14.00 10002 16.00 10003 15.00 10002 17.00 10003 16.00 10002 18.00 10003 17.00 10002 19.00 10003 18.00 10002 20.00 10003 19.00 10002 21.00 10003 20.00 10002 22.00 10003 21.00 10002 23.00 10003 22.00

If we don't consider the intermediate else and calculate tincr from TFLAG alone, its perfect. the modification is following:

date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
time_s = time_s.astype('>f') / 10000.
date_s = date_s%(date_s//100000*100000)
if 'ETFLAG' in ncffile.variables.keys():
    date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
    time_e = time_e.astype('>f') / 10000.
    date_e = date_e%(date_e//100000*100000)
else:
    tincr = np.diff(time_s)[0]
    date_e = date_s.copy()
    time_e = time_s.copy() + tincr
    date_e += (time_e // 24).astype('i')
    time_e -= (time_e // 24) * 24

This works perfect without ETFLAG or TSTEP. This can be updated and this will work perfectly as the time difference is uniform.

On Mon, May 8, 2017 at 10:53 PM, barronh notifications@github.com wrote:

Thanks for the proposed update. The ETFLAG is not always available in the IOAPI format, which this writer relies on. I took your proposed change and tried to make it a little more flexible with respect to the ETFLAG variable. I confirmed that it works with a 1-length time that is missing the ETFLAG variable. If you can confirm it works, I'll push it to the code base.

  • date_s, time_s = ncffile.variables['TFLAG'][:, 0].T
  • time_s = time_s.astype('>f') / 10000.
  • date_s = date_s%(date_s//100000*100000)
  • if 'ETFLAG' in ncffile.variables.keys():
  • date_e, time_e = ncffile.variables['ETFLAG'][:, 0].T
  • time_e = time_e.astype('>f') / 10000.
  • date_e = date_e%(date_e//100000*100000)
  • else:
  • if hasattr(ncffile, 'TSTEP')
  • tincr = ncffile.TSTEP / 10000
  • else:
  • tincr = np.diff(time_s)[0]
  • date_e = date_s.copy()
  • time_e = time_s.copy() + tincr
  • date_e += (time_e // 24).astype('i')
  • time_e -= (time_e // 24) * 24

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#issuecomment-299932370, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjj5AXTUMyW7_jNAh_zhr6A1rRU7Zks5r30-OgaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

barronh commented 7 years ago

Yep. I'll upload it as the new default shortly.

crchanda98 commented 6 years ago

Hi Barron,

Thanks a ton for pseudonecdf in python 2.7. Helped a lot. In my dispersion modelling learning process, I am looking for sector/activity specific emission factors (including break-up of NOx, SOx and VOCs). If you know any of such database where I can find them, it will be great.

On Mon, May 15, 2017 at 7:22 PM, barronh notifications@github.com wrote:

Closed #20 https://github.com/barronh/pseudonetcdf/issues/20.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/barronh/pseudonetcdf/issues/20#event-1082284307, or mute the thread https://github.com/notifications/unsubscribe-auth/AMsQjiw7O8rnvv-u5afH4DuEFwIOVXLJks5r6FiNgaJpZM4NQNs_ .

-- Regards

Arijit https://www.linkedin.com/pub/arijit-chanda/36/99/882 Research Engineer

Center for Study of Science, Technology and Policy, 10th Cross, Mayura Street, Papanna Layout, Nagashettyhalli, RMV Extension Stage 2, Bangalore - 560094, Karnataka, INDIA. Ph: 080-6690-2563 www.cstep.in https://mail.cstep.in/owa/redir.aspx?C=f080fda1f6514e4b8b8646c9680f26f0&URL=http%3a%2f%2fwww.cstep.in

barronh commented 6 years ago

Emission factors can be found by SCC in the EPA emission modeling platform. Search for the NEI TSD and look for links to the ftp site. There is a preliminary 2016 modeling platform. If you can’t find it, I recommend reaching out to the Emissions Inventory Assessment Group at the EPA.