Test and document the initialization of the database.

[kandread]

Although a database initialization script exists for East Africa, I need to document how to initialize the database for any part of the world (code to do that already exists for the most part).

[AdamJDuncan]

Trying on the Phnom Penh area, hoping a novice user on a vanilla build (post database initialization script) helps out with spotting bugs.

This:

[domain] minlat: 8 maxlat: 35 minlon: 93 maxlon: 111

Is fine, it says the database has been updated (though geography_columns, geometry_columns, raster_columns, raster_overviews, & raster_resampled remain unpopulated)

When I add [trmm], I get this:

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas -u dataTest1.conf
Updating database!
Traceback (most recent call last):
  File "./rheas", line 14, in <module>
    sys.exit(rheas.run())
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 73, in run
    update(dbname, config_filename)
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 61, in update
    mod.download(dbname, dt, bbox)
  File "/media/adam/UbuData/RHEAS/src/datasets/trmm.py", line 30, in download
    data, lat, lon, t = fetch(dbname, dt, bbox)
  File "/media/adam/UbuData/RHEAS/src/datasets/decorators.py", line 50, in wrapper
    tdata = ds.variables[varname][ti, i[0]:i[-1] + 1, j[0]:j[-1] + 1]
  File "netCDF4/_netCDF4.pyx", line 3610, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4/_netCDF4.c:35672)
  File "/media/adam/UbuData/RHEAS/eggs/netCDF4-1.2.2-py2.7-linux-x86_64.egg/netCDF4/utils.py", line 248, in _StartCountStride
    raise IndexError(IndexErrorMsg)
IndexError: only integers, slices (`:`), ellipsis (`...`), and 1-d integer or boolean arrays are valid indices

When I delete [trmm] and add

[iri] startdate: 2000-2-1 enddate: 2000-3-1

I get this:

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas -u dataTest1.conf
Updating database!
Traceback (most recent call last):
  File "./rheas", line 14, in <module>
    sys.exit(rheas.run())
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 73, in run
    update(dbname, config_filename)
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 61, in update
    mod.download(dbname, dt, bbox)
  File "/media/adam/UbuData/RHEAS/src/datasets/iri.py", line 71, in download
    t0 = np.where(t == ((dt.year - 1960) * 12 + dt.month - 0.5))[0]
AttributeError: 'tuple' object has no attribute 'year'

When I delete that and add

[chirps] startdate: 2014-1-3

I get this:

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas -u dataTest1.conf
Updating database!
ERROR:  column "fdate" does not exist at character 12
STATEMENT:  select max(fdate) from precip.chirps
Traceback (most recent call last):
  File "./rheas", line 14, in <module>
    sys.exit(rheas.run())
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 73, in run
    update(dbname, config_filename)
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 52, in update
    dt = mod.dates(dbname)
  File "/media/adam/UbuData/RHEAS/src/datasets/chirps.py", line 32, in dates
    dts = datasets.dates(dbname, table)
  File "/media/adam/UbuData/RHEAS/src/datasets/__init__.py", line 39, in dates
    cur.execute(sql)
psycopg2.ProgrammingError: column "fdate" does not exist
LINE 1: select max(fdate) from precip.chirps
                   ^

[kandread]

I cannot reproduce your error for TRMM, I tested with your coordinates and it worked. Do you have the latest code? The IRI error should have been fixed with 0b62556235e3728cfa5ceb655588596169816d29 The CHIRPS error suggests that nothing was downloaded (it wouldn't work anyway, since the CHIRPS version included here is only for Africa).

[AdamJDuncan]

[trmm] definitely works now. I'll use that for the precipitation.

[iri] is the new script, I'm trying different date ranges, and always get this:

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas -u dataPP.conf
Updating database!
Traceback (most recent call last):
  File "./rheas", line 14, in <module>
    sys.exit(rheas.run())
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 76, in run
    update(dbname, config_filename)
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 64, in update
    mod.download(dbname, dt, bbox)
  File "/media/adam/UbuData/RHEAS/src/datasets/iri.py", line 80, in download
    data = pds.variables["prob"][tt, m, i, j, ci]
  File "netCDF4/_netCDF4.pyx", line 3611, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4/_netCDF4.c:35768)
  File "/media/adam/UbuData/RHEAS/eggs/netCDF4-1.2.2-py2.7-linux-x86_64.egg/netCDF4/utils.py", line 391, in _out_array_shape
    c = count[..., i].ravel()[0] # All elements should be identical.
IndexError: index out of bounds

The other meteorological choice -- [ncep] -- gives this:

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas -u dataPP.conf
Updating database!
Traceback (most recent call last):
  File "./rheas", line 14, in <module>
    sys.exit(rheas.run())
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 76, in run
    update(dbname, config_filename)
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 64, in update
    mod.download(dbname, dt, bbox)
  File "/media/adam/UbuData/RHEAS/src/datasets/ncep.py", line 72, in download
    _downloadVariable(varname, dbname, dt, bbox)
  File "/media/adam/UbuData/RHEAS/src/datasets/ncep.py", line 55, in _downloadVariable
    data = pds.variables[dsvar[ui]][ti, 0, i, j]
  File "netCDF4/_netCDF4.pyx", line 3610, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4/_netCDF4.c:35672)
  File "/media/adam/UbuData/RHEAS/eggs/netCDF4-1.2.2-py2.7-linux-x86_64.egg/netCDF4/utils.py", line 361, in _StartCountStride
    start[...,i] = np.apply_along_axis(lambda x: e*x, i, np.ones(sdim[:-1]))
  File "/usr/lib/python2.7/dist-packages/numpy/lib/shape_base.py", line 79, in apply_along_axis
    res = func1d(arr[tuple(i.tolist())],*args)
IndexError: index 0 is out of bounds for axis 3 with size 0

And if one of those two meteorological scripts work we've fulfilled the bare requirements for nowcast data, right? This is what a nowcast returns... so it's the lack of met forcing files, isn't it?

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas nowcast.conf
Traceback (most recent call last):
  File "./rheas", line 14, in <module>
    sys.exit(rheas.run())
  File "/media/adam/UbuData/RHEAS/src/rheas.py", line 81, in run
    nowcast.execute(dbname, options)
  File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 187, in execute
    runVIC(dbname, options)
  File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 29, in runVIC
    runDeterministicVIC(dbname, options)
  File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 75, in runDeterministicVIC
    prec, tmax, tmin, wind = model.getForcings(options['vic'])
  File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 353, in getForcings
    rtables[v] = self.createIndexTable("{0}.{1}".format(v, options[v]))
  File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 295, in createIndexTable
    rtable = ".".join(cur.fetchone()[:2])
TypeError: 'NoneType' object has no attribute '__getitem__'

[AdamJDuncan]

I was having trouble with the init db script specifically at the ncep part... everything else worked fine (I had to slice it out)... which is why I can't just try a nowcast with a date range in the 90s using that old ncep data as met forcing. Apologies, it's initializing on another computer now and I'll try once that completes.