Check when data already exist in database

kandread commented 8 years ago

When updating the database and a start date has been defined in the configuration file, the software does not check if that date has been downloaded already. Possible solution to add a force flag to overwrite the data, after checking if that date already exists.

kandread commented 8 years ago

Decided on forcing overwrite, assuming that if user explicitly sets start date they want to replace existing raster.

AdamJDuncan commented 8 years ago

I'm still getting this after updating to the new dbio.py and rheas.py files...

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas nowcast.conf Traceback (most recent call last): File "./rheas", line 14, in sys.exit(rheas.run()) File "/media/adam/UbuData/RHEAS/src/rheas.py", line 78, in run nowcast.execute(dbname, options) File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 187, in execute runVIC(dbname, options) File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 29, in runVIC runDeterministicVIC(dbname, options) File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 75, in runDeterministicVIC prec, tmax, tmin, wind = model.getForcings(options['vic']) File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 348, in getForcings 'precip', 'tmax', 'tmin', 'wind']} File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 347, in rtables = {v: self.createIndexTable("{0}.{1}".format(v, options[v])) for v in [ File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 298, in createIndexTable cur.execute(sql) psycopg2.ProgrammingError: relation "precip_xy" already exists

kandread commented 8 years ago

My guess is that you had a corrupted database. I've added a check to delete this table, so d1ae6c1ed7ef1e3c331328ad2cfc43c59ffbd73e should fix your problem.

AdamJDuncan commented 8 years ago

Thanks! Now it's showing this (which is what I got before I initialized the database, as well):

adam@adam-X750LA:/media/adam/UbuData/RHEAS/bin$ ./rheas nowcast.conf Traceback (most recent call last): File "./rheas", line 14, in sys.exit(rheas.run()) File "/media/adam/UbuData/RHEAS/src/rheas.py", line 78, in run nowcast.execute(dbname, options) File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 187, in execute runVIC(dbname, options) File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 29, in runVIC runDeterministicVIC(dbname, options) File "/media/adam/UbuData/RHEAS/src/nowcast.py", line 75, in runDeterministicVIC prec, tmax, tmin, wind = model.getForcings(options['vic']) File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 349, in getForcings 'precip', 'tmax', 'tmin', 'wind']} File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 348, in rtables = {v: self.createIndexTable("{0}.{1}".format(v, options[v])) for v in [ File "/media/adam/UbuData/RHEAS/src/vic/vic.py", line 292, in createIndexTable rtable = ".".join(cur.fetchone()[:2]) TypeError: 'NoneType' object has no attribute 'getitem'

AdamJDuncan commented 8 years ago

For the conference call, this is where I am now:

[nowcast]
model: vic
startdate: 1991-5-1
enddate: 1991-5-21
basin: /media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/data/Bbox/BBox_PP.shp
name:PhnomPenh
resolution: 0.25

[vic]
precip: trmm
temperature: ncep
wind: ncep
initialize: no
save to: db
save: net_short

mrc@mrc-VPCZ114GX:/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/bin$ ./rheas nowcast.conf
Traceback (most recent call last):
  File "./rheas", line 17, in <module>
    sys.exit(rheas.run())
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/rheas.py", line 81, in run
    nowcast.execute(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 187, in execute
    runVIC(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 29, in runVIC
    runDeterministicVIC(dbname, options)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/nowcast.py", line 76, in runDeterministicVIC
    model.writeForcings(prec, tmax, tmin, wind)
  File "/media/mrc/91413921-20fa-49f6-a9f3-5cef68ef67b8/RHEAS/src/vic/vic.py", line 380, in writeForcings
    tmin) == len(self.lat) * ndays and len(wind) == len(self.lat) * ndays
AssertionError

I've tried different date ranges.

AdamJDuncan commented 8 years ago

Oops... clearly an issue with the data... I've tried it with CHIRPS and NCEP in Africa and the model is running.

nasa / RHEAS

Check when data already exist in database #6