Output files - Githubissues

rburghol commented 2 years ago

Overview

One work product will be to define the format that we will output for river segment model files, and land Segment model files. The output formats will contain a mixture of columns representing the data that comes out of the model and its raw form, as well as some things that may be derived products where we calculate something and add a column with the results, or in some cases just a renamed, or "alias" version of one of the standard model output columns.

One reason for aliasing, the main reason really, is that we have existing model post processing routines that expect certain call names in the files that they analyze. Qout is one of these, which is the outflow from a river segment at any time step.

General

Write R script to harvest values and save to csv

HYDR

Use columns already defined in #237 File: [uci base]_hydr.csv Path: /media/model/p6/out/river/[scenario]/stream/ Ex: OR1_7700_7980_hydr.csv

Qout: this is an alias of the river segment OVOL
year, month, day, hr, min: comonents of dates
wd_mgd: alias/conversion of DIV, withdrawal in million gallons per day. I don't know what the actual units for DIVR so this may or may not involve a conversion.

PERLND

Default to all outputs in a single tree. Describe trees found in the forA51037.h5 file here. File: [uci base]_perlnd_pwater.csv Ex: forA51037_perlnd_pwater.csv Path: /media/model/p6/out/land/[scenario]/eos/ Note: we need to create a scenario for these, we will name it hsp2_2022

AGWS
LZS
UZS
SURO
IFWO
AGWO
...

juliabruneau commented 2 years ago

Update:

New scenario called hsp2_2022 created in the directory Comment with link to online index and directory path: https://github.com/HARPgroup/HARParchive/issues/237#issuecomment-1171233735

juliabruneau commented 2 years ago

Trees in forA51800

CONTROL/
    EXT_SOURCES/
    GLOBAL/
    OP_SEQUENCE/
        table*

PERLND/
    ATEMP/
        PARAMETERS/
        SAVE/
    GENERAL/
        ACTIVITY/
        INFO/
    PWATER/
        MONTHLY/
        PARAMETERS/
        SAVE/
        STATES/
    SNOW/
        FLAGS/
        PARAMETERS/
        SAVE/
        STATES/
           table*

RESULTS/
    PERNLD_P001/
        ATEMP/
        PWATER/
        SNOW/
            table*

RUN_INFO/
    LOGFILE/
        table*

TIMESERIES/
    LAPSE_Table/
    SEASONS_Table/
    SUMMARY/
    Saturated_Vapor_Pressure_Table/
    TS****/
        table*

(*) tables in each subcategory

juliabruneau commented 2 years ago

from #237

DEFINITIONS: A51800.h5

This includes what the data points look like from the HDFView tables Definitions from HSPF manual: https://github.com/respec/HSPsquared/blob/master/docs/HSPF_v12.2_manual%2Bnav.pdf

RESULTS/PERLND_P001/PWATER/table

AGWET = evapotranspiration from groundwater [in/interval] Data: often zero, but occasional periods of values

AGWI = active groundwater inflow [in/ivld] Data: often zero, but occasional periods of values

AGWLI = active groundwater lateral inflow [in/ivld] Data: zeros

AGWO = active groundwater outflow [in/interval] Data: often values, but occasional periods of zeros

AGWS = active groundwater storage at the start of the interval [in] Data: usually values but some zeros

BASET = E-T taken for active groundwater outflow (baseflow) [in/ivld] Data: zeros

CEPE = evap. From interception storage [in/ivld] Data: often zero, but occasional periods of values

CEPS = interception storage [in] Data: approx equal amount of values and zeros

GWVS = index to groundwater slope [in] Data: negative values and zeros in the beginning, but positive mostly

IFWI = interflow inflow (excluding lateral) [in/ivld] Data: approx equal amount of values and zeros

IFWLI = interflow lateral inflow [in/ivld] Data: zeros

IFWO = interflow outflow [in/interval] Data: approx equal amount of values and zeros

IFWS = interflow storage at the start of the interval [in] Data: approx equal amount of values and zeros

IGWI = inflow to inactive (deep) GW [in/ivld] Data: zeros

INFFAC = factor to account for frozen ground effects, if applicable [none] Data: usually always 1.0, but some periods when 0.9…

INFIL = infiltration to the soil [in/ivld] Data: zeros

LZET = E-T from lower zone [in/ivld] (does E-T mean evapotranspiration here?) Data: mostly zeros but occasional positive values lower than 10

LZI = lower zone inflow [in/ivld] Data: zeros

LZLI = lower zone lateral inflow [in/ivld] Data: zeros

LZS = initial lower zone storage [in] Data: All values are approximately = 5 , but slightly less (4.97499..), and values change very slightly

PERC = percolation from upper to lower zone [in/ivld] Data: zeros

PERO = total outflow from PLS [in/ivld] (what is PLS? - It is a “Pervious Land Segment”) Data: values begin at approximately 0.003 but slowly decrease

PERS = total water stored in the PLS [in] Data: values start around 6.5 and seem to steadily decrease

PET = potential E-T, adjusted for snow cover and air temperature [in/ivld] Data: mostly zeros, occasional values less than 10

PETADJ = adjustment factor for potential ET [no units (fraction)] Data: Values are all either 0, 0.5, or 1

SUPY = water supply to soil surface [in/ivld] Data: zeros

SURI = surface inflow [in/ivld] Data: zeros

SURLI = surface lateral inflow [in/ivld] Data: zeros

SURO = surface outflow [in/ivld] Data: zeros

SURS = surface detention storage [in] Data: zeros

TAET = total simulated E-T [in/ivld] Data: mix of zeros and values less than 10

TGWS = total groundwater storage [in] Data: values start at ~ 1 and decrease steadily

UZET = E-T from upper zone [in/ivld] Data: zeros and small decimals

UZI = upper zone inflow [in/ivld] Data: zeros

UZLI = upper zone lateral inflow [in/ivld] Data: zeros

UZS = upper zone storage [in] Data: Values start at 0.6 and decrease very slowly

RESULTS/PERLND_P001/ATEMP/table

AIRTMP = corrected air temperature [degrees F] Data: all positive values approximately around 20-80, no zeros

GATMP = air temperature at gage [degrees F] Data: all positive values approximately around 20-50, no zeros

RESULTS/PERLND_P001/SNOW/table

ALBEDO = reflectivity of snowpack (only available if SNOPFG = 0) [none] Data: mostly all zeros or very close to zero

CONVINX = snow cover index [in] Data: all data points are 0.106

DEWTMP = dew point [degrees F] Data: No zeros, data ranges from 20-90

DULL = dullness index of snowpack, available if SNOPFG = 0) [none] Data: all zero or close to zero

MELT = quantity of melt from PACKF [in/ivld] Data: mostly all zero, range from 0-3

NEGHTS = negative heat storage [in] Data: all zero or very close to zero

PACK = total contents of pack (water equivalent) [in] Data: ranges from 0-0.6

PACKF = frozen contents of the pack (snow and ice) [in] Data: mostly all zero

PACKI = ice in pack [in] Data: mostly all zeros

PACKW = liquid water in pack [in] Data: mostly all zeros

PAKTMP = mean temperature of the snowpack [degrees F] Data: all data points are 32

PDEPTH = pack depth [in] Data: mostly all zeros

PRAIN = rainfall directly onto the snowpack [in/ivld] Data: all zero

RAINF = rainfall [in/ivld] Data: mostly all zero or close to zero

RDENPF = relative density of frozen contents of pack (PACKF/PDEPTH) [none] Data: all “NaN”

SKYCLR = fraction of sky assumed clear [none] Data: range from 0.1-1.0

SNOCOV = fraction of land segment covered by pack [none] Data: range from 0-1

SNOTMP = max air temperature for which snowfall occurs [degrees F] Data: either 32.0 or 33.0

SNOWE = evaporation from PACKF (sublimation) [in/ivld] Data: almost all zero

SNOWF = snowfall [in/invld] Data: almost all zero or close to zero

WYIELD = water yielded by the pack (released to the land-surface) [in/ivld] Data: almost all zero or close to zero

XLNMLT = maximum increment to ice in pack [in] Data: some zeros, some range from 0-0.3

megpritch commented 2 years ago

Generating .csv's From .h5 Files in Online hsp2_2022 Directory (with timestamps!)

We were able to take the PWATER table from the RESULTS group of A51800.h5 and turn it into a .csv with converted timestamps through the method below:

Since the .h5 files will be shared in the online directory, we can create the .csv's within that directory and avoid having to move or copy files around.

We had one issue though-- only I have been able to add things to the hsp2_2022 directory (we think because I created it). Glenn keeps receiving permission errors when trying to add files to it and both Gavriel and Nicole have had trouble navigating into the directory through cd (it says the directory cannot be found).

~$ cd /media/model/p6/out/land/hsp2_2022/eos    # navigate to online directory
$ R
> library(rhdf5)
> fid = H5Fopen("forA51800.h5")
> did = H5Dopen(fid, "RESULTS/PERLND_P001/PWATER/table")
> pwater <- H5Dread(did, bit64conversion = "double")     # setup converting timestamps
> origin<- "1970-01-01"
> pwater$index <- as.POSIXct((pwater$index)/1000000000, origin = origin, tz = "UTC")     # convert timestamps
> write.table(pwater, file = "forA51800_pwater.csv", sep = "," , row.names = FALSE)      # creates csv
> H5Fclose(fid)
> H5Dclose(did)     # always close what you open
> q()
$

Here is a preview of the csv:

For reference, this is Glenn's error while following the same workflow that was successful for Megan yesterday:

gcampagna11@deq2:/media/model/p6/out/land/hsp2_2022$ cp ~/A51800_atemp.csv ./eos/
cp: cannot create regular file './eos/A51800_atemp.csv': Permission denied

juliabruneau commented 2 years ago

I was able to connect to the directory, but these are all the errors I received in R...

> library("rhdf5")
> fid = H5Fopen("forA51800.h5")
Error in H5Fopen("forA51800.h5") :
  HDF5. File accessibility. Unable to open file.
> did = H5Dopen(fid, "RESULTS/PERLND_P001/PWATER/table")
Error: Error in h5checktype(). H5Identifier not valid.
> pwater <- H5Dread(did, bit64conversion = "double")
Error: Error in h5checktype(). H5Identifier not valid.
> origin <- "1970-01-01"
> pwater$index <- as.POSIXct((pwater$index)/1000000000, origin = origin, tz = "UTC")
Error in Ops.POSIXt((pwater$index), 1e+09) :
  '/' not defined for "POSIXt" objects
> write.table(pwater,file = "forA51800_pwater.csv", sep = ",", row.names = FALSE)
Error in file(file, ifelse(append, "a", "w")) :
  cannot open the connection
In addition: Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'forA51800_pwater.csv': Permission denied

juliabruneau commented 2 years ago

I was able to connect to the directory, but these are all the errors I received in R...

> library("rhdf5")
> fid = H5Fopen("forA51800.h5")
Error in H5Fopen("forA51800.h5") :
  HDF5. File accessibility. Unable to open file.
> did = H5Dopen(fid, "RESULTS/PERLND_P001/PWATER/table")
Error: Error in h5checktype(). H5Identifier not valid.
> pwater <- H5Dread(did, bit64conversion = "double")
Error: Error in h5checktype(). H5Identifier not valid.
> origin <- "1970-01-01"
> pwater$index <- as.POSIXct((pwater$index)/1000000000, origin = origin, tz = "UTC")
Error in Ops.POSIXt((pwater$index), 1e+09) :
  '/' not defined for "POSIXt" objects
> write.table(pwater,file = "forA51800_pwater.csv", sep = ",", row.names = FALSE)
Error in file(file, ifelse(append, "a", "w")) :
  cannot open the connection
In addition: Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'forA51800_pwater.csv': Permission denied

We are still receiving the same error even after Rob's effort to fix it.

rburghol commented 2 years ago

@juliabruneau thansk for the extensive details on this. I think for now we should limit our exports to the "PWATER" series for land models.

Everyone -- I believe we're in business. For real this time. :)

Update from Analysts: yes we are in business, all of us are now able to write .csv files to the online index/directory hsp2_2022 without accessibility errors

HARPgroup / HARParchive

Output files #243

Overview

General

HYDR

PERLND

Trees in forA51800

DEFINITIONS: A51800.h5

RESULTS/PERLND_P001/PWATER/table

RESULTS/PERLND_P001/ATEMP/table

RESULTS/PERLND_P001/SNOW/table

Generating .csv's From .h5 Files in Online hsp2_2022 Directory (with timestamps!)