HARPgroup / HARParchive

This repo houses HARP code development items, resources, and intermediate work products.
1 stars 0 forks source link

HARP Weekly 2022/07/18 #306

Open rburghol opened 2 years ago

rburghol commented 2 years ago
glenncampagna commented 2 years ago

Error when running summary script to document example

@rburghol when we run from the home directory with input: Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/hsp_pwater.R A51019 hsp2_2022 for /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /media/model/p532/out/land/hsp2_2022/images I get an error:

Error: object 'RomDataSource' not found
Execution halted

Do you know what might be causing this? We didn't change any of the code involving export to VAHydro, which is where the RomDataSource is referenced

rburghol commented 2 years ago

RomDataSource is from hydrotools, thus this means you must load the "hydrotools" library.

Since your script already tries to load the library, it must mean that it's not successfully loading the config.R script? Can you share where you're running this from and I can take a look?

glenncampagna commented 2 years ago

@rburghol the issue is solved when loading hydrotools within the summary R script. The config.R file that our summary script calls is in /var/www/R on the server. We viewed the file and it appears to be the same as the file we have saved locally, so we are not sure where the issue is

rburghol commented 2 years ago

I just made an adjustment to your script on the server, hope you don't mind :). The first thing I did was to load config.R, this appears to resolve everything. This is why we have config.R, because it loads all of the libraries and it sets our library location in a central place so that we don't have to struggle with loading them in our own installs separately. The next step will be to eliminate the superfluous library loads, and then if there are no libraries that you actually need to load in addition to the standard keep them.

I believe it ran successfully, although I did nothing more than just watch it go, that is to say I did not verify that it made it to VAHydro although I saw no obvious errors at a cursory glance.

@glenncampagna let me know if you can confirm.

@rburghol Yes, I can confirm the script works to export constants and graphs to VAHydro

glenncampagna commented 2 years ago

Packages removed from summary scripts so far

hsp_pwater.R : hydrotools, stats (both loaded by the edited config.R file) hsp_pwater_stats.R : hydrotools, stats, IHA, caTools, RColorBrewer, plyr Note: both scripts still run successfully If the 'omsite' variable in /var/www/R/config.R is changed to http://deq1.bse.vt.edu:81/ then we can also call that variable from the config.R file rather than hard coding it in

glenncampagna commented 2 years ago

Notes about Impervious land uses:

The IWATER table within an h5 for an impervious landuse, which corresponds to the PWATER table for a pervious land use (/RESULTS/IMPLND_I001/IWATER/table) does not contain any groundwater variables (AGWS, AGWO etc.) and therefore cannot be used by our current export or summary scripts For example:

> h5ls('/opt/model/p53/p532c-sova/output/hspf/land/out/cid/hsp2_2022/cidA51037.h5', recursive=FALSE)
  group       name     otype dclass dim
0     /    CONTROL H5I_GROUP           
1     /     IMPLND H5I_GROUP           
2     /    RESULTS H5I_GROUP           
3     /   RUN_INFO H5I_GROUP           
4     / TIMESERIES H5I_GROUP           
> fid <- H5Fopen('/opt/model/p53/p532c-sova/output/hspf/land/out/cid/hsp2_2022/cidA51037.h5')
> did <- H5Dopen(fid, '/RESULTS/IMPLND_I001/IWATER/table')
> IWATER <- H5Dread(did, bit64conversion='double')

> head(IWATER)
         index IMPEV PET PETADJ  RETS SUPY SURI         SURO         SURS
1 4.417668e+17     0   0      0 0.001    0    0 1.927639e-04 0.0008072361
2 4.417704e+17     0   0      0 0.001    0    0 1.348083e-04 0.0006724278
3 4.417740e+17     0   0      0 0.001    0    0 9.935596e-05 0.0005730718
4 4.417776e+17     0   0      0 0.001    0    0 7.607375e-05 0.0004969981
5 4.417812e+17     0   0      0 0.001    0    0 5.997056e-05 0.0004370275
6 4.417848e+17     0   0      0 0.001    0    0 4.838106e-05 0.0003 

Conclusion: the landuse 'cid' is impervious according to hspf

glenncampagna commented 2 years ago

Batch script version 1 (simple)

rburghol commented 2 years ago

Excellent work and documentation/update. A couple clarifications, requests and comments:

juliabruneau commented 2 years ago

Encountered error when running batch_analysis_pwater

@rburghol :

Error in plot.new() :
  could not open file '/media/model/p532/out/land/hsp2_2022/images/ursA51037.fig.AGWSdecomp.png'
Calls: plot ... plot -> plot.ts -> plotts -> plot.default -> plot.new
Execution halted
R analysis scripts were run for ursA51037_pwater.csv
juliasb@deq2:/opt/model/p53/p532c-sova$ cd ~
juliasb@deq2:~$ cd /media/model/p532/out/land/hsp2_2022/images/
juliasb@deq2:/media/model/p532/out/land/hsp2_2022/images$ ls -l
total 2064
-rw-rw-r-- 1 7416721 allmodelers 37051 Jul 21 14:11 alfA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45042 Jul 21 14:11 alfA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22668 Jul 21 14:11 alfA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 37711 Jul 20 19:34 forA51011.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 44768 Jul 20 19:34 forA51011.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23310 Jul 20 19:34 forA51011.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 37553 Jul 20 13:50 forA51019.fig.AGWS.25perc.png
-rw-rw-r-- 1 7416721 allmodelers 37175 Jul 20 17:59 forA51019.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 42963 Jul 20 17:58 forA51019.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23058 Jul 20 17:59 forA51019.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 36940 Jul 21 13:22 forA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 46499 Jul 21 13:22 forA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23187 Jul 21 13:22 forA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 39205 Jul 21 14:35 homA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36009 Jul 21 14:41 homA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45277 Jul 21 14:41 homA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22252 Jul 21 14:41 homA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 80643 Jul 21 14:35 homA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 37430 Jul 21 18:20 lwmA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36361 Jul 21 18:20 lwmA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 44976 Jul 21 18:20 lwmA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22538 Jul 21 18:20 lwmA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 84515 Jul 21 18:20 lwmA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 40010 Jul 21 18:40 nexA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36940 Jul 21 18:41 nexA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 46499 Jul 21 18:41 nexA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23187 Jul 21 18:41 nexA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 70825 Jul 21 18:40 nexA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 39028 Jul 21 19:35 nhiA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 35893 Jul 21 19:36 nhiA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45424 Jul 21 19:36 nhiA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22195 Jul 21 19:36 nhiA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 82230 Jul 21 19:35 nhiA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 37370 Jul 21 19:39 nhyA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 37059 Jul 21 19:40 nhyA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45037 Jul 21 19:40 nhyA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22660 Jul 21 19:40 nhyA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 85632 Jul 21 19:39 nhyA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 36513 Jul 21 15:10 npaA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36269 Jul 21 15:11 npaA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 44001 Jul 21 15:11 npaA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22430 Jul 21 15:11 npaA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 80149 Jul 21 15:10 npaA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 40010 Jul 21 13:46 rexA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36940 Jul 21 13:34 rexA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 46499 Jul 21 13:34 rexA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23187 Jul 21 13:34 rexA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 70825 Jul 21 13:46 rexA51037.fig.totalOut.png
juliasb@deq2:/media/model/p532/out/land/hsp2_2022/images$ cd ursA51037.fig.AGWSdecomp.png
-bash: cd: ursA51037.fig.AGWSdecomp.png: No such file or directory
rburghol commented 2 years ago

Checking into this now. Per your hypotheses:

My suspicion is that:

I will check into the permissions. Can you do 3 things to help:

  1. Send me the command that you used, and paste more of the command output prior to the error messages, there might be some helpful warnings in there.
  2. See if you can find the urs file, and verify that it has data in it.
  3. Also, try running the command with just the urs file, not in batch mode. Should be the syntax that the script assembles for you.
juliabruneau commented 2 years ago

Checking into this now. Per your hypotheses:

  • permissions: I would expect it would explicitly report "permission denied", which doesn't appear to be the case. Also, that directory has the right permissions, group-writable, and you are in the group allmodelers so that should be straight.
  • fname/image_file_path: the image file path seems good. When you try to cd ursA51037.fig.AGWSdecomp.png and get an error, this is not a problem, since even if it existed (id does not), you couldn't cd into it since it is a file, not a directory, and cd only allows you to change directory.

My suspicion is that:

  • Regardless of error message, it is permissions -- maybe this is something at the Denton level.
  • Perhaps there is something wrong with the image itself, and thus when the function tries to create the plot file, it fails, and tells you the file is the problem -- maybe it can't write an empty image?
  • Maybe the urs data file that it draws from is blank/missing, which causes the error above?

I will check into the permissions. Can you do 3 things to help:

  1. Send me the command that you used, and paste more of the command output prior to the error messages, there might be some helpful warnings in there.
  2. See if you can find the urs file, and verify that it has data in it.
  3. Also, try running the command with just the urs file, not in batch mode. Should be the syntax that the script assembles for you.

1. More of the code before the error:

Rows: 1 Columns: 12
── Column specification ───────────────────────────────────────────────────────────────────
Delimiter: "\t"
chr (10): datatype, varkey, timeunits, varcode, vardesc, varunits, vocabular...
dbl  (2): varid, nodataval

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
[1] "Searching for  6831905"       "Searching for  dh_properties"
[3] "Searching for  p_25th"        "Searching for  1358"
[5] "Searching for  dh_properties"
Skipping NULL startdate
Skipping NULL enddate
Skipping NULL propcode
select * from propvalues_tmp where    propname = 'p_25th' AND varid = '1358' AND featureid = 6831905 AND entity_type = 'dh_properties'
No encoding supplied: defaulting to UTF-8.
{"self":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties?featureid=6831905\u0026entity_type=dh_properties\u0026propname=p_25th\u0026varid=1358\u0026bundle=dh_properties\u0026limit=0\u0026page=0","first":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties?featureid=6831905\u0026entity_type=dh_properties\u0026propname=p_25th\u0026varid=1358\u0026bundle=dh_properties\u0026limit=0\u0026page=0","last":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties?featureid=6831905\u0026entity_type=dh_properties\u0026propname=p_25th\u0026varid=1358\u0026bundle=dh_properties\u0026limit=0\u0026page=0","list":[{"proptext":null,"dh_link_admin_pr_condition":[],"pid":"6831911","propname":"p_25th","propvalue":"0.88988137727143","propcode":"","startdate":null,"enddate":null,"featureid":"6831905","modified":"1658767933","entity_type":"dh_properties","bundle":"dh_properties","varid":"1358","uid":"1608","vid":"7415663","status":"1","module":null,"url":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties\/6831911","feed_nid":null}]}
http://deq1.bse.vt.edu/d.dh/dh_properties.json
[1] "Total = 1"
Warning message:
In fn_get_rest("dh_properties", "pid", config, self$site, private$token) :
  NAs introduced by coercion
site http://deq1.bse.vt.edu/d.dh
token XtSwqmo0XZz1B4M9EYsK1nlZazML0eaBERiPxhQvJdM
$pid
[1] 6831911

$entity_type
[1] "dh_properties"

$varid
[1] 1358

$bundle
[1] "dh_properties"

$featureid
[1] 6831905

$propname
[1] "p_25th"

$propvalue
[1] 0.8898814

$propcode
[1] ""

Called fn_post_rest( dh_properties , pid , pkid= 6831911 )
6831911dh_properties1358dh_properties6831905p_25th0.889881377271427
pk=  6831911
----- Updating dh_properties...
PUT URL:  http://deq1.bse.vt.edu/d.dh/dh_properties/6831911
REST returned 6831911
$pid
[1] 6831911

$entity_type
[1] "dh_properties"

$varid
[1] 1358

$bundle
[1] "dh_properties"

$featureid
[1] 6831905

$propname
[1] "p_25th"

$startdate
[1] NA

$enddate
[1] NA

$propvalue
[1] 0.8898814

$propcode
[1] ""

[1] "Searching for  6831911"
select * from propvalues_tmp where   pid =  6831911
Storing prop
Error in plot.new() :
  could not open file '/media/model/p532/out/land/hsp2_2022/images/ursA51037.fig.AGWSdecomp.png'
Calls: plot ... plot -> plot.ts -> plotts -> plot.default -> plot.new
Execution halted
R analysis scripts were run for ursA51037_pwater.csv

Command that was used: /opt/model/p53/p532c-sova$ bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_pwater.bat hsp2_2022 OR1_7700_7980

rburghol commented 2 years ago

Awesome. Try one more thing:

  1. Run R, just type: R and hit enter
  2. Manually paste in all of the commands from hsp_pwater_stats.R in order, with the exception that instead of getting your arguments from argst, you will set them manually with a single landuse set of params. In otehr words, instead of the line that says land_segment_name <- argst[1], you would type land_segment_name <- 'A51037'

This will allow you to check in realtime just as if you were in Rstudio.

juliabruneau commented 2 years ago

Awesome. Try one more thing:

  1. Run R, just type: R and hit enter
  2. Manually paste in all of the commands from hsp_pwater_stats.R in order, with the exception that instead of getting your arguments from argst, you will set them manually with a single landuse set of params. In otehr words, instead of the line that says land_segment_name <- argst[1], you would type land_segment_name <- 'A51037'

This will allow you to check in realtime just as if you were in Rstudio.

Running the Rscript hsp_pwater_stats in terminal:

Setting the arguments manually: (I pulled the .csv file from the directory to make sure it existed, and it did.)

> landuse <- 'alf'
> land_segment_name <- 'A51037'
> scenario_name <- 'hsp2_2022'

> image_directory_path <- '/media/model/p532/out/land/hsp2_2022/images'

> pwater <- 
fread("/media/model/p532/out/land/hsp2_2022/pwater/alfA51037_pwater.csv")

I received the same error immediately after trying to plot the graph:

> monthlyAGWS <- aggregate(pwater$AGWS, by = list(pwater$month, pwater$year), FUN = "mean")
colnames(monthlyAGWS) <- c("month","year", "AGWS")
yearlyAGWS <- aggregate(pwater$AGWS, by = list(pwater$year), FUN = "mean")
colnames(yearlyAGWS) <- c("year", "AGWS")

> colnames(monthlyAGWS) <- c("month","year", "AGWS")
> yearlyAGWS <- aggregate(pwater$AGWS, by = list(pwater$year), FUN = "mean")
> colnames(yearlyAGWS) <- c("year", "AGWS")
>
> AGWS_ts <- ts(monthlyAGWS$AGWS, start = c(1984,1), end = c(2020,12), frequency = 12)
>
> agws_decomp <- decompose(AGWS_ts, type = "multiplicative") #multiplicative seasonality was chosen
> save_url = paste(omsite, '/', path_string_m2, sep ='')
> fname <- paste(
+   image_directory_path,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSdecomp', '.png'), # building file name
+   sep = '/'
+ )
> furl <- paste(
+   save_url,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSdecomp', '.png'),
+   sep = '/'
+ )
> png(fname)
> plot(agws_decomp)
Error in plot.new() :
  could not open file '/media/model/p532/out/land/hsp2_2022/images/alfA51037.fig.AGWSdecomp.png'
>

I tried running the code for the second graph as well to make sure that the issue isn't with the first graph, and I received the same error:

> AGWS_median <- aggregate(pwater$AGWS, by = list(pwater$year), FUN = "median")
> colnames(AGWS_median) <- c("year", "median")
> fname2 <- paste(
+   image_directory_path,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSmedian', '.png'), # building file name
+   sep = '/'
+ )
> furl2 <- paste(
+   save_url,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSmedian', '.png'),
+   sep = '/'
+ )
> png(fname2)
> plot(AGWS_median$year, AGWS_median$median, type = 'l', col = 'blue', ylab = "AGWS Median (in)", xlab = NA)
Error in plot.new() :
  could not open file '/media/model/p532/out/land/hsp2_2022/images/alfA51037.fig.AGWSmedian.png'
> title(main = "Annual Active Groundwater Storage")
Error in title(main = "Annual Active Groundwater Storage") :
  plot.new has not been called yet
> abline(lm(AGWS_median$median ~ AGWS_median$year), col='red')
Error in int_abline(a = a, b = b, h = h, v = v, untf = untf, ...) :
  plot.new has not been called yet
>
rburghol commented 2 years ago

Thanks for working through the manual code execution. I'm stumped! @jdkleiner Do you have any thoughts about this? The directory is writeable…

rburghol commented 2 years ago

It is DEFINITELY permissions (again). I have to get with Denton. The permissions look great, but they are not.

juliabruneau commented 2 years ago

Success!

The batch_analysis_pwater.bat script was ran successfully in 20 minutes!

Link to land segment A51037: http://deq1.bse.vt.edu/d.dh/om-model-info/6831636/dh_properties

Example land use alfA51037: image