Open rburghol opened 2 years ago
@rburghol when we run from the home directory with input:
Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/hsp_pwater.R A51019 hsp2_2022 for /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /media/model/p532/out/land/hsp2_2022/images
I get an error:
Error: object 'RomDataSource' not found
Execution halted
Do you know what might be causing this? We didn't change any of the code involving export to VAHydro, which is where the RomDataSource is referenced
RomDataSource is from hydrotools, thus this means you must load the "hydrotools" library.
Since your script already tries to load the library, it must mean that it's not successfully loading the config.R script? Can you share where you're running this from and I can take a look?
@rburghol the issue is solved when loading hydrotools within the summary R script. The config.R file that our summary script calls is in /var/www/R on the server. We viewed the file and it appears to be the same as the file we have saved locally, so we are not sure where the issue is
I just made an adjustment to your script on the server, hope you don't mind :). The first thing I did was to load config.R, this appears to resolve everything. This is why we have config.R, because it loads all of the libraries and it sets our library location in a central place so that we don't have to struggle with loading them in our own installs separately. The next step will be to eliminate the superfluous library loads, and then if there are no libraries that you actually need to load in addition to the standard keep them.
I believe it ran successfully, although I did nothing more than just watch it go, that is to say I did not verify that it made it to VAHydro although I saw no obvious errors at a cursory glance.
@glenncampagna let me know if you can confirm.
@rburghol Yes, I can confirm the script works to export constants and graphs to VAHydro
hsp_pwater.R : hydrotools, stats (both loaded by the edited config.R file) hsp_pwater_stats.R : hydrotools, stats, IHA, caTools, RColorBrewer, plyr Note: both scripts still run successfully If the 'omsite' variable in /var/www/R/config.R is changed to http://deq1.bse.vt.edu:81/ then we can also call that variable from the config.R file rather than hard coding it in
The IWATER table within an h5 for an impervious landuse, which corresponds to the PWATER table for a pervious land use (/RESULTS/IMPLND_I001/IWATER/table) does not contain any groundwater variables (AGWS, AGWO etc.) and therefore cannot be used by our current export or summary scripts For example:
> h5ls('/opt/model/p53/p532c-sova/output/hspf/land/out/cid/hsp2_2022/cidA51037.h5', recursive=FALSE)
group name otype dclass dim
0 / CONTROL H5I_GROUP
1 / IMPLND H5I_GROUP
2 / RESULTS H5I_GROUP
3 / RUN_INFO H5I_GROUP
4 / TIMESERIES H5I_GROUP
> fid <- H5Fopen('/opt/model/p53/p532c-sova/output/hspf/land/out/cid/hsp2_2022/cidA51037.h5')
> did <- H5Dopen(fid, '/RESULTS/IMPLND_I001/IWATER/table')
> IWATER <- H5Dread(did, bit64conversion='double')
> head(IWATER)
index IMPEV PET PETADJ RETS SUPY SURI SURO SURS
1 4.417668e+17 0 0 0 0.001 0 0 1.927639e-04 0.0008072361
2 4.417704e+17 0 0 0 0.001 0 0 1.348083e-04 0.0006724278
3 4.417740e+17 0 0 0 0.001 0 0 9.935596e-05 0.0005730718
4 4.417776e+17 0 0 0 0.001 0 0 7.607375e-05 0.0004969981
5 4.417812e+17 0 0 0 0.001 0 0 5.997056e-05 0.0004370275
6 4.417848e+17 0 0 0 0.001 0 0 4.838106e-05 0.0003
Conclusion: the landuse 'cid' is impervious according to hspf
We hadn't had any luck running and debugging our original test batch script that included the CBP variables and multiple for loops
I started from scratch to simplify things and created a batch script that runs all three of our R scripts (export, summary, and summary stats) with a given land segment, land use, and scenario name and successfully pushes all expected outputs to VAHydro
The script asks for and reads in the needed variables
Running the script:
~$ bash test_pwater.bat
land segment:A51037
land use:nhy
scenario:hsp2_2022
Outputs from echo lines in the script showing how input vars were used to generate paths:
h5_file_path: /opt/model/p53/p532c-sova/output/hspf/land/out/nhy/hsp2_2022/nhyA51037.h5
output_file_path: /opt/model/p53/p532c-sova/output/hspf/land/out/nhy/hsp2_2022/eos/nhyA51037_pwater.csv
data_source_table: /RESULTS/PERLND_P001/PWATER/table
land_segment: A51037
scenario_name: hsp2_2022
landuse: nhy
image_file_path: /media/model/p532/out/land/hsp2_2022/images
Exports to VAHydro:
Implementing the loops and CBP variables to our batch script for more automation will be a bit easier now that we have a simpler working version to start with
File: https://github.com/HARPgroup/HARParchive/blob/master/HARP-2022-Summer/test_pwater.bat
Excellent work and documentation/update. A couple clarifications, requests and comments:
batch_analysis_pwater
@rburghol :
Error in plot.new() :
could not open file '/media/model/p532/out/land/hsp2_2022/images/ursA51037.fig.AGWSdecomp.png'
Calls: plot ... plot -> plot.ts -> plotts -> plot.default -> plot.new
Execution halted
R analysis scripts were run for ursA51037_pwater.csv
juliasb@deq2:/opt/model/p53/p532c-sova$ cd ~
juliasb@deq2:~$ cd /media/model/p532/out/land/hsp2_2022/images/
juliasb@deq2:/media/model/p532/out/land/hsp2_2022/images$ ls -l
total 2064
-rw-rw-r-- 1 7416721 allmodelers 37051 Jul 21 14:11 alfA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45042 Jul 21 14:11 alfA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22668 Jul 21 14:11 alfA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 37711 Jul 20 19:34 forA51011.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 44768 Jul 20 19:34 forA51011.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23310 Jul 20 19:34 forA51011.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 37553 Jul 20 13:50 forA51019.fig.AGWS.25perc.png
-rw-rw-r-- 1 7416721 allmodelers 37175 Jul 20 17:59 forA51019.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 42963 Jul 20 17:58 forA51019.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23058 Jul 20 17:59 forA51019.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 36940 Jul 21 13:22 forA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 46499 Jul 21 13:22 forA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23187 Jul 21 13:22 forA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 39205 Jul 21 14:35 homA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36009 Jul 21 14:41 homA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45277 Jul 21 14:41 homA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22252 Jul 21 14:41 homA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 80643 Jul 21 14:35 homA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 37430 Jul 21 18:20 lwmA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36361 Jul 21 18:20 lwmA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 44976 Jul 21 18:20 lwmA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22538 Jul 21 18:20 lwmA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 84515 Jul 21 18:20 lwmA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 40010 Jul 21 18:40 nexA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36940 Jul 21 18:41 nexA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 46499 Jul 21 18:41 nexA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23187 Jul 21 18:41 nexA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 70825 Jul 21 18:40 nexA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 39028 Jul 21 19:35 nhiA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 35893 Jul 21 19:36 nhiA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45424 Jul 21 19:36 nhiA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22195 Jul 21 19:36 nhiA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 82230 Jul 21 19:35 nhiA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 37370 Jul 21 19:39 nhyA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 37059 Jul 21 19:40 nhyA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 45037 Jul 21 19:40 nhyA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22660 Jul 21 19:40 nhyA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 85632 Jul 21 19:39 nhyA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 36513 Jul 21 15:10 npaA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36269 Jul 21 15:11 npaA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 44001 Jul 21 15:11 npaA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 22430 Jul 21 15:11 npaA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 80149 Jul 21 15:10 npaA51037.fig.totalOut.png
-rw-rw-r-- 1 7416721 allmodelers 40010 Jul 21 13:46 rexA51037.fig.AGWS.png
-rw-rw-r-- 1 7416721 allmodelers 36940 Jul 21 13:34 rexA51037.fig.AGWS25perc.png
-rw-rw-r-- 1 7416721 allmodelers 46499 Jul 21 13:34 rexA51037.fig.AGWSdecomp.png
-rw-rw-r-- 1 7416721 allmodelers 23187 Jul 21 13:34 rexA51037.fig.AGWSmedian.png
-rw-rw-r-- 1 7416721 allmodelers 70825 Jul 21 13:46 rexA51037.fig.totalOut.png
juliasb@deq2:/media/model/p532/out/land/hsp2_2022/images$ cd ursA51037.fig.AGWSdecomp.png
-bash: cd: ursA51037.fig.AGWSdecomp.png: No such file or directory
Checking into this now. Per your hypotheses:
cd ursA51037.fig.AGWSdecomp.png
and get an error, this is not a problem, since even if it existed (id does not), you couldn't cd
into it since it is a file, not a directory, and cd only allows you to change directory.My suspicion is that:
I will check into the permissions. Can you do 3 things to help:
Checking into this now. Per your hypotheses:
- permissions: I would expect it would explicitly report "permission denied", which doesn't appear to be the case. Also, that directory has the right permissions, group-writable, and you are in the group allmodelers so that should be straight.
- fname/image_file_path: the image file path seems good. When you try to
cd ursA51037.fig.AGWSdecomp.png
and get an error, this is not a problem, since even if it existed (id does not), you couldn'tcd
into it since it is a file, not a directory, and cd only allows you to change directory.My suspicion is that:
- Regardless of error message, it is permissions -- maybe this is something at the Denton level.
- Perhaps there is something wrong with the image itself, and thus when the function tries to create the plot file, it fails, and tells you the file is the problem -- maybe it can't write an empty image?
- Maybe the urs data file that it draws from is blank/missing, which causes the error above?
I will check into the permissions. Can you do 3 things to help:
- Send me the command that you used, and paste more of the command output prior to the error messages, there might be some helpful warnings in there.
- See if you can find the urs file, and verify that it has data in it.
- Also, try running the command with just the urs file, not in batch mode. Should be the syntax that the script assembles for you.
1. More of the code before the error:
Rows: 1 Columns: 12
── Column specification ───────────────────────────────────────────────────────────────────
Delimiter: "\t"
chr (10): datatype, varkey, timeunits, varcode, vardesc, varunits, vocabular...
dbl (2): varid, nodataval
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
[1] "Searching for 6831905" "Searching for dh_properties"
[3] "Searching for p_25th" "Searching for 1358"
[5] "Searching for dh_properties"
Skipping NULL startdate
Skipping NULL enddate
Skipping NULL propcode
select * from propvalues_tmp where propname = 'p_25th' AND varid = '1358' AND featureid = 6831905 AND entity_type = 'dh_properties'
No encoding supplied: defaulting to UTF-8.
{"self":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties?featureid=6831905\u0026entity_type=dh_properties\u0026propname=p_25th\u0026varid=1358\u0026bundle=dh_properties\u0026limit=0\u0026page=0","first":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties?featureid=6831905\u0026entity_type=dh_properties\u0026propname=p_25th\u0026varid=1358\u0026bundle=dh_properties\u0026limit=0\u0026page=0","last":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties?featureid=6831905\u0026entity_type=dh_properties\u0026propname=p_25th\u0026varid=1358\u0026bundle=dh_properties\u0026limit=0\u0026page=0","list":[{"proptext":null,"dh_link_admin_pr_condition":[],"pid":"6831911","propname":"p_25th","propvalue":"0.88988137727143","propcode":"","startdate":null,"enddate":null,"featureid":"6831905","modified":"1658767933","entity_type":"dh_properties","bundle":"dh_properties","varid":"1358","uid":"1608","vid":"7415663","status":"1","module":null,"url":"http:\/\/deq1.bse.vt.edu\/d.dh\/dh_properties\/6831911","feed_nid":null}]}
http://deq1.bse.vt.edu/d.dh/dh_properties.json
[1] "Total = 1"
Warning message:
In fn_get_rest("dh_properties", "pid", config, self$site, private$token) :
NAs introduced by coercion
site http://deq1.bse.vt.edu/d.dh
token XtSwqmo0XZz1B4M9EYsK1nlZazML0eaBERiPxhQvJdM
$pid
[1] 6831911
$entity_type
[1] "dh_properties"
$varid
[1] 1358
$bundle
[1] "dh_properties"
$featureid
[1] 6831905
$propname
[1] "p_25th"
$propvalue
[1] 0.8898814
$propcode
[1] ""
Called fn_post_rest( dh_properties , pid , pkid= 6831911 )
6831911dh_properties1358dh_properties6831905p_25th0.889881377271427
pk= 6831911
----- Updating dh_properties...
PUT URL: http://deq1.bse.vt.edu/d.dh/dh_properties/6831911
REST returned 6831911
$pid
[1] 6831911
$entity_type
[1] "dh_properties"
$varid
[1] 1358
$bundle
[1] "dh_properties"
$featureid
[1] 6831905
$propname
[1] "p_25th"
$startdate
[1] NA
$enddate
[1] NA
$propvalue
[1] 0.8898814
$propcode
[1] ""
[1] "Searching for 6831911"
select * from propvalues_tmp where pid = 6831911
Storing prop
Error in plot.new() :
could not open file '/media/model/p532/out/land/hsp2_2022/images/ursA51037.fig.AGWSdecomp.png'
Calls: plot ... plot -> plot.ts -> plotts -> plot.default -> plot.new
Execution halted
R analysis scripts were run for ursA51037_pwater.csv
Command that was used: /opt/model/p53/p532c-sova$ bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_pwater.bat hsp2_2022 OR1_7700_7980
Awesome. Try one more thing:
R
and hit enterland_segment_name <- argst[1]
, you would type land_segment_name <- 'A51037'
This will allow you to check in realtime just as if you were in Rstudio.
Awesome. Try one more thing:
- Run R, just type:
R
and hit enter- Manually paste in all of the commands from hsp_pwater_stats.R in order, with the exception that instead of getting your arguments from argst, you will set them manually with a single landuse set of params. In otehr words, instead of the line that says
land_segment_name <- argst[1]
, you would typeland_segment_name <- 'A51037'
This will allow you to check in realtime just as if you were in Rstudio.
Running the Rscript hsp_pwater_stats in terminal:
Setting the arguments manually: (I pulled the .csv file from the directory to make sure it existed, and it did.)
> landuse <- 'alf'
> land_segment_name <- 'A51037'
> scenario_name <- 'hsp2_2022'
> image_directory_path <- '/media/model/p532/out/land/hsp2_2022/images'
> pwater <-
fread("/media/model/p532/out/land/hsp2_2022/pwater/alfA51037_pwater.csv")
I received the same error immediately after trying to plot the graph:
> monthlyAGWS <- aggregate(pwater$AGWS, by = list(pwater$month, pwater$year), FUN = "mean")
colnames(monthlyAGWS) <- c("month","year", "AGWS")
yearlyAGWS <- aggregate(pwater$AGWS, by = list(pwater$year), FUN = "mean")
colnames(yearlyAGWS) <- c("year", "AGWS")
> colnames(monthlyAGWS) <- c("month","year", "AGWS")
> yearlyAGWS <- aggregate(pwater$AGWS, by = list(pwater$year), FUN = "mean")
> colnames(yearlyAGWS) <- c("year", "AGWS")
>
> AGWS_ts <- ts(monthlyAGWS$AGWS, start = c(1984,1), end = c(2020,12), frequency = 12)
>
> agws_decomp <- decompose(AGWS_ts, type = "multiplicative") #multiplicative seasonality was chosen
> save_url = paste(omsite, '/', path_string_m2, sep ='')
> fname <- paste(
+ image_directory_path,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSdecomp', '.png'), # building file name
+ sep = '/'
+ )
> furl <- paste(
+ save_url,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSdecomp', '.png'),
+ sep = '/'
+ )
> png(fname)
> plot(agws_decomp)
Error in plot.new() :
could not open file '/media/model/p532/out/land/hsp2_2022/images/alfA51037.fig.AGWSdecomp.png'
>
I tried running the code for the second graph as well to make sure that the issue isn't with the first graph, and I received the same error:
> AGWS_median <- aggregate(pwater$AGWS, by = list(pwater$year), FUN = "median")
> colnames(AGWS_median) <- c("year", "median")
> fname2 <- paste(
+ image_directory_path,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSmedian', '.png'), # building file name
+ sep = '/'
+ )
> furl2 <- paste(
+ save_url,paste0(landuse,'',land_segment_name,'.', 'fig.AGWSmedian', '.png'),
+ sep = '/'
+ )
> png(fname2)
> plot(AGWS_median$year, AGWS_median$median, type = 'l', col = 'blue', ylab = "AGWS Median (in)", xlab = NA)
Error in plot.new() :
could not open file '/media/model/p532/out/land/hsp2_2022/images/alfA51037.fig.AGWSmedian.png'
> title(main = "Annual Active Groundwater Storage")
Error in title(main = "Annual Active Groundwater Storage") :
plot.new has not been called yet
> abline(lm(AGWS_median$median ~ AGWS_median$year), col='red')
Error in int_abline(a = a, b = b, h = h, v = v, untf = untf, ...) :
plot.new has not been called yet
>
Thanks for working through the manual code execution. I'm stumped! @jdkleiner Do you have any thoughts about this? The directory is writeable…
It is DEFINITELY permissions (again). I have to get with Denton. The permissions look great, but they are not.
The batch_analysis_pwater.bat
script was ran successfully in 20 minutes!
Link to land segment A51037: http://deq1.bse.vt.edu/d.dh/om-model-info/6831636/dh_properties
Example land use alfA51037: