calculate_vp autoconf preferred settings

[ElmoreEcology]

I noticed that in the biorad help for calculate_vp that some of the preferred settings are different than in Dokter et al (2018) bioRad paper. for example, the range_max in bioRad help lists 25000 but your updated paper lists 35000 as the recommended setting for the same parameter. Have these autoconf settings been updated in your package and the documentation is out of date or do I need to set autoconf=FALSE and update to recommended parameters in my code?

If the latter, then I am running into some type of overwrite issue. When I have autoconf=TRUE, my files output to the designated output with no problems. With autoconf=FALSE, I get the following error using the following code.

for (file_in in files){
  file_out<- paste(outputdir, "/", basename(file_in), "_vp.h5", sep="")
  try(vp<-calculate_vp(file_in, fileout, autoconf=FALSE, dual_pol=TRUE, rho_hv=0.95, sd_vvp_threshold = 1, range_min=5000, range_max=35000))
}
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning messages:
1: In calculate_vp(pvolfile = file_in, vpfile = file_out, pvolfile_out = fileout,  :
  options.conf file found in directory D:/RadarData/KINX. Renamed to options.conf.save to prevent overwrite...
2: In file(file, ifelse(append, "a", "w")) :
  cannot open file 'D:/RadarData/KINX/options.conf': Permission denied

[adokter]

Hi @ElmoreEcology, Yes there has been some going back and forth on whether to recommend 25000 or 35000 as maximum range. For a long time we have used 25000, as the best compromise between data availability, altitude coverage and altitude resolution. But we have found for several US NEXRAD radars that 25000 meter can be insufficient, as certain volume coverage patterns that don't scan at some of the higher elevation angles. In those cases it's better to use 35000.

But you have a good point that it's good to have the default setting match the recommendations in the paper (35000), I will fix that in the next release

[adokter]

Closed by commit https://github.com/adokter/bioRad/commit/f7e2f7be7c717d0435a24d24dc1794419af025c1

[adokter]

@ElmoreEcology regarding you're other problem with writing permissions, did you verify that your Docker daemon has read/writing access to your D:\ drive? It has to be added as a shared drive under your Docker preferences

[ElmoreEcology]

Yes, I double checked Docker settings. When I run the following code, the loop works perfectly yet when I run as I mentioned in the first comment, I get the write permission error. for (file_in in files){ file_out<- paste(outputdir, "/", basename(file_in), "_vp.h5", sep="") try(vp<-calculate_vp(file_in, fileout, autoconf=TRUE)) }

[adokter]

Necessary to address adokter/vol2bird#117 in next release to have this fixed fully

[adokter]

I tried to reproduce your error with almost the identical script as you provided above:

files=list.files("~/Desktop/KINX", full.names = T)
outputdir="~/Desktop/KINX"
for (file_in in files){
  file_out<- paste(outputdir, "/", basename(file_in), "_vp.h5", sep="")
  try(vp<-calculate_vp(file_in, file_out, autoconf=FALSE, dual_pol=TRUE, rho_hv=0.95, sd_vvp_threshold = 1, range_min=5000, range_max=35000))
}

However I find the code runs as expected.

I think you have to check carefully the read/write permissions you have set for your D: drive, as you don't seem to have write permissions from R for that drive. One thing to try is if you can succesfully write a simple file to that drive from R, e.g.

write.csv("this is a test", file = "D:/RadarData/my_test.csv")

[ElmoreEcology]

I wrote the test csv to that drive, so I definitely have read/write permissions on it. I have been reading and writing to this drive for months before as well. So, I performed a full reset on Docker, restarted my machine, rechecked my D drive on shared drives in Docker, then restarted my machine again. Tried to rerun my script and it gives me the same error. It did process one file however before it started cycling errors (see below). I also noticed that there is now an 'options.conf' file in the input radar files folder. When I try to delete this, it says that I need administrator access and will not allow me to delete it. Which is strange since I am administrator. Could this file be where the problem is coming in?

> for (file_in in files){
+   file_out<- paste(outputdir, "/", basename(file_in), "_vp.h5", sep="")
+   try(vp<-calculate_vp(file_in, file_out, autoconf=FALSE, dual_pol=TRUE, rho_hv=0.95, sd_vvp_threshold = 1, range_min=5000, range_max=35000))
+ }
Loaded user configuration file 'options.conf' ...
Error: ODIM HDF5 file is not a polar volume
Error: failed to read radar volume
Error in calculate_vp(file_in, file_out, autoconf = FALSE, dual_pol = TRUE,  : 
  failed to run vol2bird Docker container
Warning: no user configuration file 'options.conf' found. Using default settings ...
Reading RSL polar volume with nominal time 20130901-011929, source: RAD:KINX,PLC:TULSA,state:OK,radar_name:KINX
Warning: radial velocities will be dealiased...
Warning: disabling single-polarization precipitation filter for S-band data, continuing in DUAL polarization mode
# vol2bird Vertical Profile of Birds (VPB)
# source: RAD:KINX,PLC:TULSA,state:OK,radar_name:KINX
# polar volume input: KINX20130901_011929_V06.gz
# volume coverage pattern (VCP): 32
# date   time HGHT    u      v       w     ff    dd  sd_vvp gap dbz     eta   dens   DBZH   n   n_dbz n_all n_dbz_all
20130901 0119    0    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119  200   0.36   6.18   26.15  6.20   3.4   2.69 F  12.60  393.6  35.78  13.00 52052 54556 52070 54574
20130901 0119  400   0.75   5.89    8.88  5.94   7.3   1.99 F   4.81   65.4   0.00   5.43 32040 32079 32066 32105
20130901 0119  600   1.65   4.59    3.94  4.88  19.8   1.95 F  -0.07   21.3   0.00  -0.11 31557 31624 32810 32891
20130901 0119  800   2.68   3.64   -0.84  4.52  36.4   1.81 F  -2.96   10.9   0.00  -2.93 15770 15997 18573 18903
20130901 0119 1000   3.71   2.69   -2.62  4.58  54.0   1.83 F  -4.57    7.5   0.00  -4.81  8301  8630 12910 13551
20130901 0119 1200   3.83   1.86   -5.00  4.25  64.1   1.89 F  -6.01    5.4   0.00  -6.05  4306  5030  9192 10602
20130901 0119 1400   3.96   0.97   -5.13  4.07  76.2   1.92 F  -6.38    5.0   0.00  -5.95  2300  3034  5866  7746
20130901 0119 1600   4.12  -0.31   -9.77  4.14  94.3   1.88 F  -5.72    5.8   0.00  -6.76  1497  2863  3844  6899
20130901 0119 1800   3.93  -2.62   -5.82  4.73 123.7   1.94 F  -7.68    3.7   0.00  -8.28   374  1375  1664  3396
20130901 0119 2000   3.80  -3.75   -7.50  5.34 134.6   1.94 F -12.63    1.2   0.00 -10.82   334  1236  1194  2701
20130901 0119 2200    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 2400    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 2600    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 2800    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 3000    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 3200    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 3400    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 3600    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 3800    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 4000    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 4200    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 4400    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 4600    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130901 0119 4800    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning messages:
1: In calculate_vp(file_in, file_out, autoconf = FALSE, dual_pol = TRUE,  :
  options.conf file found in directory D:/RadarData/KINX. Renamed to options.conf.save to prevent overwrite...
2: In file(file, ifelse(append, "a", "w")) :
  cannot open file 'D:/RadarData/KINX/options.conf': Permission denied

[ElmoreEcology]

Update: I backed up the .gz files from my desktop to an external drive and am currently running the same script on my laptop. Before running, I deleted the options.conf file from the folder. It seems to be working properly but as of now it has only processed a few hundred files. Also, I shut down my desktop for the night last night (I usually leave it running or in sleep mode). After coming in this morning, it allowed me to delete the options.conf file without asking for administrator privileges. So, I started the script and it seems to be working as well (again, only a couple hundred files have been processed out of ~38,000 so we'll see what happens). This firms my suspicion that something is going on with the options.conf file. I'll update later when the scripts finish or start throwing errors.

[ElmoreEcology]

I typed too soon. Laptop script was running properly longer than desktop but both started errors after a couple hundred files. I caught it as soon as it happened. See below. It threw an error, then processed one more file, then started giving the same errors I had above. Also, I cannot delete the options.conf file on either computer.

Loaded user configuration file 'options.conf' ...
Unknown input file type.  File <KVNX20130902_082304_V07.gz> is not recognized by RSL.
critical error, cannot open file KVNX20130902_082304_V07.gz
Error: failed to read radar volume
Error in calculate_vp(file_in, file_out, autoconf = FALSE, dual_pol = TRUE,  : 
  failed to run vol2bird Docker container
Warning: no user configuration file 'options.conf' found. Using default settings ...
Reading RSL polar volume with nominal time 20130902-083220, source: RAD:KVNX,PLC:VANCE_AFB,state:OK,radar_name:KVNX
Warning: resampling DBZH sweep at elevation 0.483395 (361 rays into 360 azimuth-bins) ...
Warning: resampling RHOHV sweep at elevation 0.483395 (361 rays into 360 azimuth-bins) ...
Warning: resampling VRADH sweep at elevation 1.494140 (361 rays into 360 azimuth-bins) ...
Warning: radial velocities will be dealiased...
Warning: disabling single-polarization precipitation filter for S-band data, continuing in DUAL polarization mode
# vol2bird Vertical Profile of Birds (VPB)
# source: RAD:KVNX,PLC:VANCE_AFB,state:OK,radar_name:KVNX
# polar volume input: KVNX20130902_083219_V07.gz
# volume coverage pattern (VCP): 32
# date   time HGHT    u      v       w     ff    dd  sd_vvp gap dbz     eta   dens   DBZH   n   n_dbz n_all n_dbz_all
20130902 0832    0    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832  200    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832  400  -6.02  -4.57   11.25  7.55 232.8   2.56 F  18.08 1388.5 126.23  18.24 30319 31679 30328 31688
20130902 0832  600  -9.68  -4.18    1.98 10.54 246.6   1.64 F  18.61 1568.2   0.00  18.61 20238 20270 20238 20270
20130902 0832  800  -8.56  -6.00   -5.76 10.45 235.0   2.20 F  18.28 1453.1 132.10  18.28 25109 25140 25109 25140
20130902 0832 1000  -6.74  -9.94  -17.91 12.01 214.2   2.58 F  17.81 1304.4 118.58  17.81 15529 15566 15529 15566
20130902 0832 1200  -5.03  -9.96  -25.30 11.16 206.8   2.71 F  16.00  861.3  78.30  16.00 13965 14057 13965 14057
20130902 0832 1400  -2.81  -9.56  -13.14  9.97 196.4   3.00 F  13.10  441.6  40.15  13.10  9865  9950  9865  9950
20130902 0832 1600   0.68  -9.12   -5.08  9.15 175.7   2.79 F  10.99  271.6  24.69  10.99  7867  7867  7867  7867
20130902 0832 1800   3.08  -9.05   -3.40  9.56 161.2   2.78 F  10.32  232.9  21.17  10.32  5712  5714  5712  5714
20130902 0832 2000   5.06  -8.17    4.87  9.61 148.2   2.84 F  10.07  219.8  19.98  10.07  3683  3683  3683  3683
20130902 0832 2200   6.23  -7.34   13.08  9.63 139.7   2.76 F   8.55  154.9  14.09   8.55  2053  2061  2053  2061
20130902 0832 2400    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 2600    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 2800    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 3000    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 3200    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 3400    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 3600    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 3800    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 4000    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 4200    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 4400    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 4600    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
20130902 0832 4800    nan    nan     nan   nan   nan    nan T    nan    nan    nan    nan     0     0     0     0
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning messages:
1: In calculate_vp(file_in, file_out, autoconf = FALSE, dual_pol = TRUE,  :
  options.conf file found in directory D:/RadarData/KVNX. Renamed to options.conf.save to prevent overwrite...
2: In file(file, ifelse(append, "a", "w")) :
  cannot open file 'D:/RadarData/KVNX/options.conf': Permission denied

Also, another thought that I noticed when starting scripts on both my laptop and desktop. I recently updated R on my desktop to 3.5.3 and am running 3.5.1 on my laptop. Before this update everything seemed to be processing fine on my desktop (Don't remember what version but it was 3.5.0 or previous). Do you think this could be causing the problem?

[adokter]

Hi @ElmoreEcology, could you check in the D:/RadarData/KVNX folder above that there is an options.conf.save file? If there is already an options.conf in your folder and you run calculate_vp with your own options it will rename the existing options.conf to options.conf.save and generate a new options.conf. The fact that you cannot delete options.conf suggests you have no user permissions, you probably need to login as administrator or superuser to get rid of it. Could you check which user owns the options.conf on your machine, that might give a clue.

Maybe the Docker program in your machine is running with more permissions or a different identity than you as user, and when it adds an options.conf you don't have access to it; but this clearly shouldn't be possible to happen so I'd like to figure out what is going on.

[ElmoreEcology]

When I created these folders they did not have any files other than .gz and .ar2v files. When I start the script on the folder and run calculate_vp, an options.conf file is created within that input folder. If I stop the script, that particular file will be locked and I do not have user permissions on either machine, which is strange since I am administrator on both machines. However, if I wait a few hours or restart my computer, I somehow automatically have permissions and can delete that file. When I go into the security properties for that file, it shows that I am the creator and owner of the file.

As far as the docker program goes, I do think that my login for Docker uses a different email address than the username for my system, but I don't think that should make a difference.

[adokter]

Ok thanks. Each time you ran into this problem, did you get a critical error, cannot open file ... message, as in the last example you sent?

[ElmoreEcology]

Possibly, but I'm not sure. I only saw this one because I caught it as it happened. In that specific case, it processed one more file before continuously throwing errors. I usually only scrolled up and found the last error and don't remember checking before.