search

etd530 / Hysplit_R_interface

1 stars 0 forks source link

Restoration of R environment failing #1

Closed td296 closed 1 month ago

td296 commented 3 months ago

After cloning the repository and opening the project using R version 4.1.2 as recommended, the command renv::restore() is prompted to restore the environment using the lock file. However, packages fail to download in these conditions, with seemingly no reason why and nothing mentioned on troubleshooting. The issue appears to be due to the version dependencies of all packages, for some reason it can't download the older package versions and manual efforts only return the most updated package versions which seem incompatible with the script. Cli 3.1.0 and terra in particular are being troublesome.

Cheers

etd530 commented 3 months ago

Hi @td296, Is there any specific error message when you run renv::restore()? Can you paste the output? What operative system are you using?

td296 commented 3 months ago

Hi @etd530 Here is the current error. We are now stuck on desc 1.4.0 package.

Warning: failed to find binary for 'desc 1.4.0' in package repositories Warning: failed to retrieve 'https://mran.microsoft.com/snapshot/2022-03-06/bin/windows/contrib/4.1/desc_1.4.0.zip' [error code 6] Warning: failed to find source for 'desc 1.4.0' in package repositories Warning: failed to find binary for 'rprojroot 2.0.2' in package repositories Warning: failed to retrieve 'https://mran.microsoft.com/snapshot/2022-04-03/bin/windows/contrib/4.1/rprojroot_2.0.2.zip' [error code 6] Error: failed to retrieve package 'desc@1.4.0' In addition: There were 13 warnings (use warnings() to see them)

I am running this on Windown 10

Cheers

etd530 commented 3 months ago

Hi, I think the problem is related to this issue: https://github.com/rstudio/renv/issues/1770

The repository from where packages were downloaded for Windows was closed, but this was fixed in an upgraded version of renv. Can you try upgrading renv to its latest version and then restoring the environment again? This can be done by running renv::upgrade() and then renv::restore(). Doing this seems to fix it on my machine.

Alternatively, you can try to change the options of the .Rprofile file as explained in the above link.

Let me know if it worked!

td296 commented 3 months ago

Hi @etd530,

Thanks this has worked and all packages have installed. I am now having issues running test commands in RStudio terminal.

I have changed shebang, hy_path location and change locations of .libPaths() and Sys.setenv().

Using a test command I ge this error

username@5CG9525LTR MINGW64 /c/Hysplit_R_interface/Hysplit_script (main) $ ./Hysplit_wind_analysis.R --from 2013-10-22-06-00 --to 2013-10-25-06-00 --lat 5.745974 --lon -53.934047 --altitude 500,1000,2000 --duration -24 --byhour 1 bash: ./Hysplit_wind_analysis.R: cannot execute: required file not found

Is this an issue likely brought about because I am running it in RStudio terminal?

Cheers

etd530 commented 2 months ago

Hi,

Sorry for the slow reply. I have not managed to reproduce your issue for now. How are you working exactly? Using the Windows Subsystem for Linux (WSL) within RStudio?

One possible issue is, when editing the shebang in Windows, you may introduce carriage return symbols, since end of line characters in Windows and in Unix systems differ. See here: https://unix.stackexchange.com/questions/721844/linux-bash-shell-script-error-cannot-execute-required-file-not-found

Have you tried running fixing the end of line characters? If not, this can be done as follows:

# install dos2unix if you don't have it already (assuming Ubuntu WSL)
sudo apt install dos2unix
# navigate to the folder where you have the script
sudo dos2unix Hysplit_wind_analysis.R

Let me know if this works :)

Eric

td296 commented 2 months ago

Hi Eric,

Thank you for you continued correspondence,

We can't use Ubunutu with WSL as it is blocked by out institution. Since we are constrained to using Windows here are you able to make the scripts compatible with Windows?

We have a couple of work arounds using VM's but we are not very familiar with this.

Is it possible to edit the carriage return symbols outside of linux systems, we have tried the dos2unix program in windows but it didn't return anything.

Cheers

padpadpadpad commented 2 months ago

Hi Eric

Just tried to install this on Mac and the renv file is failing to install most packages.

We can run HySplit online and download the ASCII file. Is there a place in your script where we could read that in to do the processing from the HySplit output? I assume there must be but I cannot work out where.

Many thanks Dan

etd530 commented 2 months ago

Hi @td296,

If you cannot use WSL how are you using a bash terminal on RStudio? If your terminal is a Windows command prompt then for running the script you should move to the folder where you cloned to repository and run something like:

"C:\Program Files\R\R-4.1.2\bin\Rscript.exe" Hysplit_wind_analysis.R --from 2013-10-28-06-00 --to 2013-10-28-06-00 --lat 5.745974 --lon -53.934047 --altitude 500,1000,2000 --duration -200 --out test_0.pdf --byyear 0 --bymonth 0 --byday 0 --byhour 0 --verbose --windrose_times '-100,-200,Inf'

Changing the path to wherever you installed R in your system.

Eric

etd530 commented 2 months ago

@padpadpadpad Hi Dan,

Please, could you open a separate issue so we don't mix topics?

When you do so, please indicate what error messages you had when trying to install the packages with renv. Also, the `ProcTrajMod function takes care of writing the input files for running HYSPLIT locally and then loading the output back into R and transforming it into a dataframe, so you would have to discard the preceding code and modify that function to read your files. But this will require substantial modification.

Eric

td296 commented 1 month ago

Hi @etd530,

We were running the code in the Rstudio terminal. Have you been able to run this code on a Window machine at all?

Cheers

Toby

etd530 commented 1 month ago

Hi @td296,

Yes, I have been able to run the code on a Windows 10 machine. What I do is opening a Windows terminal (the cmd, not the new PowerShell although I guess it is also possible to do from there), navigate to the folder containing the R program and then running the command as in the example I provided in my previous comment. I tried right now and it works the same way in RStudio as long as the terminal is the cmd (this can be set up in Tools -> Global Options -> Terminal -> New terminal open with -> Select "Command prompt" option; then restart RStudio).

There was a small mistake in the example before as the name of the script was not correct; I have fixed that now. Also, please note you should change the initial part to match your installation path of R if you changed it.

Please let me know if can run it using these instructions,

Eric

td296 commented 1 month ago

Hi Eric,

Thanks for your reply. We are able to run the script from the command line, however, the script fails in loading packages from the library.

C:\Hysplit_R_interface\Hysplit_script>"C:\Program Files\R\R-4.1.2\bin\Rscript.exe" Hysplit_wind_analysis.R --from 2013-10-28-06-00 --to 2013-10-28-06-00 --lat 5.745974 --lon -53.934047 --altitude 500,1000,2000 --duration -200 --out test_0.pdf --byyear 0 --bymonth 0 --byday 0 --byhour 0 --verbose --windrose_times '-100,-200,Inf' Error: package or namespace load failed for 'opentraj': Function found when exporting methods from the namespace 'raster' which is not S4 generic: 'all.equal' In addition: Warning message: no function found corresponding to methods exports from 'raster' for: 'direction', 'gridDistance' Execution halted

This is the error we get when running the example command. We have tried reinstalling/downloading the project again several times and we have tried the renv::restore and upgrade functions. We have also tried the renv::rebuild function which loads the packages fine including opentraj. However, when trying to manually library(opentraj) it seems to try an load from :

library(opentraj, lib.loc = "C:/Users/td296/AppData/Local/R/cache/R/renv/sandbox/R-4.1/x86_64-w64-mingw32/ef755dc4") Error: package or namespace load failed for ‘opentraj’: Function found when exporting methods from the namespace ‘raster’ which is not S4 generic: ‘all.equal’ In addition: Warning message: no function found corresponding to methods exports from ‘raster’ for: ‘direction’, ‘gridDistance’

This doesn't seem right to me? The libPath was specified as per your script, but it is still loading from here and failing to load the opentraj package.

Any ideas?

Cheers

Toby

etd530 commented 1 month ago

@td296 Hi Toby,

I managed to reproduce the error in my machine and then updating the package raster fixed it. Could you please try to update it (either run renv::update("raster"), or delete and clone again the repository, which now should install the updated versions of renv and raster automatically) and let me know if it works now?

Regarding the long path, this is because the packages are actually in there in AppData, but are linked to the other folder with shortcuts.

Thanks!

td296 commented 1 month ago

Hi Eric,

Thanks for all your help! We have managed to make some progress running the script, however there are a couple of new issues.

  1. The R environment appears to be out of sync with the lock file, this is because of the splitr package. Restoring the environment which is the suggest approach does not solve this issue. However, this does not stop the script running from the command line.

C:\Hysplit_R_interface\Hysplit_script>"C:\Program Files\R\R-4.1.2\bin\Rscript.exe" Hysplit_wind_analysis.R --from 2013-10-28-06-00 --to 2013-10-28-06-00 --lat 5.745974 --lon -53.934047 --altitude 500,1000,2000 --duration -200 --out test_0.pdf --byyear 0 --bymonth 0 --byday 0 --byhour 0 --verbose --windrose_times '-100,-200,Inf'

renv::status() The following package(s) are out of sync [lockfile != library]:

GitHub ---------------------------------------------------------------------

See ?renv::status for advice on resolving these issues.

renv::restore() The following package(s) will be updated:

GitHub ---------------------------------------------------------------------

Do you want to proceed? [Y/n]: y

Installing packages --------------------------------------------------------

  1. The command prompt is throwing up some erros while calculating trajectories. e.g.

[1] "Computing trajectory for 2013-10-25 at 03:00 at altitude of 2000 and coordinates of 5.745974,-53.934047" <simpleError in setwd(hy.split.wd): cannot change working directory> [1] "Unexpected error when running date: 2013-10-25 at 03:00 at an altitude of 2000 meteres and coordinates of 5.745974 -53.934047 . Please revise that date manually."

  1. also we ahve noticed that some packages are built under version 4.1.3, but we are still running version 4.1.2. Does this matter?

library(geosphere) # needed for bearing() library(viridis) # colorblind-friendly color palettes Loading required package: viridisLite Warning message: package ‘viridis’ was built under R version 4.1.3

Any help would be greatly appreciated! We are nearly there!

Cheers

Toby

td296 commented 1 month ago

Hi Eric,

We have made significant progress to getting the Hysplit script running. However, we have run into an error in the Rscript after hysplit has run the trajectory calculation:

Percent complete: 99.5 Percent complete: 100.0 Complete Hysplit <simpleError in gzfile(file, "wb"): cannot open the connection> [1] "Unexpected error when running date: 2013-10-28 at 06:00 at an altitude of 500 meteres and coordinates of 5.745974 -53.934047 . Please revise that date manually." Error in FUN(X[[i]], ...) : object 'merged_trajs' not found Calls: lapply -> FUN In addition: Warning message: In gzfile(file, "wb") : cannot open compressed file 'C:/HYSPLIT/working/Out_files/traj2013.RData', probable reason 'No such file or directory' Execution halted

The error appears to be with a gzfile function and the compute_trajectories function although, I'm not sure why the merged trajectory object not being found is an error, from the code it should refer to the currentTraj object in this case.

Also there is a warning message at the start: Warning message: In rm(datesList_tmp) : object 'datesList_tmp' not found

Not sure if this is important or not though.

Seems like we've almost got it. Any help on this latest issue would be much appreciated.

Cheers, Toby

etd530 commented 1 month ago

Hi, Does the folder C:/HYSPLIT/working/Out_files folder exist? If not, create it manually and try again please. Thanks, Eric

td296 commented 1 month ago

Awesome Eric, It has worked! thanks very much for all your help.

etd530 commented 1 month ago

Awesome! I have updated the code so it should create that folder automatically if missing. I will close this issue now, but of course do not hesitate to open another one if more issues appear or you have suggestions/feature requests.

Eric