install_whitebox not able to download executable in Github Actions

[hewag1975]

Hi, we have a set of docker images which we create and maintain using Github Actions. Running whitebox::install_whitebox() perfectly works when executed in a local docker container, but fails to download the zip-file when executed during a github actions workflow. The same goes when we download the zip file using wget during image creation.

Reprex for installation in a local container (works well):

$ docker pull meterds/r-aws-spatial
$ docker run -it meterds/r-aws-spatial
root@8cd54e265d6a:/# Rscript -e "whitebox::install_whitebox()"

Same goes when instead using wget inside the container:

wget https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip
unzip WhiteboxTools_linux_amd64.zip -d /usr/local/bin/

In Github Actions this fails. Maybe the logs are helpful to understand what happens, see https://github.com/meterds/mdsrocker/actions

Any ideas what causes this? Thanks

[brownag]

Hi @hewag1975

We have encountered this before in one context: our GHA R CMD CHECK workflow that runs on Windows fails downloads every time and essentially functions as tests of the package without a binary present. @giswqs and I spend some time trying to diagnose it to no avail several months ago.

From the most recent run: https://github.com/giswqs/whiteboxR/runs/6026899039?check_suite_focus=true#step:9:601

Performing one-time download of WhiteboxTools binary from https://www.whiteboxgeo.com/WBT_Windows/WhiteboxTools_win_amd64.zip (This could take a few minutes, please be patient...) trying URL 'https://www.whiteboxgeo.com/WBT_Windows/WhiteboxTools_win_amd64.zip' trying URL 'https://www.whiteboxgeo.com/WBT_Windows/WhiteboxTools_win_amd64.zip' trying URL 'https://www.whiteboxgeo.com/WBT_Windows/WhiteboxTools_win_amd64.zip' Unable to download by any method! Try downloading ZIP manually from https://www.whiteboxgeo.com/download-whiteboxtools/. Installation involves just extracting to your desired directory. Set path to binary with wbt_init(exe_path = '/path/to/whitebox_tools') Warning messages: 1: In utils::download.file(url, exe_zip, method = method) : InternetOpenUrl failed: 'A connection with the server could not be established' 2: In utils::download.file(url, exe_zip, method = method) : URL 'https://www.whiteboxgeo.com/WBT_Windows/WhiteboxTools_win_amd64.zip': status was 'Couldn't connect to server' 3: In utils::download.file(url, exe_zip, method = method) : InternetOpenUrl failed: 'A connection with the server could not be established'

When actions run on my fork of this repo the downloads occasionally fail but usually it is just Windows.

I don't think we are sure what causes this, my suspicions past few months are that it is server-side. We have never been able to reproduce the problem locally, and above error messages are not informative (to me at least).

That it fails with wget RE your error message #6 32.69 /rocker_scripts/install_spatial.sh: line 32: wget: command not found is even odder. I have set up install_whitebox() to attempt several possible download option via download.file() but that does not help in these cases. On Linux it should use libcurl method. I figure you checked this... but you do have wget installed, right? The auto method may fall back to wget if libcurl is not available? We actually currently do not attempt wget or curl download methods explicitly. THis makes me realizew the download.file() method argument should probably be exposed via install_whitebox() for use cases like this.

Since this repository is just the for the R wrapper of WhiteboxTools, and we use the whiteboxgeo.com links, it may not be something we can address in whitebox package other than adding the other download methods as options.

Prior to my tenure as maintainer we did use to use a GitHub link to download the binaries, perhaps I could set up a workflow so I can trigger download of ZIP files following WBT releases, store as an artifact, and use link to artifact as a "mirror" for when the official links fail?

@jblindsay @afrancioni @giswqs Have you had anyone encounter issues with download of binaries in remote/headless/Github Actions contexts over on https://github.com/jblindsay/whitebox-tools or any of the other wrapper packages? I did a quick search and didn't see anything specifically addressing it.

[giswqs]

I had this issue before. GitHub Actions randomly failed when downloading binaries from whiteboxgeo.com. However, it works fine if users install the package locally. It could be the GitHub Actions automated download requests are too much for whiteboxgeo.com to handle and the download requests are denied. I am not sure. One workaround I used was to have backup download links on GitHub. Those binary links do not host updated binaries. The sole purpose was to pass GitHub Actions.

See the relevant code on how I handled that in the whitebox-python package. Lines 30-41 https://github.com/giswqs/whitebox-python/blob/master/whitebox/download_wbt.py#L30

Lines 69-74 https://github.com/giswqs/whitebox-python/blob/master/whitebox/download_wbt.py#L69

You can see the same timeout errors I encountered several months ago. https://github.com/giswqs/whitebox-python/actions/runs/1203542459

[hewag1975]

Thanks for your support and suggestions! We will add a copy of the executable to the repo from which we create the docker images and copy it over on build. If the download from whiteboxgeo.com fails during build, we are using this one instead. r -e 'success = whitebox::install_whitebox(pkg_dir = "/usr/local/bin"); if (is.null(success)) unzip("/tmp/WhiteboxTools_linux_amd64.zip", exdir = "/usr/local/bin")'