abrupt in downloading using sftp_download

[sunxm19]

Hi, Stenevang:

First, thanks for creating this powerful and easy-to-use package. I like sftp a lot. By the way, I am Sam from University of Illinois.

My question is that when I set up sftp_download to download files from a SFTP server, it abruptly stopped after successfully downloaded first 2-3 files, as shown below:

sftp_con <- sftp_connect(server = "sftp://quagmire.igb.illinois.edu/",
                         folder = "home/shared/licor_data/maize/raw/",
                         username = Sys.getenv("MY_SFTP_USER"),
                         password = Sys.getenv("MY_SFTP_PASS"),
                         protocol = "sftp://",
                         port = 22)

files_list <- 
  sftp_list(sftp_connection = sftp_con,
                               verbose = TRUE,
                               curlPerformVerbose = FALSE,
                               type = "file")

sftp_download(files_list$name, tofolder = "content/post/eddy_covariance/data/energy_farm/maize")

and here is the error:

SFTP source: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ Save to folder: content/post/eddy_covariance/data/energy_farm/maize 100 file(s) to download. 2019-12-07T013000_AIU-0496_maize.ghg downloaded 2019-12-07T020000_AIU-0496_maize.ghg downloaded Error in function (type, msg, asError = TRUE) : Failed to connect to quagmire.igb.illinois.edu port 22: Timed out

[stenevang]

Hi, the error message says that the connection failed with a Time out, meaning that the server quagmire.igb.illinois.edu did not respond soon enough following the request to connect. It is hard to tell what the reason for this might be. If the download of each file takes considerable time, such as several minutes, it could be that your login session expires. You may have to modify your code a bit, so that you loop over your list of files to download one file at a time, and re-create the sftp_con object in each iteration, i.e. immediately before downloading each file. Please try it.

[sunxm19]

Hi, Stenevang:

Really appreciated for your quick response. I tried both ways for iterate/loop: for loop and purrr::map, but regrettably not working:

For loop ` ## for loop

for (i in 1: nrow(files_list)){ sftp_con <- sftp_connect(server = "sftp://quagmire.igb.illinois.edu/", folder = "home/shared/licor_data/maize/raw/", username = Sys.getenv("MY_SFTP_USER"), password = Sys.getenv("MY_SFTP_PASS"), protocol = "sftp://", port = 22)

files_list <- sftp_list(sftp_connection = sftp_con, verbose = TRUE, curlPerformVerbose = FALSE, type = "file") %>% tail(100)

sftp_download(files_list$name[i], tofolder = "./content/post/eddy_covariance/data/energy_farm/maize/", verbose = TRUE) }`

SFTP url: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ 8521 file(s) in SFTP folder. SFTP source: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ Save to folder: ./content/post/eddy_covariance/data/energy_farm/maize 1 file(s) to download. 2019-12-07T013000_AIU-0496_maize.ghg downloaded 1 file(s) downloaded. SFTP url: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ 8521 file(s) in SFTP folder. SFTP source: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ Save to folder: ./content/post/eddy_covariance/data/energy_farm/maize 1 file(s) to download. Error in function (type, msg, asError = TRUE) : Failed to connect to quagmire.igb.illinois.edu port 22: Timed out

purrr::map purrr::map(files_list$name, sftp_download, tofolder = "./content/post/eddy_covariance/data/energy_farm/maize/", verbose = TRUE)

SFTP source: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ Save to folder: ./content/post/eddy_covariance/data/energy_farm/maize 1 file(s) to download. 2019-12-07T013000_AIU-0496_maize.ghg downloaded 1 file(s) downloaded. SFTP source: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ Save to folder: ./content/post/eddy_covariance/data/energy_farm/maize 1 file(s) to download. 2019-12-07T020000_AIU-0496_maize.ghg downloaded 1 file(s) downloaded. SFTP source: sftp://quagmire.igb.illinois.edu/home/shared/licor_data/maize/raw/ Save to folder: ./content/post/eddy_covariance/data/energy_farm/maize 1 file(s) to download. Error in function (type, msg, asError = TRUE) : Failed to connect to quagmire.igb.illinois.edu port 22: Timed out

[sunxm19]

Hi, Stenevang: I guess my issue might be similar to this stackoverflow question.

Would you mind take a looking?

Appreciated

[stenevang]

Yes, it would be a good idea to expose the RCurl connecttimeout setting so that users can raise it when needed. I'll try to add that as soon as I have time.

[stenevang]

connecttimeout setting added.