search

ropensci / osfr

R interface to the Open Science Framework (OSF)
https://docs.ropensci.org/osfr
Other
142 stars 27 forks source link

curl error #123

Open weir12 opened 4 years ago

weir12 commented 4 years ago

Hi: I encountered this error when trying to download a large folder

Requesting folder 'Vero-Infected' from OSF
Downloading Vero-Infected.zip
Downloading: 30 GB     Error in curl::curl_fetch_disk(x$url$url, x$disk, handle = x$url$handle) : 
  transfer closed with outstanding read data remaining
Calls: osf_download ... verb_func -> <Anonymous> -> crul_fetch -> <Anonymous>
Execution halted

How to solve this error ,thanks! btw,If there is another way to download OSF data,I would be happy to try it:)

aaronwolen commented 4 years ago

Can you verify the error is consistently reproducible (seems like it might be connectivity related). If it is, please post your code so I can attempt to reproduce.

weir12 commented 4 years ago

The error does not seem to consistently repeated. Sometimes it's an error with a timeout aroused bycurl here are my code

library(osfr)
anes_project <- osf_retrieve_node("https://osf.io/8f6n9")
anes_files <- osf_ls_files(anes_project)
anes_files=osf_ls_files(anes_files[1,])
osf_download(
  anes_files,
  recurse = TRUE,
  conflicts = "overwriter",
  verbose = F,
  progress = T

Can osfr continue to download partially completed files?(likewget -c) Thank you for your support!

weir12 commented 4 years ago

I ran the command several more times.Each time it's blocked at random,Maybe it's stuck at 30GB or 40GB.For a large folder(500GB), this error can be fatal.