copy_files() is a pretty handy function for working between local and remote interfaces, particularly for any file type arrow doesn't handle (arvo, ncdf, h5, etc etc).
Unfortunately, it seems to work only from directory-to-directory, at least in the direction of copying S3 -> local file system. e.g. this reprex:
library(arrow)
local_dir <- tempfile()
fs::dir_delete(local_dir)
fs::dir_create(local_dir) # create dir if it doesn't exist
l3 <- SubTreeFileSystem$create(local_dir)
l3$ls() #empty
s3 <- s3_bucket("neon4cast-targets/aquatics", endpoint_override = "data.ecoforecast.org", anonymous=TRUE)
s3$ls() #not empty, good
copy_files(s3$path("aquatics-targets.csv.gz"), l3$path("aquatics-targets.csv.gz"))
l3$ls() # darn, nothing!
copy_files(s3$path("aquatics-targets.csv.gz"), l3)
l3$ls() # darn, nothing!
copy_files(s3, l3)
l3$ls() # Finally! only this works
copy_files()
is a pretty handy function for working between local and remote interfaces, particularly for any file type arrow doesn't handle (arvo, ncdf, h5, etc etc).Unfortunately, it seems to work only from directory-to-directory, at least in the direction of copying S3 -> local file system. e.g. this reprex:
Reporter: Carl Boettiger / @cboettig
Related issues:
Note: This issue was originally created as ARROW-17120. Please see the migration documentation for further details.