apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.3k stars 3.47k forks source link

[C++][R] copy_files() does not take paths to specific files #20324

Open asfimport opened 2 years ago

asfimport commented 2 years ago

copy_files() is a pretty handy function for working between local and remote interfaces, particularly for any file type arrow doesn't handle (arvo, ncdf, h5, etc etc). 

Unfortunately, it seems to work only from directory-to-directory, at least in the direction of copying S3 -> local file system.  e.g. this reprex:


library(arrow)
local_dir <- tempfile()
fs::dir_delete(local_dir)
fs::dir_create(local_dir) # create dir if it doesn't exist
l3 <- SubTreeFileSystem$create(local_dir)
l3$ls() #empty

s3 <- s3_bucket("neon4cast-targets/aquatics", endpoint_override = "data.ecoforecast.org", anonymous=TRUE)
s3$ls() #not empty, good

copy_files(s3$path("aquatics-targets.csv.gz"), l3$path("aquatics-targets.csv.gz"))
l3$ls() # darn, nothing!

copy_files(s3$path("aquatics-targets.csv.gz"), l3)
l3$ls() # darn, nothing!

copy_files(s3, l3)
l3$ls()  # Finally! only this works

Reporter: Carl Boettiger / @cboettig

Related issues:

Note: This issue was originally created as ARROW-17120. Please see the migration documentation for further details.

kostovasandra commented 11 months ago

DO you have any solution on this?