private files downloadable without credentials?

brokamp-group / s3

More efficient downloading and usage of files hosted on AWS S3

http://brokamp-group.github.io/s3/

Other

3 stars 2 forks source link

private files downloadable without credentials? #11

Closed andrew-vancil closed 3 years ago

andrew-vancil commented 3 years ago

On Windows, s3_get() produces the error:

> test1<-s3_get("s3://geomarker/testing_downloads/mtcars.rds")
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

This happens for public and private files. Using the same setup on MacOS works properly with no errors.

cole-brokamp commented 3 years ago

Thanks for reporting. I am still trying to figure out how to do more testing and debugging on Windows. Right now, I can't tell which tests are being ran or skipped on github actions. Will circle back to this soon.

cole-brokamp commented 3 years ago

Just to update our thread here:

download.file when using wininet library seems to alter the header of the fst file and make it non-readable (https://github.com/fstpackage/fst/issues/253); we can fix this with enforcing curl when using download.file
does httr::GET suffer from this same problem?; check to see if s3_get works with a public file online when AWS credentials are unset

cole-brokamp commented 3 years ago

also linking issues in other packages that download fst files:

We will update these downstream packages when s3 is fixed.

andrew-vancil commented 3 years ago

After updating the s3 package and removing my AWS credentials, I am able to use s3_get successfully on the public file mtcars.fst and then successfully read it in using read.fst. I also tried downloading mtcars_private.fst; s3_get appears to run successfully, but the file size is only a couple hundred bytes and read.fst produces an expected error.

I replaced my AWS credentials and reran the testing but the same errors as before are produced when running s3_get.