noamross / targets-minio-versioning

MIT License
15 stars 2 forks source link

`tar_resources_aws` or `tar_resources` seem to require a region name. #2

Open mattocci27 opened 2 years ago

mattocci27 commented 2 years ago

I originally got this error.

r$> targets::tar_make()
Error in get_region(cfgs$credentials$profile) : No region provided
✖ Problem with the pipeline.
ℹ Show errors: tar_meta(fields = error, complete_only = TRUE)
ℹ Learn more: https://books.ropensci.org/targets/debugging.html
Error:
! problem with the pipeline.

After adding a random region here (AWS_REGION="us-east-1"), the error has gone.

https://github.com/noamross/targets-minio-versioning/blob/8c2dde713ba1e7d4b7b73c7174b8b4527418fead/_targets.R#L7-L10

r$> targets::tar_make()
• start target data
• built target data
• start target model
• built target model
• end pipeline: 0.213 seconds
noamross commented 2 years ago

That's odd, as I don't have an AWS_REGION variable in my environment. However, based on the output here, you are not actually using the min.io server cache, because it is building rather than skipping the targets. So there may be some other reason that the right endpoint is not passed to targets.

mattocci27 commented 2 years ago

Sorry for the confusion. I tried it again and it's actually skipped building.

 r$> targets::tar_make()
✔ skip target data
✔ skip target model
✔ skip pipeline: 0.153 seconds

I can also access to a new bucket (different name) in the min.io server but I still need to specify AWS_REGION. I copied the minio version. The R environment is as in renv.lock. Someone may experience the same problem, so I just wanted to share. Thank you for the great tutorial!

> minio --version
minio version RELEASE.2022-07-17T15-43-14Z (commit-id=b6eb8dff649b0f46c12d24e89aa11254fb0132fa)
Runtime: go1.18.4 darwin/arm64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-0000 MinIO, Inc.
snowpong commented 1 year ago

Hi there,

I ran into the same problem. Without an AWS_REGION set it fails. I get:

targets::tar_make() Error in get_region(cfgs$credentials$profile) : No region provided x Problem with the pipeline. i Show errors: tar_meta(fields = error, complete_only = TRUE) i Learn more: https://books.ropensci.org/targets/debugging.html Error: ! problem with the pipeline. Called from: signal_abort(cnd, .file)

If I set it, for example as suggested above to AWS_REGION="us-east-1" it works fine. And it skips making the targets and finds them in minio. I can see it access the bucket when I Watch the bucket in the minio console.

Note when grepping for region we do find it in the meta file as region=NULL and setting AWS_REGION="NULL" works as well. It seems that paws::get_region(..) requires a region or will stop. It's unclear what the region is used for, but maybe it's adds it to the meta file when properly set?

Note: I used renv::restore(..) to get the project dependencies as documented - but I had to upgrade two packages (cpp11 [0.4.6], igraph [1.5.1]) to succeed on this. I'm on R 4.1 not 4.2 and on Windows.

Thanks for finding the workaround @mattocci27 btw! And of course thanks to you @noamross for the example repo.

noamross commented 1 year ago

I realized that even though I don't have AWS_REGION set, I do have a region set in my .aws/credentials file. Since targets uses the paws package for AWS/S3 interaction, and that is AWS-specific, my guess is that the paws specification is looking for region even though non-AWS services may not need it.