search

tripl-ai / questions

Forum for asking questions relating to Arc which are not defects.
https://arc.tripl.ai
0 stars 2 forks source link

Query about the AmazonIAM auth method #4

Closed a140262 closed 4 years ago

a140262 commented 4 years ago

Could you please provide an example of CSV ingestion from a S3 bucket, via the AmazonIAM authentication method? As my understanding of the ARC internal, it requires an access key and secret to download the ETL configuration file initially. How can we avoid using the access key?

seddonm1 commented 4 years ago

It should work without any config if you have assigned an IAM role to the instance. If you have KMS then a change may be required (but need to check)

On Thu, 2 Jan 2020, 12:41 pm Melody Yang, notifications@github.com wrote:

Could you please provide an example of CSV ingestion from a S3 bucket, via the AmazonIAM authentication method? As my understanding of the ARC internal, it requires an access key and secret to download the ETL configuration file initially. How can we avoid using the access key?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tripl-ai/questions/issues/4?email_source=notifications&email_token=AA2O2UCRDIITC4MQIQJW5F3Q3VEORA5CNFSM4KB53LG2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IDTISKA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2O2UETFC2GZ4WM2EICLTTQ3VEORANCNFSM4KB53LGQ .

a140262 commented 4 years ago

The error is: AWS Access Key not provided for: s3a://xxx.json. Set etl.config.fs.s3a.access.key property or ETL_CONF_S3A_ACCESS_KEY environment variable. I think it is fired up by the config/ConfigUtils.scala

seddonm1 commented 4 years ago

Yes you are correct. I will fix this in the next few days.

On Fri, Jan 3, 2020 at 3:10 PM Melody Yang notifications@github.com wrote:

The error is: AWS Access Key not provided for: s3a://xxx.json. Set etl.config.fs.s3a.access.key property or ETL_CONF_S3A_ACCESS_KEY environment variable. I think it is fired up by the config/ConfigUtils.scala

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tripl-ai/questions/issues/4?email_source=notifications&email_token=AA2O2UGGVSCJEQ7QDTD3W2DQ326TVA5CNFSM4KB53LG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIAIRZA#issuecomment-570460388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2O2UFYCJ5CUROR7KUM74LQ326TVANCNFSM4KB53LGQ .

seddonm1 commented 4 years ago

FYI just working through this Pull Request: https://github.com/tripl-ai/arc/pull/24