Issues with using --profile with s3cp

ramziyassine commented 2 years ago

Overview

I want to be able to run static s3cp with an assumed role without creating an AWS user key and secret. When using --profile I am getting a class not found exception since sts jar is not packaged with igluctl

java.lang.IllegalStateException: To use assumed roles in the '***' profile, the 'sts' service module must be on the class path.
        at software.amazon.awssdk.auth.credentials.internal.ProfileCredentialsUtils.stsCredentialsProviderFactory(ProfileCredentialsUtils.java:293)
        at software.amazon.awssdk.auth.credentials.internal.ProfileCredentialsUtils.roleAndSourceProfileBasedProfileCredentialsProvider(ProfileCredentialsUtils.java:230)
        at software.amazon.awssdk.auth.credentials.internal.ProfileCredentialsUtils.credentialsProvider(ProfileCredentialsUtils.java:126)
        at software.amazon.awssdk.auth.credentials.internal.ProfileCredentialsUtils.credentialsProvider(ProfileCredentialsUtils.java:100)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.lambda$new$0(ProfileCredentialsProvider.java:73)
        at java.base/java.util.Optional.flatMap(Optional.java:294)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.<init>(ProfileCredentialsProvider.java:72)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.<init>(ProfileCredentialsProvider.java:43)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider$BuilderImpl.build(ProfileCredentialsProvider.java:212)
        at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.create(ProfileCredentialsProvider.java:114)
        at com.snowplowanalytics.iglu.ctl.commands.S3cp$.$anonfun$getS3$2(S3cp.scala:67)
        at attemptNarrow$extension @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.upload(S3cp.scala:165)
        at leftMap @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.process(S3cp.scala:47)
        at flatMap @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.getS3(S3cp.scala:75)
        at flatMap @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.getS3(S3cp.scala:75)
        at leftMap @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.process(S3cp.scala:47)
        at leftMap @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.process(S3cp.scala:47)
        at leftMap @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.process(S3cp.scala:47)
        at flatMap @ com.snowplowanalytics.iglu.ctl.commands.S3cp$.getS3(S3cp.scala:75)
        at map @ fs2.internal.CompileScope.$anonfun$close$9(CompileScope.scala:246)
        at flatMap @ fs2.internal.CompileScope.$anonfun$close$6(CompileScope.scala:245)
        at map @ fs2.internal.CompileScope.fs2$internal$CompileScope$$traverseError(CompileScope.scala:222)
        at flatMap @ fs2.internal.CompileScope.$anonfun$close$4(CompileScope.scala:244)
        at map @ fs2.internal.CompileScope.fs2$internal$CompileScope$$traverseError(CompileScope.scala:222)
        at flatMap @ fs2.internal.CompileScope.$anonfun$close$2(CompileScope.scala:242)
        at flatMap @ fs2.internal.CompileScope.close(CompileScope.scala:241)
        at main$ @ com.snowplowanalytics.iglu.ctl.Main$.main(Main.scala:23)
Caused by: java.lang.ClassNotFoundException: software.amazon.awssdk.services.sts.internal.StsProfileCredentialsProviderFactory

Version

0.8.0

Workaround

I downloaded the sts JAR from maven and tried using --classpath, but that did not work

java -cp '.:/opt/sts-2.17.42.jar' \
-jar /opt/igluctl \
static s3cp schemas/ my_bucket --region us-east-1 --profile a_profile

Any help is appreciated, maybe there is a better way to get igluctl working without creating a user.

istreeter commented 2 years ago

I'm fairly sure we can fix this by adding aws-java-sdk-sts as a runtime dependency. We recently did something similar in the collector and it makes sense to roll out that change for more applications, including igluctl.

Short term, I think this might be a valid workaround for you:

java -cp '/opt/sts-2.17.42.jar:/opt/igluctl' \
com.snowplowanalytics.iglu.ctl.Main \
static s3cp schemas/ my_bucket --region us-east-1 --profile a_profile

i.e. specify all jars with the -cp option and specify the class to run.

ramziyassine commented 2 years ago

Using the Main class instead of the Jar worked for me like you showed above, as I was including all in the classpath. Thank you so much @istreeter

snowplow / igluctl