Working with multiple AWS Credentials

[Heiko-san]

Hi,

sorry that I contact you this way, I can't seem to find an email address (besides jlmartinez@cpan.org which always gives me a "rejected from server" bounce).

I'm interested in using Paws for a project in my company. I'm tied to use Perl for communication to AWS API, since I have to talk to other interfaces,too for which only Perl code exists.

Since there is no official Perl SDK for AWS, I was searching CPAN and found, that Paws seems to be the only sufficient SDK. The Problem I'm facing right now is, that our software would act upon multiple AWS accounts and therefore need to provide different credential pairs for the API calls we make (mainly describe-something).

As described in the CPAN documentation in section AUTHENTICATION, the module uses fixed environment variable or the default profile of the aws cli config (~/.aws/config). The latter would be what I would prefere but I would need a way to tell Paws not to use profile default but another one (for example provided as parameter at instantiation 'profile' => 'myotherprofile').

Is there a way to do this or would this be a feature request?

Thanks for answer, Heiko

[pplu]

It's OK to contact me this way :) (it's a good way to have everybody get info)

To get Paws to use another profile in the credentials file, you can do this:

use Paws::Credential::File;
my $prof1 = Paws->new(config => { credentials =>
  Paws::Credential::File->new( profile => 'xxx' )
});
$prof1->service('XXX')->DescribeYYY;

You can also implement your own Credential provider if you store credentials in a special way. There are hints here (http://www.slideshare.net/pplusdomain/paws-perl-aws-sdk-update-november-2015). You can also take a look at the examples in the Paws::Credential namespace.

Although, maybe for your case I'd take a look at setting up CrossAccount access with Roles, which Paws supports too :) (there is an example in https://github.com/pplu/aws-sdk-perl/blob/master/examples/cross_account.pl)

[Heiko-san]

Hi that sounds quite awesome , thanks :) I will have a look at the examples as soon as possible.

[pplu]

Did this solve your issue?

[Heiko-san]

Hi,

actually not, if i do the following:

~/.aws/credentials

[profile myprof]
aws_access_key_id = ...
aws_secret_access_key = ...

#!/usr/bin/perl

use strict;
use warnings;
use Paws;
use Data::Dumper;
use Paws::Credential::File;
my $prof1 = Paws->new(config => { credentials =>
  Paws::Credential::File->new( profile => 'myprof' )
});
my $ec2 = $prof1->service('EC2', region => 'eu-west-1');
my $result = $ec2->DescribeInstances;
print Dumper $result;

I get the following error:

Use of uninitialized value in concatenation (.) or string at /usr/local/share/perl/5.20.2/Net/Amazon/Signature/V4.pm line 133.
Use of uninitialized value in join or string at /usr/local/share/perl/5.20.2/Net/Amazon/Signature/V4.pm line 139.
Authorization header or parameters are not formatted correctly.

Trace begun at /usr/local/share/perl/5.20.2/Paws/API/Caller.pm line 97
Paws::API::Caller::handle_response('Paws::EC2=HASH(0x3824188)', 'Paws::EC2::DescribeInstances=HASH(0x37d4568)', 401, '<?xml version="1.0" encoding="UTF-8"?>^J<Response><Errors><Error><Code>AuthFailure</Code><Message>Authorization header or parameters are not formatted correctly.</Message></Error></Errors><RequestID>1daa1e43-1c9b-4643-8941-51b9196f984c</RequestID></Response>', 'HASH(0x4c01d60)') called at /usr/local/share/perl/5.20.2/Paws/Net/Caller.pm line 48
Paws::Net::Caller::do_call('Paws::Net::Caller=HASH(0x384b3e8)', 'Paws::EC2=HASH(0x3824188)', 'Paws::EC2::DescribeInstances=HASH(0x37d4568)') called at /usr/local/share/perl/5.20.2/Paws/EC2.pm line 458
Paws::EC2::DescribeInstances('Paws::EC2=HASH(0x3824188)') called at xxx.pl line 29

Same works if I declare the same credentials [default] in credentials file and use

my $ec2 = Paws->service('EC2', region => 'eu-west-1');

PS: The dump of the Paws::Credential::File object looks good, as far as I can tell.

$VAR1 = bless( {
                 'path' => '/home/hfi/.aws/',
                 'profile' => 'myprof',
                 'file_name' => 'credentials'
               }, 'Paws::Credential::File' );

Well the service object looks quite different, for Paws->service('EC2', region => 'eu-west-1'):

$VAR1 = bless( {
                 'credentials' => bless( {
                                           'providers' => [
                                                            'Paws::Credential::Environment',
                                                            'Paws::Credential::File',
                                                            'Paws::Credential::InstanceProfile'
                                                          ],
                                           'selected_provider' => bless( {
                                                                           'access_key' => '...',
                                                                           'path' => '/home/hfi/.aws/',
                                                                           '_ini_contents' => {
                                                                                                'profile myprof' => {
                                                                                                                    'aws_secret_access_key' => '...',
                                                                                                                    'aws_access_key_id' => '...'
                                                                                                                  },
                                                                                                'default' => {
                                                                                                               'aws_secret_access_key' => '...',
                                                                                                               'aws_access_key_id' => '...'
                                                                                                             }
                                                                                              },
                                                                           'secret_key' => '...',
                                                                           'file_name' => 'credentials',
                                                                           'profile' => 'default',
                                                                           'credentials_file' => '/home/hfi/.aws//credentials'
                                                                         }, 'Paws::Credential::File' )
                                         }, 'Paws::Credential::ProviderChain' )
...

With the profile provider it looks like:

$VAR1 = bless( {
                 'credentials' => bless( {
                                           'file_name' => 'credentials',
                                           'profile' => 'myprof',
                                           'path' => '/home/hfi/.aws/'
                                         }, 'Paws::Credential::File' ),
...

Since selected_provider has the field profile and seems to know my "myprof" profile, too ... Is there a way to tell the first object to change the selected profile to "myprof"?

[Heiko-san]

I found the error, in Paws::Credential::File I needed to add the following lines:

  has access_key => (is => 'ro', lazy => 1, default => sub {
    my $self = shift;
    my $ini_section = $self->profile;
+  $ini_section = "profile $ini_section" unless $ini_section eq 'default';
    my $ak = $self->_ini_contents->{ $ini_section }->{ aws_access_key_id };
    return $ak;
  });
  has secret_key => (is => 'ro', lazy => 1, default => sub {
    my $self = shift;
    my $ini_section = $self->profile;
+  $ini_section = "profile $ini_section" unless $ini_section eq 'default';
    my $sk = $self->_ini_contents->{ $ini_section }->{ aws_secret_access_key };
    return $sk;
  });

After this my object looks like this after using it in the describe call:

$VAR1 = bless( {
                 'credentials' => bless( {
                                           'profile' => 'myprof',
                                           'credentials_file' => '/home/hfi/.aws//credentials',
                                           'file_name' => 'credentials',
                                           'access_key' => '...',
                                           '_ini_contents' => {
                                                                'profile myprof' => {
                                                                                      'aws_access_key_id' => '...',
                                                                                      'aws_secret_access_key' => '...'
                                                                                    },
                                                                'myprof' => {},
                                                                'default' => {
                                                                             'aws_secret_access_key' => '...',
                                                                             'aws_access_key_id' => '...'
                                                                           }
                                                              },
                                           'session_token' => undef,
                                           'secret_key' => '...',
                                           'path' => '/home/hfi/.aws/'
                                         }, 'Paws::Credential::File' ),
...

However this is a very quick and dirty solution, since aws cli allows for more than 1 space between "profile" and the profile name in the ini file and even spaces between profile name and the closing "]" but the latter seems to be handled (truncated) by Config::INI::Reader. So maybe it would be better to do a s/^profile\s+// on each section entry of the ini instead of my lines from above.

  has _ini_contents => (is => 'ro', isa => 'HashRef', lazy => 1, default => sub {
    my $self = shift;
    my $ini_file = $self->credentials_file;
    return {} if (not -e $ini_file);
    my $ini = Config::INI::Reader->read_file($ini_file);
+   my %hash = map { s/^profile\s+//r => $ini->{$_} } keys %{$ini};
~   return \%hash;
  })

[pplu]

Thanks for the detailed info, but I think there's some confusion between the credentials and the config file. On http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html, it doesn't state that profiles should have the "profile" prefix in ~/.aws/credentials. Instead, it looks like that is for "~/.aws/config", which doesn't specify credentials (just defaults for the aws CLI), but is not supported by Paws (yet).

Is there any documentation that points to the .aws/credentials files supporting the "[profile XXX]" format in some other SDK? I wouldn't want to implement a behavior very different from that of the other SDKs (so config files can just work the same between the different AWS SDKs).

[Heiko-san]

Oh , you're right, the cli doesn't support [profile myprof] in ~/.aws/credentials, but only in ~/.aws/config. Strange since I could have sworn I tested just that. Oo However thanks for the hint.

[Heiko-san]

Just in case you are interested in this / want to publish it somewhere. I installed Paws on a debian 8 (jessie) and wanted to install as many modules via deb packages as possible:

cat install/Paws.deps.jessie

Before "cpanm Paws" install the following deb packages

libaliased-perl
libconfig-ini-perl
libdata-compare-perl
libdatetime-perl
libdatetime-format-iso8601-perl
libdigest-hmac-perl
liberror-perl
libfile-homedir-perl
libfile-slurp-perl
libhash-flatten-perl
libjson-maybexs-perl
libmodule-find-perl
libmoose-perl
libmoosex-classattribute-perl
libmoosex-getopt-perl
libmoosex-types-common-perl
libtest-exception-perl
libtest-moose-more-perl
libtest-tester-perl
libtest-warnings-perl
libtest-without-module-perl
libthrowable-perl
liburi-template-perl
libxml-simple-perl
libyaml-perl

Leftover deps without deb (need cpanm install, can be installed implicitly with "cpanm Paws")

ARGV::Struct
Crypt::Digest::SHA256
File::Slurper
Net::Amazon::Signature::V3
Net::Amazon::Signature::V4
Number::Misc
String::Util
Test::Timer
URL::Encode
URL::Encode::XS

Note: The above packages include dependencies of the modules installed via cpanm, so Paws itself and the named leftover modules are the only ones installed via cpanm, they didn't have to install any further dependencies.

[pplu]

Thanks for the info: it's greatly appreciated. I'm looking to organize some type of documentation for this type of stuff.

[tryorfry]

Hi Jose, Thanks for the excellent Paws. Sorry for reopening this discussion. Is there a way to run two different profiles clients on same program life span/scope? Currently, 1st one succeeds but the 2nd one fails. Because the Authentication header from 1st client is still active and used for the 2nd request as well which then result in Forbidden error.

#!/usr/bin/perl
use Mojo::Base -strict;
use DDP;
use Core::AWS::S3;  # some simple wrapper on Paws::S3 with these two modules. 
use Core::CleverSafe;
use diagnostics;
$| = 1;

#1st client AWS
my $aws = Core::AWS::S3->new;
my $aws_res = $aws->s3->ListAllObjects(Bucket => $aws->bucket_name);
p $aws_res;

#2nd client / By the way Paws can be tweaked to make work with AWS compatible API - CleverSafe for example. Excellent!
my $cs = Core::CleverSafe->new;
my $bucket_name = $cs->bucket_name;
my $cs_result = $cs->s3->ListAllObjects(Bucket => $cs->bucket_name);
p $cs_result;

Appreciate if you can show me ways to override the Authentication header for 2nd client request. or is there a way to tell Http::Tiny not to use 1s Authentication header?

thanks, Sachin