Kinesis: AT_TIMESTAMP not working

[melo]

Hi,

Using Paws version 0.36.

I'm trying to use Kinesis GetShardIteractor with AT_TIMESTAMP, but something in the request encoding is not working properly.

I debugged this, and found the issue, but I'm not sure what to do next. Any pointers would be appreciated. I can send a PR for this afterwards.

First, this works just fine:

aws kinesis get-shard-iterator --shard-id shardId-000000000000 --stream-name logger --shard-iterator-type AT_TIMESTAMP --timestamp 1515322000

Using the AWS CLI I get back an iterator without any problems.

But this:

        $iter = $k->GetShardIterator(
          ShardId           => 'shardId-000000000000',
          ShardIteratorType => 'AT_TIMESTAMP',
          Timestamp         => 1515322000,
          StreamName        => $stream,
        );

fails with this exception:

class java.lang.String can not be converted to milliseconds since epoch

Trace begun at $HOME/tools/log-pattern-tester/local/lib/perl5/Paws/Net/JsonResponse.pm line 21
Paws::Net::JsonResponse::handle_response('Paws::Kinesis=HASH(0x7fceaf982b50)', 'Paws::Kinesis::GetShardIterator=HASH(0x7fceb2993ab0)', 400, '{"__type":"SerializationException","Message":"class java.lang.String can not be converted to milliseconds since epoch"}', 'HASH(0x7fceaf2477a8)') called at $HOME/tools/log-pattern-tester/local/lib/perl5/Paws/Net/Caller.pm line 40
Paws::Net::Caller::caller_to_response('Paws::Net::Caller=HASH(0x7fceb27236f0)', 'Paws::Kinesis=HASH(0x7fceaf982b50)', 'Paws::Kinesis::GetShardIterator=HASH(0x7fceb2993ab0)', 400, '{"__type":"SerializationException","Message":"class java.lang.String can not be converted to milliseconds since epoch"}', 'HASH(0x7fceaf2477a8)') called at $HOME/tools/log-pattern-tester/local/lib/perl5/Paws/Net/RetryCallerRole.pm line 21
Paws::Net::RetryCallerRole::do_call('Paws::Net::Caller=HASH(0x7fceb27236f0)', 'Paws::Kinesis=HASH(0x7fceaf982b50)', 'Paws::Kinesis::GetShardIterator=HASH(0x7fceb2993ab0)') called at $HOME/tools/log-pattern-tester/local/lib/perl5/Paws/Kinesis.pm line 65
Paws::Kinesis::GetShardIterator('Paws::Kinesis=HASH(0x7fceaf982b50)', 'ShardId', 'shardId-000000000000', 'ShardIteratorType', 'AT_TIMESTAMP', 'Timestamp', 1515322000, 'StreamName', 'logger') called at log-pattern-tester line 157
App::LogPatternTester::scan_cloudwatch_records('logger', 'shardId-000000000000', 'CODE(0x7fceb1fcdc30)') called at log-pattern-tester line 71

I added a Data::Dumper::Dumper call to Paws::Net::JsonCaller just before the encode_json() call to see the $call and $data, and they look like this:

          bless( {
                   'ShardIteratorType' => 'AT_TIMESTAMP',
                   'Timestamp' => 1515322000,
                   'StreamName' => 'logger',
                   'ShardId' => 'shardId-000000000000'
                 }, 'Paws::Kinesis::GetShardIterator' ),

          {
            'Timestamp' => '1515322000',
            'ShardIteratorType' => 'AT_TIMESTAMP',
            'ShardId' => 'shardId-000000000000',
            'StreamName' => 'logger'
          }

Notice that the Timestamp goes in as a Number and comes back as a String.

The solution was to update Paws::Kinesis::GetShardIterator and set the type of Timestamp slot to Int, and this made it work.

The documentation seems to indicate that it should accept dates like 2016-04-04T19:58:46.480-00:00 (see the Timestamp field description), but at the same time, right at the top of that link, you have:

Request Syntax

{
   "ShardId": "string",
   "ShardIteratorType": "string",
   "StartingSequenceNumber": "string",
   "StreamName": "string",
   "Timestamp": number
}

and TimeStamp is clearly mentioned as a Number.

I think moving to Int is a good fix, but I would appreciate your feedback before looking into this project more to see how to do it properly.

Thank you in advance for your time, Paws is AWSome...

[melo]

Hi,

I later remembered that Paws uses botocore...

I traced this to botocore. The declaration for timestamp in there is also inconsistent with the API documentation.

I created botocore issue 1362 to track this issue.

Not sure if you prefer to close this issue here, as the problem is upstream from Paws...

Thanks,

[pplu]

Hi Pedro,

Thanks for your work tracking this down :)

[melo]

No worries... :) I want to stop using my private patched copy as soon as possible :)

[SpencerDoesCode]

Is the same issue as with the timestamps in CloudTrail Lookup Events? If so, I'm getting something similar...

my $cloudtrail = Paws->service('CloudTrail', region => $region); ## Yes, $region is defined and valid
my $data = $cloudtrail->LookupEvents(StartTime => '2018-09-18T13:50:00', EndTime => '2018-09-18T14:00:00');
print Dumper $data; 

yields:

class java.lang.String can not be converted to milliseconds since epoch
Trace begun at <path>/Paws/Net/JsonResponse.pm line 20
Paws::Net::JsonResponse::process('Paws::Net::JsonResponse=HASH(0x5b1b520)', 'Paws::CloudTrail::LookupEvents=HASH(0x5b2b5d8)', 'Paws::Net::APIResponse=HASH(0x5b5a060)') called at <path>/Paws/Net/Caller.pm line 46
Paws::Net::Caller::caller_to_response('Paws::Net::Caller=HASH(0x50cf288)', 'Paws::CloudTrail=HASH(0x5a8b3c0)', 'Paws::CloudTrail::LookupEvents=HASH(0x5b2b5d8)', 'Paws::Net::APIResponse=HASH(0x5b5a060)') called at <path>/Paws/Net/RetryCallerRole.pm line 19
Paws::Net::RetryCallerRole::do_call('Paws::Net::Caller=HASH(0x50cf288)', 'Paws::CloudTrail=HASH(0x5a8b3c0)', 'Paws::CloudTrail::LookupEvents=HASH(0x5b2b5d8)') called at <path>/Paws/CloudTrail.pm line 61
Paws::CloudTrail::LookupEvents('Paws::CloudTrail=HASH(0x5a8b3c0)', 'StartTime', '2018-09-18T13:50:00', 'EndTime', '2018-09-18T14:00:00') called at <script>.pl line 23

[pplu]

@shangothrax : does changing the Str fields for Ints that @melo proposes fix the issue for you?

[SpencerDoesCode]

Yes. Thank you.

[melo]

@pplu I didn't noticed but botocore issue 1362 was closed with the recommendation of serialising all timestamps as epochs. I don't have the time to trace this down on Paws side today, but if you agree I can try and fix this on our side: for timestamp types, use Int as the field, and maybe add coercions from DateTime and Time::Moment?

What do you think?

[melo]

Ping?

[JohnathanSwan]

I ran into this today and your proposed solution sounds sensible - let me know if you need any help (it's not critical for me but would be nice to get this solved)