search

boto / botocore

The low-level, core functionality of boto3 and the AWS CLI.
Apache License 2.0
1.48k stars 1.08k forks source link

S3 raises Invalid timestamp for string format time #2902

Closed satheler closed 1 year ago

satheler commented 1 year ago

Describe the bug

When I try to give a Head Object of an object in S3 sometimes it returns LastModified in the format "Mon, 05 Apr 2023 14:30:00 EDT".

When trying to convert it raises an error 

ValueError: Invalid timestamp "Mon, 05 Apr 2023 14:30:00 EDT": Unknown string format: Mon, 05 Apr 2023 14:30:00 EDT

Expected Behavior

Returns a datetime

Current Behavior

Raises ValueError

Reproduction Steps

Try to give a Head Object of an object in S3 that the LastModified is in the format "Mon, 05 Apr 2023 14:30:00 EDT".

Possible Solution

botocore/utils.py : 908

def _parse_timestamp_with_tzinfo(value, tzinfo):
    """Parse timestamp with pluggable tzinfo options."""
    if isinstance(value, (int, float)):
        # Possibly an epoch time.
        return datetime.datetime.fromtimestamp(value, tzinfo())
    else:
        try:
            return datetime.datetime.fromtimestamp(float(value), tzinfo())
        except (TypeError, ValueError):
            pass
    try:
        # In certain cases, a timestamp marked with GMT can be parsed into a
        # different time zone, so here we provide a context which will
        # enforce that GMT == UTC.
        STRING_FORMAT_PATTERN = r'^[a-zA-Z]{3}, \d{2} [a-zA-Z]{3} \d{4} \d{2}:\d{2}:\d{2} [A-Z]{3}$'
        if re.match(STRING_FORMAT_PATTERN, value):
            return datetime.datetime.strptime(value, '%a, %d %b %Y %H:%M:%S %Z')

        return dateutil.parser.parse(value, tzinfos={'GMT': tzutc()})
    except (TypeError, ValueError) as e:
        raise ValueError(f'Invalid timestamp "{value}": {e}')

Additional Information/Context

No response

SDK version used

1.29.106

Environment details (OS name and version, etc.)

Darwin Satheler 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:04:44 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T8103 arm64

alexeiverbny commented 1 year ago

I believe I am running into the same issue when I run Kinesis.Client.list_stream_consumers(StreamARN = my_arn). It works for some values of my_arn but for others I get 

  File "/opt/homebrew/lib/python3.11/site-packages/botocore/utils.py", line 912, in _parse_timestamp_with_tzinfo
    return datetime.datetime.fromtimestamp(value, tzinfo())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: year 55226 is out of range
RyanFitzSimmonsAK commented 1 year ago

Hi @alexeiverbny, thanks for reaching out. I'm having some trouble reproducing this behavior. Could you provide debug logs for two runs, one that produces this behavior, and another that doesn't? You can get debug logs by adding boto3.set_stream_logger('') to the top of your script and removing any sensitive information. Thanks!

alexeiverbny commented 1 year ago

Hi @RyanFitzSimmonsAK, thanks for the response. Sure thing, just give me a little time to scrub the logs to make sure I don't share anything sensitive.

alexeiverbny commented 1 year ago

Hi @RyanFitzSimmonsAK, below are minimal versions of logs for two runs - 1 that has a response from AWS and no erros and another that has a response from AWS and then what appears to be a parsing error. Please let me know if there is anything else I can provide. 

2023-04-07 18:53:13,466 botocore.parsers [DEBUG] Response body:
b'{"Consumers":[{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.677176556E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.67831235E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.680042431E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.67600062E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.68063568E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.680655375E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.670905657E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.676413966E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.679691482E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.675975397E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.680883075E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.680711591E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.675817565E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.680626484099E12,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.680840854E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"}]}'
Traceback (most recent call last):
  File "XXX", line 7, in <module>
    client.list_stream_consumers(StreamARN='arn:aws:kinesis:us-east-1:XXX:stream/XXX')
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 943, in _make_api_call
    http, parsed_response = self._make_request(
                            ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 966, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 199, in _send_request
    success_response, exception = self._get_response(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 241, in _get_response
    success_response, exception = self._do_get_response(
                                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 308, in _do_get_response
    parsed_response = parser.parse(
                      ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 252, in parse
    parsed = self._do_parse(response, shape)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 849, in _do_parse
    parsed = self._handle_json_body(response['body'], shape)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 873, in _handle_json_body
    return self._parse_shape(shape, parsed_json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
    return handler(shape, node)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 663, in _handle_structure
    final_parsed[member_name] = self._parse_shape(
                                ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
    return handler(shape, node)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 340, in _handle_list
    parsed.append(self._parse_shape(member_shape, item))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
    return handler(shape, node)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 663, in _handle_structure
    final_parsed[member_name] = self._parse_shape(
                                ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
    return handler(shape, node)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 682, in _handle_timestamp
    return self._timestamp_parser(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 941, in parse_timestamp
    return _parse_timestamp_with_tzinfo(value, tzinfo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 912, in _parse_timestamp_with_tzinfo
    return datetime.datetime.fromtimestamp(value, tzinfo())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: year 55226 is out of range

No error: 

2023-04-10 09:35:34,170 botocore.parsers [DEBUG] Response body:
b'{"Consumers":[{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.679518495E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.679422098E9,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"}]}'
2023-04-10 09:35:34,171 botocore.hooks [DEBUG] Event needs-retry.kinesis.ListStreamConsumers: calling handler <botocore.retryhandler.RetryHandler object at 0x104213590>
2023-04-10 09:35:34,171 botocore.retryhandler [DEBUG] No retry needed.
RyanFitzSimmonsAK commented 1 year ago

Hi @alexeiverbny, thanks for following up. I'm still having some trouble reproducing this. Could you provide a minimal reproducible code snippet? Ideally, the shortest code snippet that still has the behavior you described. This will make it much easier to root cause. Thanks!

alexeiverbny commented 1 year ago

Hi @RyanFitzSimmonsAK sure, here is the code that produced both of the above examples with account_number and stream_name redacted: 

import boto3

boto3.set_stream_logger('')

client=boto3.client('kinesis', 'us-east-1')

client.list_stream_consumers(StreamARN='arn:aws:kinesis:us-east-1:<my_account_number>:stream/<my_stream_name>')

I think the issue here is that this is fact hard to reproduce. This code works with one value of but not with another. I don't want to share stream names here, and even if I did you would not be able to access the streams yourself since they are tied to my aws account. But note that in both of the logs above, I am in fact getting a valid response. So its not an issue of incorrect stream_name or account number. Please let me know if there is anything else I can do to help.

RyanFitzSimmonsAK commented 1 year ago

The title mentioned that this happened with S3 as well, is that using client.head_object()?

alexeiverbny commented 1 year ago

That was a different person that wrote that, I have not had issues with client.head_object(). I just saw that this issue was with the same same function that I am having an issue with - _parse_timestamp_with_tzinfo

RyanFitzSimmonsAK commented 1 year ago

Hi @satheler; I see in the PR you submitted that you mentioned you found the error. Could you share what that problem was, since another user did have this same problem? If it's a genuine bug, I'd like to continue looking into it.

RyanFitzSimmonsAK commented 1 year ago

Hi @alexeiverbny, I think I've found something worth investigating. In the logs you provided, you have a consumer with the following info.

{"ConsumerARN":"arn:aws:kinesis:us-east-1:XXX:stream/XXX/consumer/XXX","ConsumerCreationTimestamp":1.680626484099E12,"ConsumerName":"XXX","ConsumerStatus":"ACTIVE"}

That timestamp is 1.680626484099E12, in Unix time. If that's read as milliseconds, then it's a perfectly sensible timestamp. If it's read as seconds, then it's in the year 55226 (ValueError: year 55226 is out of range). Do you know how that timestamp ended up being formatted as milliseconds unlike the rest of them? Going to reopen this issue for the time being.

alexeiverbny commented 1 year ago

hmm nice find. Not sure why that got formatted that way. Let me dig into a bit and get back to you. Thanks.

RyanFitzSimmonsAK commented 1 year ago

Hey @alexeiverbny, were you able to find anything? The logs you provided earlier aren't actually debug logs, it's just the response and the trace. If you provide the debug logs, I could look into this as well. Thanks!

alexeiverbny commented 1 year ago

Hey @RyanFitzSimmonsAK. I have not had a chance to take a look yet - hoping to do it next week. Here are the redacted debug logs: 

2023-04-07 18:53:12,995 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2023-04-07 18:53:12,996 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2023-04-07 18:53:12,996 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2023-04-07 18:53:12,997 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2023-04-07 18:53:12,997 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2023-04-07 18:53:12,998 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2023-04-07 18:53:12,998 botocore.hooks [DEBUG] Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section
2023-04-07 18:53:12,999 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2023-04-07 18:53:12,999 botocore.hooks [DEBUG] Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section
2023-04-07 18:53:12,999 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2023-04-07 18:53:13,000 botocore.hooks [DEBUG] Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section
2023-04-07 18:53:13,002 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2023-04-07 18:53:13,002 botocore.credentials [DEBUG] Looking for credentials via: env
2023-04-07 18:53:13,002 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2023-04-07 18:53:13,002 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2023-04-07 18:53:13,002 botocore.credentials [DEBUG] Looking for credentials via: sso
2023-04-07 18:53:13,002 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2023-04-07 18:53:13,003 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2023-04-07 18:53:13,003 botocore.credentials [DEBUG] Looking for credentials via: config-file
2023-04-07 18:53:13,003 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2023-04-07 18:53:13,003 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2023-04-07 18:53:13,003 botocore.credentials [DEBUG] Looking for credentials via: container-role
2023-04-07 18:53:13,003 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2023-04-07 18:53:13,003 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80
2023-04-07 18:53:13,033 urllib3.connectionpool [DEBUG] http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 200 56
2023-04-07 18:53:13,034 urllib3.connectionpool [DEBUG] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 15
2023-04-07 18:53:13,035 urllib3.connectionpool [DEBUG] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/role-needed HTTP/1.1" 200 566
2023-04-07 18:53:13,036 botocore.credentials [INFO] Found credentials from IAM Role: role-needed
2023-04-07 18:53:13,038 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/endpoints.json
2023-04-07 18:53:13,045 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2023-04-07 18:53:13,045 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0xffff8b93d6c0>
2023-04-07 18:53:13,059 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/kinesis/2013-12-02/service-2.json
2023-04-07 18:53:13,068 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/kinesis/2013-12-02/endpoint-rule-set-1.json.gz
2023-04-07 18:53:13,068 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/partitions.json
2023-04-07 18:53:13,069 botocore.hooks [DEBUG] Event creating-client-class.kinesis: calling handler <function add_generate_presigned_url at 0xffff8b8bd800>
2023-04-07 18:53:13,070 botocore.endpoint [DEBUG] Setting kinesis timeout as (60, 60)
2023-04-07 18:53:13,071 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.11/site-packages/botocore/data/_retry.json
2023-04-07 18:53:13,071 botocore.client [DEBUG] Registering retry handlers for service: kinesis
2023-04-07 18:53:13,072 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False, 'StreamARN': 'arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream', 'OperationType': 'control'}
2023-04-07 18:53:13,072 botocore.regions [DEBUG] Endpoint provider result: https://1111111111111.control-kinesis.us-east-1.amazonaws.com
2023-04-07 18:53:13,072 botocore.hooks [DEBUG] Event before-parameter-build.kinesis.ListStreamConsumers: calling handler <function generate_idempotent_uuid at 0xffff8b993ba0>
2023-04-07 18:53:13,072 botocore.hooks [DEBUG] Event before-call.kinesis.ListStreamConsumers: calling handler <function add_recursion_detection_header at 0xffff8b9923e0>
2023-04-07 18:53:13,072 botocore.hooks [DEBUG] Event before-call.kinesis.ListStreamConsumers: calling handler <function inject_api_version_header_if_needed at 0xffff8b7816c0>
2023-04-07 18:53:13,072 botocore.endpoint [DEBUG] Making request for OperationModel(name=ListStreamConsumers) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'Kinesis_20131202.ListStreamConsumers', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.26.47 Python/3.11.1 Linux/5.10.124-linuxkit Botocore/1.29.108'}, 'body': b'{"StreamARN": "arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream"}', 'url': 'https://1111111111111.control-kinesis.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0xffff8b5b3450>, 'has_streaming_input': False, 'auth_type': None}}
2023-04-07 18:53:13,072 botocore.hooks [DEBUG] Event request-created.kinesis.ListStreamConsumers: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0xffff8b5b3410>>
2023-04-07 18:53:13,072 botocore.hooks [DEBUG] Event choose-signer.kinesis.ListStreamConsumers: calling handler <function set_operation_specific_signer at 0xffff8b993a60>
2023-04-07 18:53:13,073 botocore.auth [DEBUG] Calculating signature using v4 auth.
2023-04-07 18:53:13,073 botocore.auth [DEBUG] CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.1
host:1111111111111.control-kinesis.us-east-1.amazonaws.com
x-amz-date:20230407T185313Z
x-amz-security-token:-
x-amz-target:Kinesis_20131202.ListStreamConsumers
content-type;host;x-amz-date;x-amz-security-token;x-amz-target
-
2023-04-07 18:53:13,073 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20230407T185313Z
20230407/us-east-1/kinesis/aws4_request
eef62f8009d4831a8190f7ee16cd58c1bafc83fc1d28f5eb22a0dbb1a591ce5c
2023-04-07 18:53:13,073 botocore.auth [DEBUG] Signature:
269b3585ae64c8e37a3bf4a58aa56b79ebc120a2c8acec2655fb3fed64d13214
2023-04-07 18:53:13,073 botocore.hooks [DEBUG] Event request-created.kinesis.ListStreamConsumers: calling handler <function add_retry_headers at 0xffff8b781e40>
2023-04-07 18:53:13,073 botocore.endpoint [DEBUG] Sending 2023-04-07 18:53:13,073 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://1111111111111.control-kinesis.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'Kinesis_20131202.ListStreamConsumers', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.26.47 Python/3.11.1 Linux/5.10.124-linuxkit Botocore/1.29.108', 'X-Amz-Date': b'20230407T185313Z', 'X-Amz-Security-Token': b'-', 'Authorization': b'AWS4-HMAC-SHA256 Credential=-/us-east-1/kinesis/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=-', 'amz-sdk-invocation-id': b'-', 'amz-sdk-request': b'attempt=1', 'Content-Length': '75'}>
2023-04-07 18:53:13,074 botocore.httpsession [DEBUG] Certificate path: /usr/local/lib/python3.11/site-packages/certifi/cacert.pem
2023-04-07 18:53:13,075 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): 1111111111111.control-kinesis.us-east-1.amazonaws.com:443
2023-04-07 18:53:13,464 urllib3.connectionpool [DEBUG] https://1111111111111.control-kinesis.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 200 3491
2023-04-07 18:53:13,466 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'cf61e2d8-3762-3f3b-93cb-8a01ce9009d6', 'x-amz-id-2': 'TxP5Ee4+vl2RwSxfJuxZ2UghzZmr44B5Hc1kKADQrhsXtW60+oEcytIH4DWsEMMbmh4AovtCz5zzOEUX/sSfrkbtWYI4n8M2yN34BIWw+d0=', 'Date': 'Fri, 07 Apr 2023 18:53:13 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '3491'}2023-04-07 18:53:13,466 botocore.parsers [DEBUG] Response body:
b'{"Consumers":[{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-local-main:1677176556","ConsumerCreationTimestamp":1.677176556E9,"ConsumerName":"funky-local-main","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-local-read_local_file:1678312350","ConsumerCreationTimestamp":1.67831235E9,"ConsumerName":"funky-local-read_local_file","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-local-remove_java:1680042431","ConsumerCreationTimestamp":1.680042431E9,"ConsumerName":"funky-local-remove_java","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-local-streaming-job-bootstrap:1676000620","ConsumerCreationTimestamp":1.67600062E9,"ConsumerName":"funky-local-streaming-job-bootstrap","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-change_efo_consumer:1680635680","ConsumerCreationTimestamp":1.68063568E9,"ConsumerName":"funky-test-change_efo_consumer","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-cleanup_terraform:1680655375","ConsumerCreationTimestamp":1.680655375E9,"ConsumerName":"funky-test-cleanup_terraform","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-learning-test:1670905657","ConsumerCreationTimestamp":1.670905657E9,"ConsumerName":"funky-test-learning-test","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-learning_tests:1676413966","ConsumerCreationTimestamp":1.676413966E9,"ConsumerName":"funky-test-learning_tests","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-main:1679691482","ConsumerCreationTimestamp":1.679691482E9,"ConsumerName":"funky-test-main","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-multi-fixes:1675975397","ConsumerCreationTimestamp":1.675975397E9,"ConsumerName":"funky-test-multi-fixes","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-remove_efo_consumers:1680883075","ConsumerCreationTimestamp":1.680883075E9,"ConsumerName":"funky-test-remove_efo_consumers","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/funky-test-remove_java:1680711591","ConsumerCreationTimestamp":1.680711591E9,"ConsumerName":"funky-test-remove_java","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/pxl:1675817565","ConsumerCreationTimestamp":1.675817565E9,"ConsumerName":"pxl","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/our-consumer-prod:1680626484099","ConsumerCreationTimestamp":1.680626484099E12,"ConsumerName":"our-consumer-prod","ConsumerStatus":"ACTIVE"},{"ConsumerARN":"arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream/consumer/our-consumer:1680840854","ConsumerCreationTimestamp":1.680840854E9,"ConsumerName":"our-consumer","ConsumerStatus":"ACTIVE"}]}'
Traceback (most recent call last):
 File "/home/prod/bin/nights_watch/latest/nights_watch/monitors/test.py", line 7, in <module>
  client.list_stream_consumers(StreamARN='arn:aws:kinesis:us-east-1:1111111111111:stream/our-stream')
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call
  return self._make_api_call(operation_name, kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 943, in _make_api_call
    http, parsed_response = self._make_request(
              ^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 966, in _make_request
  return self._endpoint.make_request(operation_model, request_dict)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 119, in make_request
  return self._send_request(request_dict, operation_model)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 199, in _send_request
  success_response, exception = self._get_response(
                 ^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 241, in _get_response
  success_response, exception = self._do_get_response(
                 ^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 308, in _do_get_response
  parsed_response = parser.parse(
           ^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 252, in parse
  parsed = self._do_parse(response, shape)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 849, in _do_parse
  parsed = self._handle_json_body(response['body'], shape)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 873, in _handle_json_body
  return self._parse_shape(shape, parsed_json)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
  return handler(shape, node)
      ^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 663, in _handle_structure
  final_parsed[member_name] = self._parse_shape(
                ^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
  return handler(shape, node)
      ^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 340, in _handle_list
  parsed.append(self._parse_shape(member_shape, item))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
  return handler(shape, node)
      ^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 663, in _handle_structure
  final_parsed[member_name] = self._parse_shape(
                ^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 332, in _parse_shape
  return handler(shape, node)
      ^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/parsers.py", line 682, in _handle_timestamp
  return self._timestamp_parser(value)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 941, in parse_timestamp
  return _parse_timestamp_with_tzinfo(value, tzinfo)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 912, in _parse_timestamp_with_tzinfo
  return datetime.datetime.fromtimestamp(value, tzinfo())
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: year 55226 is out of range

Thanks for your help.

alexeiverbny commented 1 year ago

I tried using AWSCLI for this and ran into the same issue because AWSCLI uses botocore. I have opened up a support ticket with AWS.

RyanFitzSimmonsAK commented 1 year ago

@alexeiverbny, does anything look wrong with the date when you look at it in the console?

alexeiverbny commented 1 year ago

@RyanFitzSimmonsAK nope, gives a reasonable and accurate date.

RyanFitzSimmonsAK commented 1 year ago

Did you get a reply from support re : the milliseconds in the response?

alexeiverbny commented 1 year ago

Hi Ryan,

Aws is investigating. I am out of office for two weeks but can share an update when I return.

Thanks, Alex

On Tue, May 9, 2023, 10:55 PM Ryan F. @.***> wrote:

Did you get a reply from support re : the milliseconds in the response?

— Reply to this email directly, view it on GitHub https://github.com/boto/botocore/issues/2902#issuecomment-1540877861, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM4KVUAWEO7QOKQURFOEXRTXFKVLXANCNFSM6AAAAAAWUQBQY4 . You are receiving this because you were mentioned.Message ID: @.***>

alexeiverbny commented 1 year ago

Hi Ryan,

Apparently there was some bug when this stream was created. The created timestamp is in fact in the year 55K :). I think this issue can be closed - it is certainly not a botocore issue. Appreciate your help.

Thanks, Alex

RyanFitzSimmonsAK commented 1 year ago

Will do, thanks for getting back to me!