search

aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.09k stars 4.01k forks source link

Tests of TestCLITimestampParser failing, maybe due to DST #8644

Open Asday opened 2 months ago

Asday commented 2 months ago

Describe the bug

Currently when running tests in England, two of them fail, expecting two timestamps to equal when they are in fact exactly one hour apart.

Expected Behavior

All tests should pass, xfail, or be skipped.

Current Behavior

Tests fail with the following output:

=========================================================================================== FAILURES ===========================================================================================
_____________________________________________________________________________ TestCLITimestampParser.test_default ______________________________________________________________________________
[gw0] linux -- Python 3.11.8 /home/asday/.cache/yay/aws-cli-v2/src/awscli-2.15.40/venv/bin/python

self = <tests.functional.test_timeformat.TestCLITimestampParser testMethod=test_default>

    def test_default(self):
        self.driver = create_clidriver()
        self.entry_point = AWSCLIEntryPoint(self.driver)
        expected_time = datetime.datetime.fromtimestamp(0).replace(
            tzinfo=tzlocal()).isoformat()

        stdout, _, _ = self.run_cmd(self.command)
        json_response = json.loads(stdout)
        start_time = json_response["builds"][0]["startTime"]
>       self.assertEqual(expected_time, start_time)
E       AssertionError: '1970-01-01T01:00:00+00:00' != '1970-01-01T00:00:00+00:00'
E       - 1970-01-01T01:00:00+00:00
E       ?             ^
E       + 1970-01-01T00:00:00+00:00
E       ?             ^

tests/functional/test_timeformat.py:72: AssertionError
-------------------------------------------------------------------------------------- Captured log call ---------------------------------------------------------------------------------------
DEBUG    awscli.clidriver:clidriver.py:864 OrderedDict([('ids', <awscli.arguments.ListArgument object at 0x76749a5c3c90>)])
DEBUG    awscli.arguments:arguments.py:454 Unpacked value of ['foo'] for parameter "ids": ['foo']
_______________________________________________________________________________ TestCLITimestampParser.test_iso ________________________________________________________________________________
[gw0] linux -- Python 3.11.8 /home/asday/.cache/yay/aws-cli-v2/src/awscli-2.15.40/venv/bin/python

self = <tests.functional.test_timeformat.TestCLITimestampParser testMethod=test_iso>

    def test_iso(self):
        self.environ['AWS_CONFIG_FILE'] = self.files.create_file(
            'iso',
            '[default]\ncli_timestamp_format = iso8601\n')
        self.driver = create_clidriver()
        self.entry_point = AWSCLIEntryPoint(self.driver)
        expected_time = datetime.datetime.fromtimestamp(0).replace(
            tzinfo=tzlocal()).isoformat()

        stdout, _, _ = self.run_cmd(self.command)
        json_response = json.loads(stdout)
        start_time = json_response["builds"][0]["startTime"]
>       self.assertEqual(expected_time, start_time)
E       AssertionError: '1970-01-01T01:00:00+00:00' != '1970-01-01T00:00:00+00:00'
E       - 1970-01-01T01:00:00+00:00
E       ?             ^
E       + 1970-01-01T00:00:00+00:00
E       ?             ^

tests/functional/test_timeformat.py:48: AssertionError
-------------------------------------------------------------------------------------- Captured log call ---------------------------------------------------------------------------------------
DEBUG    awscli.clidriver:clidriver.py:864 OrderedDict([('ids', <awscli.arguments.ListArgument object at 0x7674910e24d0>)])
DEBUG    awscli.arguments:arguments.py:454 Unpacked value of ['foo'] for parameter "ids": ['foo']
=================================================================================== short test summary info ====================================================================================
FAILED tests/functional/test_timeformat.py::TestCLITimestampParser::test_default - AssertionError: '1970-01-01T01:00:00+00:00' != '1970-01-01T00:00:00+00:00'
FAILED tests/functional/test_timeformat.py::TestCLITimestampParser::test_iso - AssertionError: '1970-01-01T01:00:00+00:00' != '1970-01-01T00:00:00+00:00'
==================================================================== 2 failed, 68110 passed, 7 skipped in 453.25s (0:07:33) ====================================================================

Reproduction Steps

As mentioned, the timestamps are off by one hour, and I am currently in DST (against my will), so set your timezone appropriately. For reference, the current output of date is Mon 22 Apr 00:05:23 BST 2024.

The commands run to get here are documented in the AUR PKGBUILD for aws-cli-v2, important part reproduced below:

  # Avoid intermittent test failures, see git commit messages
  ulimit -S -n 4096

  cd awscli-$pkgver

  export AWS_SECRET_ACCESS_KEY=fake_key
  export AWS_ACCESS_KEY_ID=fake_id

  export PYTHONPATH="$PWD"

  # Install a temporary copy to a virtual environment, as tests/dependencies checks global site-packages
  python -m venv --system-site-packages "$PWD/venv"
  "$PWD/venv/bin/python" -m installer dist/*.whl

  # * Use --dist=loadfile following upstream. The default --dist=load may cause test failures and is not faster
  # * Disable backend tests - those tests check if aws-cli can be installed or not, and are not compatible with all kinds of environments
  "$PWD/venv/bin/python" -m pytest tests -n auto --dist loadfile --ignore=tests/backends --ignore=tests/integration

It appears in a development environment, this would be reproducible with:

AWS_SECRET_ACCESS_KEY=fake_key AWS_ACCESS_KEY_ID=fake_id python -m pytest tests -n auto --dist loadfile --ignore=tests/backends --ignore=tests/integration

Possible Solution

I haven't investigated, it's late and I hit this right as I was updating my system before bed. Generally I throw freezegun at these sorts of tests so they fail all the time instead of only for six months of the year, and deal with everything in UTC as much as possible.

Additional Information/Context

No response

CLI version used

2.15.40

Environment details (OS name and version, etc.)

Arch Linux (by the way), Python 3.11.8, yay v12.3.5, pacman v6.1.0, libalpm v14.0.0.

oniko94 commented 2 months ago

Can confirm AUR package aws-cli-v2 build failing with the same error. The relevant build log fragment (extracted stack trace and error messages):

[gw1] linux -- Python 3.11.8 /home/oniko/.cache/yay/aws-cli-v2/src/awscli-2.15.40/venv/bin/python

self = <tests.functional.test_timeformat.TestCLITimestampParser testMethod=test_default>

    def test_default(self):
        self.driver = create_clidriver()
        self.entry_point = AWSCLIEntryPoint(self.driver)
        expected_time = datetime.datetime.fromtimestamp(0).replace(
            tzinfo=tzlocal()).isoformat()

        stdout, _, _ = self.run_cmd(self.command)
        json_response = json.loads(stdout)
        start_time = json_response["builds"][0]["startTime"]
>       self.assertEqual(expected_time, start_time)
E       AssertionError: '1970-01-01T03:00:00+02:00' != '1970-01-01T02:00:00+02:00'
E       - 1970-01-01T03:00:00+02:00
E       ?             ^
E       + 1970-01-01T02:00:00+02:00
E       ?             ^

tests/functional/test_timeformat.py:72: AssertionError
-------------------------------------------------------
[gw1] linux -- Python 3.11.8 /home/oniko/.cache/yay/aws-cli-v2/src/awscli-2.15.40/venv/bin/python

self = <tests.functional.test_timeformat.TestCLITimestampParser testMethod=test_iso>

    def test_iso(self):
        self.environ['AWS_CONFIG_FILE'] = self.files.create_file(
            'iso',
            '[default]\ncli_timestamp_format = iso8601\n')
        self.driver = create_clidriver()
        self.entry_point = AWSCLIEntryPoint(self.driver)
        expected_time = datetime.datetime.fromtimestamp(0).replace(
            tzinfo=tzlocal()).isoformat()

        stdout, _, _ = self.run_cmd(self.command)
        json_response = json.loads(stdout)
        start_time = json_response["builds"][0]["startTime"]
>       self.assertEqual(expected_time, start_time)
E       AssertionError: '1970-01-01T03:00:00+02:00' != '1970-01-01T02:00:00+02:00'
E       - 1970-01-01T03:00:00+02:00
E       ?             ^
E       + 1970-01-01T02:00:00+02:00
E       ?             ^

tests/functional/test_timeformat.py:48: AssertionError
---------------------------------------------------------------
FAILED tests/functional/test_timeformat.py::TestCLITimestampParser::test_default - AssertionError: '1970-01-01T03:00:00+02:00' != '1970-01-01T02:00:00+02:00'
FAILED tests/functional/test_timeformat.py::TestCLITimestampParser::test_iso - AssertionError: '1970-01-01T03:00:00+02:00' != '1970-01-01T02:00:00+02:00'
j3m7 commented 2 months ago

Fellow arch user here, I ran into a similar error and noticed some of my own config on the failure error messages. I tried un-setting the AWS_PROFILE and AWS_REGION env vars and re-running the build. Worked like a charm.

Asday commented 2 months ago

@j3m7 thank you for investigating - neither of those are set in my environment.

tim-finnigan commented 2 months ago

Thanks for reporting this issue. I brought this up for discussion with the team, and the consensus is that more investigation is required into these tests. The tests were written a long time ago and likely need to be rewritten. If anyone has any other details to share, please feel free to add those here.

izbushka commented 2 months ago

As a workaround temporary setting timezone to one without DST helps (e.g. Asia/Baghdad)

amberkushwaha commented 1 month ago

def test_iso(self): self.environ['AWS_CONFIG_FILE'] = self.files.create_file( 'iso', '[default]\ncli_timestamp_format = iso8601\n') self.driver = create_clidriver() self.entry_point = AWSCLIEntryPoint(self.driver) expected_time = datetime.datetime.fromtimestamp(0).replace( tzinfo=tzlocal()).isoformat()

    stdout, _, _ = self.run_cmd(self.command)
    json_response = json.loads(stdout)
    start_time = json_response["builds"][0]["startTime"]
  self.assertEqual(expected_time, start_time)

E AssertionError: '1970-01-01T01:00:00+00:00' != '1970-01-01T00:00:00+00:00' E - 1970-01-01T01:00:00+00:00 E ? ^ E + 1970-01-01T00:00:00+00:00 E ? ^ the coding in the file format is different in linux and java encryption both so to smooth out the basics we have code it again in python format and

linux duer module.