Azure / azure-cli

Azure Command-Line Interface
MIT License
3.99k stars 2.97k forks source link

Assertion message from JMESPathCheck is misleading when actual result evaluates to False. #29697

Open tobiasb-ms opened 2 months ago

tobiasb-ms commented 2 months ago

Describe the bug

When running scenario tests for an extension I'm writing, with a check like this:

self.cmd(f"az some_command --output json", checks=[
    self.check("thing", True)
])

The exception when the actual value of thing is False is misleading:

>           raise JMESPathCheckAssertionError(self._query, self._expected_result, 'None',
                                              execution_result.output)
E           azure.cli.testsdk.exceptions.JMESPathCheckAssertionError: Query 'thing' doesn't yield expected value 'True', instead the actual value is 'None'. Data:
E           {
E             "otherThing": "value",
E             "thing": false
E           }

It would be better in this case to say "the actual value is False".

We encounter similar issues with empty strings, empty lists, empty dictionaries, the number 0 and generally anything that evaluates to False.

I've created a pull request with more information about why this happens and a proposed fix.

Related command

azdev test

Errors

      raise JMESPathCheckAssertionError(self._query, self._expected_result, 'None',

execution_result.output) E azure.cli.testsdk.exceptions.JMESPathCheckAssertionError: Query 'thing' doesn't yield expected value 'True', instead the actual value is 'None'. Data: E { E "otherThing": "value", E "thing": false E }

Issue script & Debug output

I feel this is unnecessary because the error message makes it clear (and see linked pull request).

Expected behavior

The error message should say the real value of the actual result -- In the provided example, that would be 'False'.

Environment Summary

az --version azure-cli 2.63.0

core 2.63.0 telemetry 1.1.0

Extensions: aks-preview 7.0.0b3 azure-devops 1.0.1

0.0.1 (dev) /workspace/code///src/ Dependencies: msal 1.30.0 azure-mgmt-resource 23.1.1 Python location '/workspace/code///.venv/bin/python' Extensions directory '/home//.azure/cliextensions' Development extension sources: /workspace/code// Python (Linux) 3.9.19 (main, Jul 31 2024, 03:47:41) [GCC 11.2.0] Legal docs and information: aka.ms/AzureCliLegal Your CLI is up-to-date. ### Additional context _No response_
yonzhan commented 2 months ago

Thank you for opening this issue, we will look into it.