Azure / azure-cli

Azure Command-Line Interface
MIT License
3.98k stars 2.96k forks source link

{testsdk} No YAML recording file is generated if there is no record-able requests #21427

Open jiasli opened 2 years ago

jiasli commented 2 years ago

Tests such as test_create_account_sas don't contain record-able requests:

https://github.com/Azure/azure-cli/blob/30216d0268f3491b8af2aa060d90a9e1f94f9b3c/src/azure-cli/azure/cli/command_modules/storage/tests/latest/test_storage_account_scenarios.py#L779-L797

ResourceGroupPreparer, StorageAccountPreparer's requests won't be recorded. In such case, running this test in live mode will delete the current (handmade) YAML and not generate a new YAML.

If this test is then run in playback mode, test framework detects there is no YAML recording file and will automatically go to live mode and cause failure in CI (https://github.com/Azure/azure-cli/pull/21403):

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1388110&view=logs&jobId=4d9a7583-3a39-5165-718c-04fb813e465f&j=4d9a7583-3a39-5165-718c-04fb813e465f&t=6c30694c-16d1-5996-0668-2b6e3a9f9dd5

E           ValueError: Unable to get authority configuration for https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000. Authority would typically be in a format of https://login.microsoftonline.com/your_tenant_name

env/lib/python3.10/site-packages/msal/authority.py:87: ValueError
- generated xml file: /home/vsts/.azdev/env_config/home/vsts/work/1/s/env/test_results.xml -
=========================== short test summary info ============================
FAILED src/azure-cli/azure/cli/command_modules/storage/tests/latest/test_storage_account_scenarios.py::StorageAccountTests::test_create_account_sas

It is the design of vcrpy that no YAML recording file is generated if there is no record-able requests. vcrpy does have an option to force to create the YAML:

https://github.com/kevin1024/vcrpy/blob/c79a06f639dd628536c9868044e78df1012985f9/vcr/cassette.py#L329-L332

    def _save(self, force=False):
        if force or self.dirty:
            self._persister.save_cassette(self._path, self._as_dict(), serializer=self._serializer)
            self.dirty = False

When force is True, this will create a YAML with empty interactions:

interactions: []
version: 1

But this force parameter is not surfaced as a public API.

yonzhan commented 2 years ago

testsdk

beneshed commented 2 years ago

What way is there to know if the request is non-recordable?

We have a test that requires AllowLargeResponse, but whenever the tests are run, the recording is deleted

@diklatze