TestSDK utilities.py fails to convert byte to string for test recording

[Hazhzeng]

az feedback auto-generates most of the information requested below, as of CLI version

2.0.73

Describe the bug When implementing an E2E test scenario to upload a zip file to a storage account (using StorageAccountPreparer), the _py3_byte_to_str() function inside testsdk/utilities.py fails to convert byte to string.

This affects the recording in env\lib\site-packages\azure_devtools\scenario_tests\base.py:157 if the test is decorated by StorageAccountPreparer and the test-covered code contains request to upload zip file.

To Reproduce

1. Paste the following e2e test scenario in azure-cli\src\azure-cli\azure\cli\command_modules\appservice\tests\latest\test_webapp_commands.py

   class FunctionappRemoteBuildScenarioTest(ScenarioTest):
   @ResourceGroupPreparer(name_prefix='azurecli-functionapp-c-e2e-remotebuild')
   @StorageAccountPreparer(sku='Standard_LRS')
   def test_functionapp_remote_build(self, resource_group, storage_account):
       functionapp_name = self.create_random_name('functionapp-remotebuild-test', 40)
       plan_name = self.create_random_name('functionapp-remotebuild-plan', 40)
       zip_file = os.path.join(TEST_DIR, 'test.zip')
       self.cmd('functionapp plan create -g {} -n {} --sku S1 --is-linux true'.format(resource_group, plan_name))
       self.cmd('functionapp create -g {} -n {} --plan {} -s {} --os-type Linux --runtime python'.format(resource_group, functionapp_name, plan_name, storage_account))
       self.cmd('functionapp deployment source config-zip -g {} -n {} --src "{}"'.format(resource_group, functionapp_name, zip_file)).assert_with_checks([
           JMESPathCheck('status', 4),
           JMESPathCheck('deployer', 'Push-Deployer'),
           JMESPathCheck('message', 'Created via a push deployment'),
           JMESPathCheck('complete', True)
       ])

2. Activate your virtual environment, setup the development environment.
3. Run azdev test test_functionapp_remote_build
4. The e2e test fails with the following traceback (in appendix).

Expected behavior

1. The test should pass without warning
2. A new recording entry should be generated.

Environment summary Install Method (e.g. pip, interactive script, apt-get, Docker, MSI, edge build) / CLI version (az --version) / OS version / Shell Type (e.g. bash, cmd.exe, Bash on Windows)

• Setup by the azdev

Appendix (Error Traceback)

(env) C:\Users\hazeng\Projects\GitHub\azure-cli [hazeng-pr4-remotebuild +3 ~4 -1 !]>  azdev test test_functionapp_remote_build

=============
| Run Tests |
=============

The tests are set to run against current profile latest.

test index found: C:\Users\hazeng\.azdev\env_config\Users\hazeng\Projects\Github\azure-cli\env\test_index\latest.json

TESTS: test_functionapp_remote_build

============================================================================================================================ test session starts =============================================================================================================================
platform win32 -- Python 3.6.8, pytest-4.4.2, py-1.8.0, pluggy-0.13.0
rootdir: c:\users\hazeng\projects\github\azure-cli\src\azure-cli
plugins: forked-1.0.2, xdist-1.29.0
gw0 [1] / gw1 [1] / gw2 [1] / gw3 [1] / gw4 [1] / gw5 [1] / gw6 [1] / gw7 [1]
F                                                                                                                                                                                                                                                                       [100%]
================================================================================================================================== FAILURES ==================================================================================================================================
______________________________________________________________________________________________________ FunctionappRemoteBuildScenarioTest.test_functionapp_remote_build ______________________________________________________________________________________________________
[gw0] win32 -- Python 3.6.8 C:\Users\hazeng\Projects\Github\azure-cli\env\Scripts\python.exe

self = <azure.cli.command_modules.appservice.tests.latest.test_webapp_commands.FunctionappRemoteBuildScenarioTest testMethod=test_functionapp_remote_build>, resource_group = 'azurecli-functionapp-c-e2e-remotebuildqryg4sju63qtegcbb3z7nxxj67s376644cjln'
storage_account = 'clitestwditlbpoox6lbg4dh'

    @ResourceGroupPreparer(name_prefix='azurecli-functionapp-c-e2e-remotebuild', location='westus')
    @StorageAccountPreparer(sku='Standard_LRS')
    def test_functionapp_remote_build(self, resource_group, storage_account):
        plan_name = self.create_random_name('functionapp-remotebuild-plan', 40)
        functionapp_name = self.create_random_name('functionappconsumption', 40)
        zip_file = os.path.join(TEST_DIR, 'test.zip')
        self.cmd('functionapp plan create -g {} -n {} --sku S1 --is-linux true'.format(resource_group, plan_name))
        self.cmd('functionapp create -g {} -n {} -p {} -s {} --os-type Linux --runtime python'
                 .format(resource_group, functionapp_name, plan_name, storage_account))
        self.cmd('webapp deployment source config-zip -g {} -n {} --src "{}"'
>                .format(resource_group, functionapp_name, zip_file))

src\azure-cli\azure\cli\command_modules\appservice\tests\latest\test_webapp_commands.py:1431:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src\azure-cli-testsdk\azure\cli\testsdk\base.py:161: in cmd
    return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
src\azure-cli-testsdk\azure\cli\testsdk\base.py:201: in __init__
    self._in_process_execute(cli_ctx, command, expect_failure=expect_failure)
src\azure-cli-testsdk\azure\cli\testsdk\base.py:264: in _in_process_execute
    raise ex.exception
env\lib\site-packages\knack\cli.py:206: in invoke
    cmd_result = self.invocation.execute(args)
src\azure-cli-core\azure\cli\core\commands\__init__.py:603: in execute
    raise ex
src\azure-cli-core\azure\cli\core\commands\__init__.py:661: in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
src\azure-cli-core\azure\cli\core\commands\__init__.py:654: in _run_job
env\lib\site-packages\six.py:693: in reraise
    raise value
src\azure-cli-core\azure\cli\core\commands\__init__.py:631: in _run_job
    result = cmd_copy(params)
src\azure-cli-core\azure\cli\core\commands\__init__.py:306: in __call__
    return self.handler(*args, **kwargs)
src\azure-cli-core\azure\cli\core\__init__.py:485: in default_command_handler
    return op(**command_args)
src\azure-cli\azure\cli\command_modules\appservice\custom.py:364: in enable_zip_deploy
    res = requests.post(zip_url, data=zip_content, headers=headers, verify=not should_disable_connection_verify())
env\lib\site-packages\requests\api.py:116: in post
    return request('post', url, data=data, json=json, **kwargs)
env\lib\site-packages\requests\api.py:60: in request
    return session.request(method=method, url=url, **kwargs)
env\lib\site-packages\requests\sessions.py:533: in request
    resp = self.send(prep, **send_kwargs)
env\lib\site-packages\requests\sessions.py:646: in send
    r = adapter.send(request, **kwargs)
env\lib\site-packages\requests\adapters.py:449: in send
    timeout=timeout
env\lib\site-packages\urllib3\connectionpool.py:600: in urlopen
    chunked=chunked)
env\lib\site-packages\urllib3\connectionpool.py:377: in _make_request
    httplib_response = conn.getresponse(buffering=True)
env\lib\site-packages\vcr\stubs\__init__.py:220: in getresponse
    if self.cassette.can_play_response_for(self._vcr_request):
env\lib\site-packages\vcr\cassette.py:254: in can_play_response_for
    request = self._before_record_request(request)
env\lib\site-packages\vcr\config.py:253: in before_record_request
    request = function(request)
env\lib\site-packages\azure_devtools\scenario_tests\base.py:157: in _process_request_recording
    request = processor.process_request(request)
src\azure-cli-testsdk\azure\cli\testsdk\utilities.py:85: in process_request
    body_string = _py3_byte_to_str(request.body)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

byte_or_str = b'PK\x03\x04\x14\x00\x00\x00\x08\x00\x04dQK\xc1\x98\x14\x97S\x00\x00\x00p\x00\x00\x00\t\x00\x00\x00test.html\xb3Qt\xf1...\x00 \x00\x00\x00\x00\x00\x00\x00test.htmlPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x007\x00\x00\x00z\x00\x00\x00\x00\x00'

    def _py3_byte_to_str(byte_or_str):
        import logging
        logger = logging.getLogger()
        logger.warning(type(byte_or_str))
        try:
>           return str(byte_or_str, 'utf-8') if isinstance(byte_or_str, bytes) else byte_or_str
E           UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 14: invalid start byte

src\azure-cli-testsdk\azure\cli\testsdk\utilities.py:57: UnicodeDecodeError
---------------------------------------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------------------------------------
WARNING: The default kind for created storage account will change to 'StorageV2' from 'Storage' in future
WARNING: Application Insights "functionappconsumptionnqp6srbglvy5vq5eat" was created for this Function App. You can visit https://portal.azure.com/#resource/subscriptions/fdaf2e05-dba9-4a35-8130-41d1a855a739/resourceGroups/azurecli-functionapp-c-e2e-remotebuildqryg4sju63qtegcbb3z7nxxj67s376644cjln/providers/microsoft.insights/components/functionappconsumptionnqp6srbglvy5vq5eat/overview to view your Application Insights component
WARNING: Getting scm site credentials for zip deployment
WARNING: Starting zip deployment. This operation can take a while to complete ...
----------------------------------------------------------------------------- generated xml file: C:\Users\hazeng\.azdev\env_config\Users\hazeng\Projects\Github\azure-cli\env\test_results.xml ------------------------------------------------------------------------------
========================================================================================================================= 1 failed in 72.77 seconds ==========================================================================================================================

 Results
=========

Time: 72.699 sec        Tests: 1        Skipped: None   Failures: 1     Errors: 0

 FAILURES
==========

test_webapp_commands.FunctionappRemoteBuildScenarioTest.test_functionapp_remote_build

[Hazhzeng]

This issue is related to https://github.com/Azure/azure-cli/pull/9784#issuecomment-505541134 cc: @ankitkumarr

[fengzhou-msft]

@Hazhzeng Thanks for reporting the issue. I will look into details next week after holidays.

[qwordy]

Thanks for your feedback.

[haroldrandom]

Sorry for the late reply, we will take a look after Ignite.

[yonzhan]

@bim-msft add to S161.

[bim-msft]

@Hazhzeng Sorry for the late reply, I found the reason by debugging into the test code. This happens when you decoding an invalid bytes into UTF-8 format string. The relevant code is here: https://github.com/Azure/azure-cli/blob/ea5496bf7ee188da599dd3b009fc63d24967154a/src/azure-cli/azure/cli/command_modules/appservice/custom.py#L373-L378 This zip-content in line 377 can not be decoded into a valid UTF-8 string.

https://github.com/Azure/azure-cli/blob/ea5496bf7ee188da599dd3b009fc63d24967154a/src/azure-cli-testsdk/azure/cli/testsdk/utilities.py#L52-L59 Finally the call str(byte_or_str, 'utf-8') in line 57 throws the above described exception.

I have opened a PR to ignore this type of errors in decoding process, and I also uncommented your test function code in this PR, please see #11251 .

[bim-msft]

Ignoring the error is not necessary. I changed the test type to LiveScenarioTest instead.

[Hazhzeng]

@bim-msft thanks for the response, if you need any update from my side, please do let me know :)

[bim-msft]

@Hazhzeng Hi, I closed the PR so I didn't change anything. If you want to make this test runnable, then you should change ScenarioTest to LiveScenarioTest for this test case, which can bypass the recording of test framework. But currently the github CI will skip all live tests, you can just test it locally (set AZURE_TEST_RUN_LIVE=true). We plan to support binary recoding for this kind of cases in the future :-)