search

Azure / azure-cli

Azure Command-Line Interface
MIT License
4.02k stars 2.99k forks source link

Multiple tests fail: [Errno 2] No such file or directory: '/home/cloudtest/.azure/dummy_cli_config_dir/xGYaEIdmTuUx36nO/commandIndex.json' #28848

Open jiasli opened 6 months ago

jiasli commented 6 months ago

https://dev.azure.com/azclitools/public/_build/results?buildId=153071&view=logs&j=f127618a-a658-5219-f5b0-4a245c9227b8&t=60a7c4e7-f35f-5fef-6efb-f1b54b8cb075&l=1659

2024-04-28T07:37:24.0504576Z _______________ HDInsightClusterTests.test_hdinsight_application _______________
2024-04-28T07:37:24.0505013Z [gw0] linux -- Python 3.11.9 /mnt/vss/_work/1/s/env/bin/python
2024-04-28T07:37:24.0505234Z self = <azure.cli.testsdk.base.ExecutionResult object at 0x7fa73199eb50>
2024-04-28T07:37:24.0505467Z cli_ctx = <azure.cli.core.mock.DummyCli object at 0x7fa7324bb8d0>
2024-04-28T07:37:24.0506071Z command = 'hdinsight create -n hdicli-000003 -g hdicli-000001 -l southcentralus -p Password1! -t spark --no-validation-timeout --storage-account hdicli000002 --storage-container default --ssh-user sshuser --ssh-password Password1!'
2024-04-28T07:37:24.0506399Z expect_failure = False
2024-04-28T07:37:24.0506463Z 
2024-04-28T07:37:24.0506626Z     def _in_process_execute(self, cli_ctx, command, expect_failure=False):
2024-04-28T07:37:24.0506823Z         from io import StringIO
2024-04-28T07:37:24.0507021Z         from vcr.errors import CannotOverwriteExistingCassetteException
2024-04-28T07:37:24.0507201Z     
2024-04-28T07:37:24.0507431Z         if command.startswith('az '):
2024-04-28T07:37:24.0507598Z             command = command[3:]
2024-04-28T07:37:24.0507741Z     
2024-04-28T07:37:24.0507874Z         stdout_buf = StringIO()
2024-04-28T07:37:24.0508041Z         logging_buf = StringIO()
2024-04-28T07:37:24.0508181Z         try:
2024-04-28T07:37:24.0508384Z             # issue: stderr cannot be redirect in this form, as a result some failure information
2024-04-28T07:37:24.0508595Z             # is lost when command fails.
2024-04-28T07:37:24.0508818Z >           self.exit_code = cli_ctx.invoke(shlex.split(command), out_file=stdout_buf) or 0
2024-04-28T07:37:24.0508940Z 
2024-04-28T07:37:24.0509214Z src/azure-cli-testsdk/azure/cli/testsdk/base.py:302: 
2024-04-28T07:37:24.0509440Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-04-28T07:37:24.0509764Z env/lib/python3.11/site-packages/knack/cli.py:245: in invoke
2024-04-28T07:37:24.0509981Z     exit_code = self.exception_handler(ex)
2024-04-28T07:37:24.0510319Z src/azure-cli-core/azure/cli/core/__init__.py:127: in exception_handler
2024-04-28T07:37:24.0510538Z     return handle_exception(ex)
2024-04-28T07:37:24.0510723Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-04-28T07:37:24.0510902Z 
2024-04-28T07:37:24.0511222Z ex = FileNotFoundError(2, 'No such file or directory'), args = (), kwargs = {}
2024-04-28T07:37:24.0511335Z 
2024-04-28T07:37:24.0511661Z     def _handle_main_exception(ex, *args, **kwargs):  # pylint: disable=unused-argument
2024-04-28T07:37:24.0511923Z         if isinstance(ex, CannotOverwriteExistingCassetteException):
2024-04-28T07:37:24.0512169Z             # This exception usually caused by a no match HTTP request. This is a product error
2024-04-28T07:37:24.0512405Z             # that is caused by change of SDK invocation.
2024-04-28T07:37:24.0512566Z             raise ex
2024-04-28T07:37:24.0512694Z     
2024-04-28T07:37:24.0512837Z >       raise CliExecutionError(ex)
2024-04-28T07:37:24.0513116Z E       azure.cli.testsdk.exceptions.CliExecutionError: The CLI throws exception FileNotFoundError during execution and fails the command.
2024-04-28T07:37:24.0513282Z 
2024-04-28T07:37:24.0513594Z src/azure-cli-testsdk/azure/cli/testsdk/patches.py:35: CliExecutionError
2024-04-28T07:37:24.0513726Z 
2024-04-28T07:37:24.0513904Z During handling of the above exception, another exception occurred:
2024-04-28T07:37:24.0514010Z 
2024-04-28T07:37:24.0514271Z self = <azure.cli.command_modules.hdinsight.tests.latest.test_hdinsight_commands.HDInsightClusterTests testMethod=test_hdinsight_application>
2024-04-28T07:37:24.0514707Z storage_account_info = ('hdicli000002', 'veryFakedStorageAccountKey==')
2024-04-28T07:37:24.0514822Z 
2024-04-28T07:37:24.0515165Z     @ResourceGroupPreparer(name_prefix='hdicli-', location=location, random_name_length=12)
2024-04-28T07:37:24.0515599Z     @StorageAccountPreparer(name_prefix='hdicli', location=location, parameter_name='storage_account')
2024-04-28T07:37:24.0515878Z     def test_hdinsight_application(self, storage_account_info):
2024-04-28T07:37:24.0516170Z >       self._create_hdinsight_cluster(
2024-04-28T07:37:24.0516388Z             HDInsightClusterTests._wasb_arguments(storage_account_info),
2024-04-28T07:37:24.0516614Z             HDInsightClusterTests._with_explicit_ssh_creds()
2024-04-28T07:37:24.0516782Z         )
2024-04-28T07:37:24.0516833Z 
2024-04-28T07:37:24.0517189Z src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py:222: 
2024-04-28T07:37:24.0517450Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-04-28T07:37:24.0517881Z src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py:577: in _create_hdinsight_cluster
2024-04-28T07:37:24.0518161Z     self.cmd(create_cluster_format, checks=[
2024-04-28T07:37:24.0518455Z src/azure-cli-testsdk/azure/cli/testsdk/base.py:176: in cmd
2024-04-28T07:37:24.0518720Z     return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
2024-04-28T07:37:24.0519070Z src/azure-cli-testsdk/azure/cli/testsdk/base.py:251: in __init__
2024-04-28T07:37:24.0519311Z     self._in_process_execute(cli_ctx, command, expect_failure=expect_failure)
2024-04-28T07:37:24.0519650Z src/azure-cli-testsdk/azure/cli/testsdk/base.py:314: in _in_process_execute
2024-04-28T07:37:24.0519846Z     raise ex.exception
2024-04-28T07:37:24.0520105Z env/lib/python3.11/site-packages/knack/cli.py:233: in invoke
2024-04-28T07:37:24.0520299Z     cmd_result = self.invocation.execute(args)
2024-04-28T07:37:24.0520606Z src/azure-cli-core/azure/cli/core/commands/__init__.py:518: in execute
2024-04-28T07:37:24.0520812Z     self.commands_loader.load_command_table(args)
2024-04-28T07:37:24.0521126Z src/azure-cli-core/azure/cli/core/__init__.py:419: in load_command_table
2024-04-28T07:37:24.0521324Z     index_result = command_index.get(args)
2024-04-28T07:37:24.0521603Z src/azure-cli-core/azure/cli/core/__init__.py:559: in get
2024-04-28T07:37:24.0521775Z     self.invalidate()
2024-04-28T07:37:24.0522044Z src/azure-cli-core/azure/cli/core/__init__.py:635: in invalidate
2024-04-28T07:37:24.0522223Z     self.INDEX[self._COMMAND_INDEX_VERSION] = ""
2024-04-28T07:37:24.0522592Z src/azure-cli-core/azure/cli/core/_session.py:75: in __setitem__
2024-04-28T07:37:24.0522757Z     self.save_with_retry()
2024-04-28T07:37:24.0523016Z src/azure-cli-core/azure/cli/core/_session.py:65: in save_with_retry
2024-04-28T07:37:24.0523181Z     self.save()
2024-04-28T07:37:24.0523320Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-04-28T07:37:24.0523401Z 
2024-04-28T07:37:24.0523545Z self = <azure.cli.core._session.Session object at 0x7fa73392b3d0>
2024-04-28T07:37:24.0523630Z 
2024-04-28T07:37:24.0523744Z     def save(self):
2024-04-28T07:37:24.0523866Z         if self.filename:
2024-04-28T07:37:24.0524130Z >           with open(self.filename, 'w', encoding=self._encoding) as f:
2024-04-28T07:37:24.0524529Z E           FileNotFoundError: [Errno 2] No such file or directory: '/home/cloudtest/.azure/dummy_cli_config_dir/xGYaEIdmTuUx36nO/commandIndex.json'
2024-04-28T07:37:24.0524672Z 
2024-04-28T07:37:24.0524920Z src/azure-cli-core/azure/cli/core/_session.py:54: FileNotFoundError
yonzhan commented 6 months ago

Thank you for opening this issue, we will look into it.

jiasli commented 6 months ago

This issue now even happens on dev branch: https://dev.azure.com/azclitools/public/_build/results?buildId=153053&view=logs&j=f127618a-a658-5219-f5b0-4a245c9227b8&t=60a7c4e7-f35f-5fef-6efb-f1b54b8cb075&l=7690

Interesting thing is that it only happens on instance 3 and certain tests:

2024-04-28T09:15:38.3043953Z =========================== short test summary info ============================
2024-04-28T09:15:38.3044484Z FAILED src/azure-cli/azure/cli/command_modules/containerapp/tests/latest/test_containerapp_commands.py::ContainerappIngressTests::test_containerapp_custom_domains_app_in_different_rg
2024-04-28T09:15:38.3045095Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_application
2024-04-28T09:15:38.3045639Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_autoscale_operation
2024-04-28T09:15:38.3046171Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_azure_monitor
2024-04-28T09:15:38.3046701Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_kafka
2024-04-28T09:15:38.3047276Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_kafka_with_optional_disk_args
2024-04-28T09:15:38.3047865Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_kafka_with_rest_proxy
2024-04-28T09:15:38.3048502Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_min_args
2024-04-28T09:15:38.3049035Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_resize
2024-04-28T09:15:38.3049595Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_availability_zones
2024-04-28T09:15:38.3050154Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_cluster_config
2024-04-28T09:15:38.3050726Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_component_version
2024-04-28T09:15:38.3051280Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_compute_isolation
2024-04-28T09:15:38.3051852Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_encryption_at_host
2024-04-28T09:15:38.3052416Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_encryption_in_transit
2024-04-28T09:15:38.3052991Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_loadbased_autoscale
2024-04-28T09:15:38.3053659Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_minimal_tls_version
2024-04-28T09:15:38.3054250Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_private_link_configurations
2024-04-28T09:15:38.3054821Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_schedulebased_autoscale
2024-04-28T09:15:38.3055373Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_cluster_with_ssh_creds
2024-04-28T09:15:38.3055902Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_monitor
2024-04-28T09:15:38.3056420Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_script_action
2024-04-28T09:15:38.3056948Z FAILED src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_usage
2024-04-28T09:15:38.3057438Z FAILED src/azure-cli/azure/cli/command_modules/apim/tests/latest/test_apim_scenario.py::ApimScenarioTest::test_apim_core_service
2024-04-28T09:15:38.3057896Z FAILED src/azure-cli/azure/cli/command_modules/apim/tests/latest/test_apim_scenario.py::ApimScenarioTest::test_apim_deletedservice
2024-04-28T09:15:38.3058356Z FAILED src/azure-cli/azure/cli/command_modules/apim/tests/latest/test_apim_scenario.py::ApimScenarioTest::test_apim_export_api
2024-04-28T09:15:38.3058915Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_account_properties.py::CognitiveServicesPropertiesTests::test_cognitiveservices_account_capabilities
2024-04-28T09:15:38.3059558Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_account_properties.py::CognitiveServicesPropertiesTests::test_cognitiveservices_account_public_network_access
2024-04-28T09:15:38.3060189Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_api_properties.py::CognitiveServicesApiPropertiesTests::test_cognitiveservices_create_api_properties
2024-04-28T09:15:38.3060827Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_byox.py::CognitiveServicesByoxTests::test_cognitiveservices_encryption
2024-04-28T09:15:38.3061369Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_byox.py::CognitiveServicesByoxTests::test_cognitiveservices_user_owned_storage
2024-04-28T09:15:38.3061937Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_cognitiveservices_command.py::CognitiveServicesTests::test_cognitiveservices_account_list_kinds
2024-04-28T09:15:38.3062543Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_cognitiveservices_command.py::CognitiveServicesTests::test_cognitiveservices_account_list_skus
2024-04-28T09:15:38.3063148Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_cognitiveservices_command.py::CognitiveServicesTests::test_cognitiveservices_account_list_skus_legacy
2024-04-28T09:15:38.3063759Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_cognitiveservices_command.py::CognitiveServicesTests::test_cognitiveservices_account_list_usage
2024-04-28T09:15:38.3064454Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_cognitiveservices_command.py::CognitiveServicesTests::test_cognitiveservices_crud
2024-04-28T09:15:38.3065023Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_cognitiveservices_softdelete.py::CognitiveServicesTests::test_cognitiveservices_softdelete
2024-04-28T09:15:38.3065723Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_commitment_plan.py::CognitiveServicesCommitmentPlanTests::test_cognitiveservices_commitment_plan
2024-04-28T09:15:38.3066301Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_custom_domain.py::CognitiveServicesCustomDomainTests::test_cognitiveservices_custom_domain
2024-04-28T09:15:38.3066888Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_deployment.py::CognitiveServicesDeploymentTests::test_cognitiveservices_deployment
2024-04-28T09:15:38.3067451Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_identity.py::CognitiveServicesByoxTests::test_cognitiveservices_identity
2024-04-28T09:15:38.3068057Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_identity.py::CognitiveServicesByoxTests::test_cognitiveservices_identity_assign_when_create
2024-04-28T09:15:38.3068629Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_model.py::CognitiveServicesModelTests::test_cognitiveservices_model
2024-04-28T09:15:38.3069266Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_network_rules.py::CognitiveServicesNetworkRulesTests::test_cognitiveservices_network_rules
2024-04-28T09:15:38.3069863Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_private_endpoint.py::CognitiveServicesPrivateEndpointTests::test_cognitiveservices_private_endpoint
2024-04-28T09:15:38.3070474Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_private_endpoint.py::CognitiveServicesPrivateEndpointTests::test_cognitiveservices_private_endpoint_connection
2024-04-28T09:15:38.3071059Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_subscription_level_apis.py::CognitiveServicesTests::test_cognitiveservices_list_models
2024-04-28T09:15:38.3071634Z FAILED src/azure-cli/azure/cli/command_modules/cognitiveservices/tests/latest/test_subscription_level_apis.py::CognitiveServicesTests::test_cognitiveservices_list_usages
2024-04-28T09:15:38.3072167Z FAILED src/azure-cli/azure/cli/command_modules/dms/tests/latest/test_service_scenarios.py::DmsServiceTests::test_project_commands
2024-04-28T09:15:38.3072725Z FAILED src/azure-cli/azure/cli/command_modules/dms/tests/latest/test_service_scenarios.py::DmsServiceTests::test_service_commands
2024-04-28T09:15:38.3073198Z FAILED src/azure-cli/azure/cli/command_modules/dms/tests/latest/test_service_scenarios.py::DmsServiceTests::test_task_commands
2024-04-28T09:15:38.3073718Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_budget_create
2024-04-28T09:15:38.3074300Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_budget_delete
2024-04-28T09:15:38.3074863Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_budget_show
2024-04-28T09:15:38.3075446Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_budget_usage_create
2024-04-28T09:15:38.3076034Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_marketplace_list
2024-04-28T09:15:38.3076624Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_marketplace_list_billing_period
2024-04-28T09:15:38.3077246Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_marketplace_list_billing_period_filter
2024-04-28T09:15:38.3077899Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_pricesheet
2024-04-28T09:15:38.3078490Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_pricesheet_billing_period
2024-04-28T09:15:38.3079080Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_usage_by_billing_period
2024-04-28T09:15:38.3079680Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_usage_by_billing_period_expand
2024-04-28T09:15:38.3080243Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_usage_list
2024-04-28T09:15:38.3080829Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_usage_list_expand
2024-04-28T09:15:38.3081421Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_consumption_usage_list_properties
2024-04-28T09:15:38.3081984Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_list_reservations_details
2024-04-28T09:15:38.3082588Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_list_reservations_details_with_reservationid
2024-04-28T09:15:38.3083180Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_list_reservations_summaries_daily
2024-04-28T09:15:38.3083801Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_list_reservations_summaries_daily_with_reservationid
2024-04-28T09:15:38.3084471Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_list_reservations_summaries_monthly
2024-04-28T09:15:38.3085106Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_list_reservations_summaries_monthly_with_reservationid
2024-04-28T09:15:38.3085706Z FAILED src/azure-cli/azure/cli/command_modules/consumption/tests/latest/test_consumption_commands.py::AzureConsumptionServiceScenarioTest::test_list_usages_subscription_custom_date_range
2024-04-28T09:15:38.3086217Z FAILED src/azure-cli/azure/cli/command_modules/maps/tests/latest/test_maps_commands.py::MapsScenarioTests::test_create_maps_account
2024-04-28T09:15:38.3086660Z FAILED src/azure-cli/azure/cli/command_modules/maps/tests/latest/test_maps_creator.py::MapsCreatorTest::test_maps_creator
2024-04-28T09:15:38.3087091Z FAILED src/azure-cli/azure/cli/command_modules/maps/tests/latest/test_maps_creator.py::MapsCreatorTest::test_maps_map
2024-04-28T09:15:38.3087356Z ============ 75 failed, 92 passed, 23 skipped in 226.24s (0:03:46) =============
jiasli commented 6 months ago

Possibly caused by https://github.com/Azure/azure-cli/pull/28673.

I am able to repro locally with

azdev test test_containerapp_manualjob_withsecret_crudoperations_e2e test_hdinsight_application -a -n 1

It seems test classes are collected by pytest in the same process, so each cli_ctx's config_dir has the same value, such as /home/cloudtest/.azure/dummy_cli_config_dir/xGYaEIdmTuUx36nO. --forked only affects test methods.

However, this still can't answer why previous tests didn't fail, such as https://dev.azure.com/azclitools/public/_build/results?buildId=152726&view=logs&j=d3f69986-121a-5641-0308-9a539278aa7f&t=2538cf34-13d6-5659-841c-bf258a998985

jiasli commented 6 months ago

This error doesn't happen without --fork, explained as below:

azure.cli.core._session.Session instances are singletons:

https://github.com/Azure/azure-cli/blob/fa307d6fdfea663e64977c4bbf8a3dbee5d97753/src/azure-cli-core/azure/cli/core/_session.py#L88-L110

Without --fork, different test methods share the same Session objects, such as INDEX, but with --forked, each test method has its own forked Session objects.

When running test_containerapp_manualjob_withsecret_crudoperations_e2e and test_hdinsight_application without --fork, test_containerapp_manualjob_withsecret_crudoperations_e2e initializes command index and saves it. As test_hdinsight_application shares the same INDEX, it reuses it and doesn't initialize command index again. But with --forked, both tests see an empty INDEX when test methods begin, and initialize it. Since the random config dir has been deleted by test_containerapp_manualjob_withsecret_crudoperations_e2e, test_hdinsight_application fails with a dir-not-exist error.

bebound commented 6 months ago

It seems that this is caused by the internal change in pytest from 8.1.1 to 8.2.0.

https://github.com/Azure/azure-cli/pull/28807 succeeded but https://github.com/Azure/azure-cli/pull/28817 failed.

azdev test test_containerapp_manualjob_withsecret_crudoperations_e2e test_hdinsight_application --series -a -rP also works in pytest 8.1.1.

I added some log in test initialization code. It seems that each test is initialized and executed in different thread, and the random config dir env is not polluted.

In 8.2.0, the __init__ method appears to execute in main thread, resulting in environment variable pollution across tests.

Ref: https://dev.azure.com/azclitools/public/_build/results?buildId=151723&view=logs&jobId=8b87d530-5472-5986-1c88-cfbbd7928469&j=5c10f295-ffed-5950-49e8-33e228ceb98b

https://dev.azure.com/azclitools/public/_build/results?buildId=153053&view=logs&j=8b87d530-5472-5986-1c88-cfbbd7928469&t=11cc7245-08a2-5894-29fb-f5b6001226e6

jiasli commented 6 months ago

This is very interesting discovery.

By injecting 

        import os
        print('__init__', os.getpid(), method_name)

at

https://github.com/Azure/azure-cli/blob/2750c65ea77638ecf699243555a5f2f7a1314ed4/src/azure-cli-testsdk/azure/cli/testsdk/base.py#L83-L86

we can see different versions of pytest show different output:

With pytest 8.1.1:

$ pip install pytest==8.1.1

$ python -m pytest -x -v --forked --log-level=WARN /home/user2/azure-cli/src/azure-cli/azure/cli/command_modules/containerapp/tests/latest/test_containerappjob_with_secrets.py::ContainerAppJobsSecretsOperationsTest::test_containerapp_manualjob_withsecret_crudoperations_e2e /home/user2/azure-cli/src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_application -rP
============================================== test session starts ===============================================
platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.5.0 -- /home/user2/py310/bin/python
cachedir: .pytest_cache
rootdir: /home/user2/azure-cli/src/azure-cli
plugins: xdist-3.5.0, forked-1.6.0
collected 2 items

src/azure-cli/azure/cli/command_modules/containerapp/tests/latest/test_containerappjob_with_secrets.py::ContainerAppJobsSecretsOperationsTest::test_containerapp_manualjob_withsecret_crudoperations_e2e PASSED [ 50%]
src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_application PASSED [100%]

===================================================== PASSES =====================================================
________ ContainerAppJobsSecretsOperationsTest.test_containerapp_manualjob_withsecret_crudoperations_e2e _________
--------------------------------------------- Captured stdout setup ----------------------------------------------
__init__ 1971704 test_containerapp_manualjob_withsecret_crudoperations_e2e
---------------------------------------------- Captured stderr call ----------------------------------------------

----------------------------------------------- Captured log call ------------------------------------------------
WARNING  cli.azure.cli.command_modules.containerapp.custom:custom.py:3233 Containerapp job 'job1000002' executions triggered now will have the added/updated secret.
WARNING  cli.azure.cli.command_modules.containerapp.custom:custom.py:3233 Containerapp job 'job1000002' executions triggered now will have the added/updated secret.
WARNING  cli.azure.cli.command_modules.containerapp.custom:custom.py:3194 Secret(s) successfully removed.
________________________________ HDInsightClusterTests.test_hdinsight_application ________________________________
--------------------------------------------- Captured stdout setup ----------------------------------------------
__init__ 1971712 test_hdinsight_application
----------------------------------------------- Captured log call ------------------------------------------------
WARNING  msrest.serialization:serialization.py:187 Readonly attribute marketplace_identifier will be ignored in class <class 'azure.mgmt.hdinsight.models._models_py3.ApplicationProperties'>
=============================================== 2 passed in 2.88s ================================================

With pytest 8.2.0:

$ pip install pytest==8.2.0

$ python -m pytest -x -v --forked --log-level=WARN /home/user2/azure-cli/src/azure-cli/azure/cli/command_modules/containerapp/tests/latest/test_containerappjob_with_secrets.py::ContainerAppJobsSecretsOperationsTest::test_containerapp_manualjob_withsecret_crudoperations_e2e /home/user2/azure-cli/src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_application -rP
============================================== test session starts ===============================================
platform linux -- Python 3.10.12, pytest-8.2.0, pluggy-1.5.0 -- /home/user2/py310/bin/python
cachedir: .pytest_cache
rootdir: /home/user2/azure-cli/src/azure-cli
plugins: xdist-3.5.0, forked-1.6.0
collecting ... __init__ 1971868 runTest
__init__ 1971868 test_containerapp_manualjob_withsecret_crudoperations_e2e
__init__ 1971868 runTest
__init__ 1971868 test_hdinsight_application
__init__ 1971868 test_hdinsight_autoscale_operation
__init__ 1971868 test_hdinsight_azure_monitor
__init__ 1971868 test_hdinsight_cluster_kafka
__init__ 1971868 test_hdinsight_cluster_kafka_with_optional_disk_args
__init__ 1971868 test_hdinsight_cluster_kafka_with_rest_proxy
__init__ 1971868 test_hdinsight_cluster_min_args
__init__ 1971868 test_hdinsight_cluster_resize
__init__ 1971868 test_hdinsight_cluster_with_availability_zones
__init__ 1971868 test_hdinsight_cluster_with_cluster_config
__init__ 1971868 test_hdinsight_cluster_with_component_version
__init__ 1971868 test_hdinsight_cluster_with_compute_isolation
__init__ 1971868 test_hdinsight_cluster_with_encryption_at_host
__init__ 1971868 test_hdinsight_cluster_with_encryption_in_transit
__init__ 1971868 test_hdinsight_cluster_with_loadbased_autoscale
__init__ 1971868 test_hdinsight_cluster_with_minimal_tls_version
__init__ 1971868 test_hdinsight_cluster_with_private_link_configurations
__init__ 1971868 test_hdinsight_cluster_with_schedulebased_autoscale
__init__ 1971868 test_hdinsight_cluster_with_ssh_creds
__init__ 1971868 test_hdinsight_monitor
__init__ 1971868 test_hdinsight_script_action
__init__ 1971868 test_hdinsight_usage
collected 2 items

src/azure-cli/azure/cli/command_modules/containerapp/tests/latest/test_containerappjob_with_secrets.py::ContainerAppJobsSecretsOperationsTest::test_containerapp_manualjob_withsecret_crudoperations_e2e PASSED [ 50%]
src/azure-cli/azure/cli/command_modules/hdinsight/tests/latest/test_hdinsight_commands.py::HDInsightClusterTests::test_hdinsight_application PASSED [100%]

===================================================== PASSES =====================================================
________ ContainerAppJobsSecretsOperationsTest.test_containerapp_manualjob_withsecret_crudoperations_e2e _________
---------------------------------------------- Captured stderr call ----------------------------------------------

----------------------------------------------- Captured log call ------------------------------------------------
WARNING  cli.azure.cli.command_modules.containerapp.custom:custom.py:3233 Containerapp job 'job1000002' executions triggered now will have the added/updated secret.
WARNING  cli.azure.cli.command_modules.containerapp.custom:custom.py:3233 Containerapp job 'job1000002' executions triggered now will have the added/updated secret.
WARNING  cli.azure.cli.command_modules.containerapp.custom:custom.py:3194 Secret(s) successfully removed.
________________________________ HDInsightClusterTests.test_hdinsight_application ________________________________
----------------------------------------------- Captured log call ------------------------------------------------
WARNING  msrest.serialization:serialization.py:187 Readonly attribute marketplace_identifier will be ignored in class <class 'azure.mgmt.hdinsight.models._models_py3.ApplicationProperties'>
=============================================== 2 passed in 3.63s ================================================

We can see pytest 8.2.0:

  1. runs extra unittest.case.TestCase.__init__ for runTest and test methods that are not even invoked
  2. runs unittest.case.TestCase.__init__ during collecting phase in the same process, instead of forked processes
jiasli commented 6 months ago

I reported an issue to pytest: https://github.com/pytest-dev/pytest/issues/12289