Resolve `windows py 311` `3221225477 ` exit code

[scbedd]

Example Build

This occurs when adding python 3.12 to the build matrix, as py 311 now runs on windows instead of ubuntu. metricsadvisor and tables both have failures collecting the test run. The actual parallel tox invocation SUCCEEDS. It's just that we're getting a weird exit code due to access violations somewhere.

Need to chase this down or make a determination to ignore the problem.

After resolving, remove eng/pipelines/templates/stages/platform-matrix-metricsadvisor_tables.json. EDIT: This is being used in communication CI now as well.

[scbedd]

Adding some context. I created a basic tox.ini at the root of azure-schemaregistry. It looks like this:

# This tox file is intended for use with any generic development feed. The only requirement is that the development feed
# contain azure* prefixed packages. It does this by assuming that `PIP_INDEX_URL` environment variable is set to target the dev feed
# specifically. Notice that in the [testenv] config, we explicitly set PIP_EXTRA_INDEX_URL to pypi.

# In all cases, whenever we install an azure-* package from a requirement (read: not a specific file), the only source will be from the dev feed.
# once we've downloaded these dependencies, only then do we install other packages from pypi.
[tox]
requires=
  # Ensure that we're running a version of tox compatible with this config
  # 4.4.10 is the lower bound because it incorporates a fix for a bug with `--root`:
  #    https://github.com/tox-dev/tox/pull/2962
  tox>=4.4.10
# note that this envlist is the default set of environments that will run if a target environment is not selected.
envlist = whl

[tools]
deps =
  -r ../../../eng/test_tools.txt

[base]
deps =
  -rdev_requirements.txt
  {[tools]deps}

[packaging]
pkgs =
    wheel==0.40.0
    packaging==23.1
    urllib3==1.26.15
    tomli==2.0.1

[testenv]
parallel_show_output = true
skip_install = true
skipsdist = true
usedevelop = false
passenv = *
download=true
setenv =
  PROXY_URL=http://localhost:5000
deps = {[base]deps}
install_command = python -m pip install {opts} {packages}
commands =
    python -m pip --version
    python -m pip install {tox_root}
    python -m pip freeze
    python -m pytest {posargs} {tox_root}/tests/test_json_encoder.py

[testenv:whl]
description=Builds a wheel and runs tests

This is extremely cut down from what the real tox.ini does. Invoke against it by

tox -r -e whl --root .

Invoking this basic utilization against azure-schemaregistry repros the problem on a local windows install running python 3.11.

However, azure-template does not trip this same behavior. I am currently running subsets of tests to try to locate the feature that is causing this to throw.

[scbedd]

Rebalanced the matrix to avoid hitting this issue. Resolving.

[scbedd]

Re-opening this as it is occurring again on #33363

Going to get @kushagraThapar a repro now.

[scbedd]

Repro Instructions

Ok @kushagraThapar , got this ready for you.

• On a different terminal tab, from repro root, invoke:

  # first install the emulator from https://aka.ms/cosmosdb-emulator
  # the default options will place it under the path designated by the `Emulator` argument in the below command
  # then run the emulator using the below script
  ./eng/common/scripts/Cosmos-Emulator.ps1 `
  -EmulatorMsiUrl "https://aka.ms/cosmosdb-emulator" `
  -StartParameters "/noexplorer /noui /enablepreview /EnableSqlComputeEndpoint /SqlComputePort=9999 /disableratelimiting /partitioncount=50 /consistency=Session" `
  -Emulator "C:/Program Files/Azure Cosmos DB Emulator/Microsoft.Azure.Cosmos.Emulator.exe" `
  -Stage "Launch"
  # as the emulator spins up, you will be bombarded by Windows security notes.

• Get a new venv with python 3.11 on a windows machine

• # from repo root, and activated venv
  python -m pip install -r .\eng\ci_tools.txt`

• # from repo root, invoke
  python .\scripts\devops_tasks\dispatch_tox.py azure-cosmos -t whl

The issue stems from something about how we are using the invoking python executable to call tox from dispatch_tox.py. This indirection is necessary because for a few reasons:

• to apply business logic to filter which packages/environments run
• To have a central place to collect logs before cleaning up the tox environment (the disks of the CI machines aren't huge, this is necessary for larger services with many packages.)
• In CI, we replace all relative dependencies with prebuilt wheels to avoid resource conflicts from parallel tox environments running and accessing the same directories

I have reason to believe that it's a specific test file registering unhandled hooks or something. The reason I say that is I locally modified this to run a single test case, and this passes. As visible in the azure-schemaregistry example above, it can repro with a single test file.

We need to identify what is causing the issue. In the meantime, I see that you removed 3.11 from the CI matrix in the PR. Probably a good idea while we investigate.