search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.53k stars 2.76k forks source link

Update `verifytypes` to handle unreleased dependencies #36963

Open scbedd opened 3 weeks ago

scbedd commented 3 weeks ago

After enabling verifytypes for everyone, the storage folks started seeing a failure in azure-storage-file-datalake. The failure was NOT a typing failure, but rather a failure to install an azure-storage-blob dependency.

The reason for this can be traced to run_verifytypes.py.

def install_from_main(setup_path: str) -> None:
    path = pathlib.Path(setup_path)
    subdirectory = path.relative_to(root_dir)
    cwd = os.getcwd()
    with tempfile.TemporaryDirectory() as temp_dir_name:
        os.chdir(temp_dir_name)
        try:
            subprocess.check_call(['git', 'init'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
            subprocess.check_call(
                ['git', 'clone', '--no-checkout', 'https://github.com/Azure/azure-sdk-for-python.git', '--depth', '1'],
                stdout=subprocess.DEVNULL,
                stderr=subprocess.STDOUT
            )
            os.chdir("azure-sdk-for-python")
            subprocess.check_call(['git', 'sparse-checkout', 'init', '--cone'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
            subprocess.check_call(['git', 'sparse-checkout', 'set', subdirectory], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
            subprocess.check_call(['git', 'checkout', 'main'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)

            if not os.path.exists(os.path.join(os.getcwd(), subdirectory)):
                # code is not checked into main yet, nothing to compare
                exit(0)

            os.chdir(subdirectory)

            # right here, we are forcing a reinstall without having the azure-storage-blob version available on pip, which throws
            command = [
                sys.executable,
                "-m",
                "pip",
                "install",
                ".",
                "--force-reinstall"
            ]

The problem is that unlike a standard tox setup where we've:

This install is asking for a reinstall of azure-storage-file-datalake WITHOUT any of those handy-dandy CI functionalities that we run before invoking tox. As a result, we FAIL. Because >12.23.0 of azure-storage-blob IS NOT present yet.

The solution, unfortunately, is to apply the same replacement logic that we use in tox_harness.py to the dev_reqs of the azure-storage-file-datalake in main. 

        if in_ci():
            replace_dev_reqs(destination_dev_req, package_dir, parsed_args.wheel_dir)

Then, install that updated dev_reqs file alongside the target package by inserting -r <path-to-dev-reqs> to the pip install command array.

To test, ensure you set TF_BUILD to true locally when invoking the tox environment.

scbedd commented 3 weeks ago

FYI @kristapratico