V3: shrink package size

[thomas11]

This is an open-ended issue to capture ideas and investigations on how we could shrink the size the provider and its SDK. The v3 major version bump might provide opportunities for breaking changes.

[olafurnielsen]

Size in megabytes isn't the most concerning problem for this provider IMO – rather the number of symbols/objects the IDE's language server has to analyze & index, for properly typed experience.

As mentioned in #3124 the sheer number of objects in this provider, due to the breadth of the Azure API and autogenerated nature of this provider, causes the Python Language Server to run out of memory and blow up.

Now with the introduction of #3400 number of symbols in the provider seems to have increased by roughly 30k:

$ grep -Ri 'ArgsDict(TypedDict)' . | wc -l
29958

I really like the direction of TypedDicts and assume they will supersede the Args classes in the next major version but this seems to have made matters even worse than before, causing more frequent crashes of the Python Language Server in VScode.

[AaronFriel]

Is LSP performance improved or affected by how pulumi_azure_native is imported?

# import named types:
from pulumi_azure_native.storage import ...
# import submodule
from pulumi_azure import storage
# import package
import pulumi_azure

[olafurnielsen]

Don't know if this is getting out of scope but I guess these are in the end symptoms caused by the size/breadth of the SDK – @thomas11 feel free to comment if you want Python/IDE/typing specific issues/symptoms to be tracked in a sub/seperate issue.

Is LSP performance improved or affected by how pulumi_azure_native is imported?

For sure, import pulumi_azure_native will trigger the LSP into scanning the whole library and should be avoided. I've made several attempts in optimizing the settings and import strategies but importing a sub-package (... import storage) vs. specific objects from [...].storage import X, Y Z doesn't seem to make much of a difference – I suspect the LSP will analyze __init__.py anyways and follow the trail to other modules in the same sub-package.

I took another shot at investigating the behaviour in a freshly created azure-native Python project and even when being extra careful with your imports something like hovering over a module and navigating to its definition will kick off a library-wide scan.

After some unstructured browsing through issues of similar nature tracked in Pyright & MyPy affecting other large Python libraries I wanted to throw some ideas into the mix that might (?) improve IDE experience in generated Python SDKs:

• Could you start generating and including typestubs (*.pyi) in the package? Those only contain extracted type information and exclude implementation for more performant type analysis. Some Python libraries choose to distribute this in a seperate -stubs package or via package extras.
• Would generating a single __init__.pyi for a sub-package with extracted type information from all of its members improve performance instead of generating typestubs for each sub-package's module? Using pulumi_azure_native.storage as an example – it has 44 Python files most of which only export 2-4 symbols each (resource & args classes or get/list/result functions/types)
• The SDK lazy loads sub-packages at runtime but imports them during type checking which is clever. Scientific Python's lazy_loader implements this in a way that also supports attaching *.pyi during type checking. This blog post and implementation might provide some insights on possible improvements?
• Sub-package __init__.pys in Pulumi Python SDKs seem to use imports (i.e. `from .blob import ) instead of explicit importsfrom .blob import Blob, BlobArgsand defining their exported interfaces viaall`. I don't know if this would improve performance but most libraries I've investigated follow the latter pattern.

[AaronFriel]

@olafurnielsen appreciate the insight and citations here. I think I can speak for Pulumi and say great feedback like this really helps us hone our product.

[cleverguy25]

Added to epic https://github.com/pulumi/home/issues/3552