Open thomas11 opened 2 months ago
Size in megabytes isn't the most concerning problem for this provider IMO – rather the number of symbols/objects the IDE's language server has to analyze & index, for properly typed experience.
As mentioned in #3124 the sheer number of objects in this provider, due to the breadth of the Azure API and autogenerated nature of this provider, causes the Python Language Server to run out of memory and blow up.
Now with the introduction of #3400 number of symbols in the provider seems to have increased by roughly 30k:
$ grep -Ri 'ArgsDict(TypedDict)' . | wc -l
29958
I really like the direction of TypedDicts and assume they will supersede the Args classes in the next major version but this seems to have made matters even worse than before, causing more frequent crashes of the Python Language Server in VScode.
Is LSP performance improved or affected by how pulumi_azure_native is imported?
# import named types:
from pulumi_azure_native.storage import ...
# import submodule
from pulumi_azure import storage
# import package
import pulumi_azure
Don't know if this is getting out of scope but I guess these are in the end symptoms caused by the size/breadth of the SDK – @thomas11 feel free to comment if you want Python/IDE/typing specific issues/symptoms to be tracked in a sub/seperate issue.
Is LSP performance improved or affected by how pulumi_azure_native is imported?
For sure, import pulumi_azure_native
will trigger the LSP into scanning the whole library and should be avoided. I've made several attempts in optimizing the settings and import strategies but importing a sub-package (... import storage
) vs. specific objects from [...].storage import X, Y Z
doesn't seem to make much of a difference – I suspect the LSP will analyze __init__.py
anyways and follow the trail to other modules in the same sub-package.
I took another shot at investigating the behaviour in a freshly created azure-native Python project and even when being extra careful with your imports something like hovering over a module and navigating to its definition will kick off a library-wide scan.
After some unstructured browsing through issues of similar nature tracked in Pyright & MyPy affecting other large Python libraries I wanted to throw some ideas into the mix that might (?) improve IDE experience in generated Python SDKs:
-stubs
package or via package extras.__init__.pyi
for a sub-package with extracted type information from all of its members improve performance instead of generating typestubs for each sub-package's module? Using pulumi_azure_native.storage
as an example – it has 44 Python files most of which only export 2-4 symbols each (resource & args classes or get/list/result functions/types)lazy_loader
implements this in a way that also supports attaching *.pyi during type checking. This blog post and implementation might provide some insights on possible improvements?__init__.py
s in Pulumi Python SDKs seem to use imports (i.e. `from .blob import ) instead of explicit imports
from .blob import Blob, BlobArgsand defining their exported interfaces via
all`. I don't know if this would improve performance but most libraries I've investigated follow the latter pattern.@olafurnielsen appreciate the insight and citations here. I think I can speak for Pulumi and say great feedback like this really helps us hone our product.
Added to epic https://github.com/pulumi/home/issues/3552
This is an open-ended issue to capture ideas and investigations on how we could shrink the size the provider and its SDK. The v3 major version bump might provide opportunities for breaking changes.