Azure SDK is over 500MB and growing on each release.

[sodul]

The azure SDK is ridiculously large for reasons that I have a hard time understanding. We pip install it for our CI pipelines and the vast majority of the size of our container is coming from the Azure SDK, in the SDK the network directory is taking almost half of the size and this is because there are 39 versions of the SDK.

I have never seen anyone doing such a strange approach to version their API clients. I fail to understand why anyone would even want to use the client from 2015 on a cloud product like Azure.

root@1bba10bd1500:~/.pyenv/versions/3.9.2/lib/python3.9/site-packages/azure/mgmt/network# du -shc * | grep M | sort -n 
1.2M    aio
2.4M    v2015_06_15
3.3M    v2016_09_01
3.5M    v2016_12_01
3.7M    v2017_03_01
4.4M    v2017_06_01
4.4M    v2017_08_01
4.9M    v2017_09_01
5.1M    v2017_10_01
5.1M    v2017_11_01
5.1M    v2018_01_01
5.7M    v2018_02_01
6.5M    v2018_04_01
6.6M    v2018_06_01
6.9M    v2018_07_01
8.3M    v2018_08_01
8.4M    v2018_10_01
8.6M    v2018_11_01
8.8M    v2018_12_01
9.0M    v2019_02_01
9.5M    v2019_04_01
10M v2019_06_01
11M v2019_07_01
11M v2019_08_01
11M v2019_09_01
11M v2019_11_01
11M v2019_12_01
12M v2020_03_01
12M v2020_04_01
13M v2020_05_01
13M v2020_06_01
13M v2020_07_01
13M v2020_08_01
259M    total

Can the default release only prove the latest version of the client libraries, or at least provide a 'lean' version of the SDK? This release model is certainly not sustainable and is causing useless grief to your users.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

Issue Details

---

The azure SDK is ridiculously large for reasons that I have a hard time understanding. We pip install it for our CI pipelines and the vast majority of the size of our container is coming from the Azure SDK, in the SDK the network directory is taking almost half of the size and this is because there are 39 versions of the SDK. I have never seen anyone doing such a strange approach to version their API clients. I fail to understand why anyone would even want to use the client from 2015 on a cloud product like Azure. ``` root@1bba10bd1500:~/.pyenv/versions/3.9.2/lib/python3.9/site-packages/azure/mgmt/network# du -shc * | grep M | sort -n 1.2M aio 2.4M v2015_06_15 3.3M v2016_09_01 3.5M v2016_12_01 3.7M v2017_03_01 4.4M v2017_06_01 4.4M v2017_08_01 4.9M v2017_09_01 5.1M v2017_10_01 5.1M v2017_11_01 5.1M v2018_01_01 5.7M v2018_02_01 6.5M v2018_04_01 6.6M v2018_06_01 6.9M v2018_07_01 8.3M v2018_08_01 8.4M v2018_10_01 8.6M v2018_11_01 8.8M v2018_12_01 9.0M v2019_02_01 9.5M v2019_04_01 10M v2019_06_01 11M v2019_07_01 11M v2019_08_01 11M v2019_09_01 11M v2019_11_01 11M v2019_12_01 12M v2020_03_01 12M v2020_04_01 13M v2020_05_01 13M v2020_06_01 13M v2020_07_01 13M v2020_08_01 259M total ``` Can the default release only prove the latest version of the client libraries, or at least provide a 'lean' version of the SDK? This release model is certainly not sustainable and is causing useless grief to your users.

Author:	sodul
Assignees:	-
Labels:	`Mgmt`, `Network`, `Service Attention`, `customer-reported`, `question`
Milestone:	-

[kristapratico]

Hi @sodul, thanks for the feedback, we'll investigate asap.

[jiasli]

Previously reported in #11149.

[sodul]

To clarify #11149 is only about azure-mgmt-network which is the largest directory but the problem is present across the entire Azure SDK.

I understand the reasoning for the approach to keep everything for backward compatibility but if you do have customers that point to the old versions then they should pin their requirement versions to the old pypi.org releases of the Azure SDK, not force everyone to keep a copy of everything around. How about providing two versions of the SDKs: one large with everything, one small with just the latest version.

[nolaexe]

Hey, is there any update?

[sodul]

I wrote a script that we run after pip install. It detects the unused versions and this got us an azure folder shrink from ~ 680MB to ~ 280MB. It cannot go any lower because for some reason some of the objects model definitions from multiple versions are merged together to make the final list that is then used. The script detects the versions that are used internally by the SDK and preserves them, making the script very safe to use.

If there is interest I can open source the script.

[sodul]

We have released our script on GitHub. It does delete a good chunk of the API folders but not all of it. With the script the Azure directory is now just under 300MB instead of over 700MB. It is compatible with most, but not all, third party packages, as long as they do not point to a version that is trimmed.

https://github.com/clumio-code/azure-sdk-trim

[KranthiPakala-MSFT]

@kristapratico Following up to see if there is any update on this issue? - Thank you

[lmazuel]

@KranthiPakala-MSFT we are working on this, and there is ongoing discussion on the issue to be sure we consider all possible impact of any decisions, and nobody would be broken by it.

[logachev]

@lmazuel I think one old proposal that won't break anything is to release separate azure-sdk-slim with only latest APIs (that are used by default) and possibly do something with comments (iirc, removing comments reduces the size by 30%)

[sodul]

Removing non latest APIs, will remove about 60% of the disk space needed. A further design issues is that some of the API definitions import prior APIs in order to have a complete set of objects. I have no idea why these API definitions where designed this way but it is definitely not very good. I did not think of the idea of stripping comments, which means that we could probably extend azure-sdk-trim to remove comments and other useless whitespace. There is probably a tool that 'compresses' python that we could run. Of course we would not want to remove docstrings, they do help.

[logachev]

@sodul Yeah, agreed. So far I saw only keyvault being broken by your tool (which should be fixed soon I guess https://github.com/Azure/azure-sdk-for-python/issues/21623).

I think there are actually 2 scenarios we're talking about.. Development - I agree, comments & doc strings are useful. However, building production image - docstrings are unnecessary.. The only trick there is - need to preserve number of empty lines as a replacement for a docstring comment to get same line numbers with exceptions.

[ghost]

Hi @sodul. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[sodul]

/unresolve

[msyyc]

we will go on to reduce size for azure-mgmt-network: https://github.com/Azure/azure-sdk-for-python/issues/21301

[kapilt]

@msyyc its not specific to azure mgmt network, its a design problem affecting every package in the sdk, network mgmt just happens to be the most egregious. unfortunately its also not clear which azure sovereign cloud maps to which version of each sdk.

[BigCat20196]

hi @kapilt, This is a multi API defined by the service team. The previous version will be retained and accumulated continuously. This also has an impact on our daily SDK release. In subsequent releases, we try to use substitution instead of accumulation.

This problem will be solved slowly in the future. Thanks!

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[sodul]

It is good to hear it is being worked on. Is there a more specific timeline that can be shared? @BigCat20196 I see you added the needs-author-feedback label, which trigger the msftbot to try to close the issue. I'm not sure what additional feedback would help move this forward.

[kapilt]

any updates, cause afaics, 18 months later, it seems like negligible progress on fixing either the continued size growth or the sdk design flaw that causes it, and that users are forced to manually trim the sdk after the fact, using third party scripts. https://github.com/clumio-code/azure-sdk-trim/

[sodul]

An update on the size. With the latest SDK the size seem to still be growing. We now see close to 1GB before trimming and over 400MB after trimming:

+ /_install/azure_sdk_trim.py
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/azure is using 987.1 MB.
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/azure is now using 416.3 MB.

This is double the size from when this issue was opened. While there has been some acknowledgment of the issue, it does not seem to be prioritized enough.

[lmazuel]

@sodul we're sorry if it felt we were not working on this, but we were actually doing it hard for quite some time now. It may seem trivial, but this is not, as this is just not acceptable to remove api versions support for customers that requires to be stuck on old API Version (there are various legit reasons why this could happen).

We have a first prototype of the Network package we were actually planning to share today: https://github.com/iscai-msft/azure-sdk-for-python/releases/download/azure-mgmt-network%40v23.0.0b1/azure_mgmt_network-23.0.0b1-py3-none-any.whl

Works with any regular pip install:

pip install https://github.com/iscai-msft/azure-sdk-for-python/releases/download/azure-mgmt-network%40v23.0.0b1/azure_mgmt_network-23.0.0b1-py3-none-any.whl

This trims Network to 25% of what it was, without removing any API versions support. Roughly it's deduplicating smartly code that didn't change between API versions using decorators.

This version only deduplicates operations for now, we're planning to work on models early 2023 to gain even more, before we consider a general release. Once it has been confirmed that this new SDK is correct, we would apply the new system to all SDK. We started by Network because it's the biggest by far.

This should not break people that never set API version, and should not break people that were using API version the way it was documented. It may break if people discovered some hidden undocumented API using Intellisense. For those, moving to the documented way to do it is the recommendation.

If some people in this thread would have the opportunity to confirm that this package is working in their workflow as expected, we would deeply appreciate the feedback. It does pass our testing on our side. This is not production ready of course, not even officially on PyPI yet.

Thanks for the patience, and sorry again for the time it took to get the first prototype.

EDIT: Release on PyPI for easier installation: https://pypi.org/project/azure-mgmt-network/23.0.0b1/

[mysiki]

Hello, full agree with @sodul , I just package Azure SDK in order to use azure ansible collection in docker image and I'm now at 1G3 ... (not only Azure but most of).

I understand that you want to provide all api version, but customer need to use package with the right api version, they can just select pip package and all will work.

At minimal, you can bring specific version / requirement with just last working api version for each Azure cloud (cloud, stack ...)

[lmazuel]

Hello, we have a new version of Network that de-duplicates models as well, that we should be able to publish as a preview next week and brings network to 5% of its initial size (while keeping full api-version support). We plan to leave it in preview for a week or two, and switch it to stable once we did enough testing. People using the latest api-version shouldn't see any difference (or it's a bug to be clear). People using a specific api-version, may have to change their code as we simplified the code structure (for instance there is no more module named after an api-version). Breaking changes will be detailed in changelog.

[iscai-msft]

Hello, we have a new version of Network that de-duplicates models as well, that we should be able to publish as a preview next week and brings network to 5% of its initial size (while keeping full api-version support). We plan to leave it in preview for a week or two, and switch it to stable once we did enough testing. People using the latest api-version shouldn't see any difference (or it's a bug to be clear). People using a specific api-version, may have to change their code as we simplified the code structure (for instance there is no more module named after an api-version). Breaking changes will be detailed in changelog.

Here is the released version https://pypi.org/project/azure-mgmt-network/23.0.0b2/. Please give it a try and let us know how it goes!

If you have any issues or questions, please create an issue in this repo and tag me.

[sodul]

@lmazuel @iscai-msft thank you for the effort with the Network part of the SDK but I did a clean install with the latest azure cli (we use both the CLI and the SDK in our CI/CD pipelines), azure-cli==2.46.0 and the azure folder has ballooned to 1.3GB. Even after running azure-sdk-trim the size is 560MB.

I did try to install azure-mgmt-network==23.0.0b2 which shaved 35MB, but the rest of the SDK is still massive and growing.

> du -shc *
528K    appconfiguration
3.1M    batch
177M    cli
 76K    common
1.3M    core
856K    cosmos
1.2M    data
380K    datalake
1.3M    graphrbac
1.0M    identity
7.8M    keyvault
152K    loganalytics
259M    mgmt
 35M    multiapi
 32K    profiles
392K    storage
9.6M    synapse
499M    total

Can we expect that the new release model will be applied to the rest of the SDK?

[iscai-msft]

@sodul yes the plan is to roll out this feature to the other sdks as well. We'll keep this issue updated because this issue isn't specific to the network sdk. Thank you again for your patience

[msyyc]

https://pypi.org/project/azure-mgmt-network/23.0.0/ is released and its size is only 5% of last stable version.

[xiangyan99]

@iscai-msft is this one addressed?

[iscai-msft]

@xiangyan99 we've addressed network, but we're going to keep this issue open as we address our other large libraries, since this issue is not specific to network

[iscai-msft]

updating this issue to say that we have also released a preview version of compute with a much smaller sdk size: https://pypi.org/project/azure-mgmt-compute/29.2.0b2/

[kapilt]

as a user who uses many different azure apis, and also uses other cloud sdks, my two cents is.

I think this is a systemic issue owing to an sdk design decision that needs to be revisited. azure sdks instead are generated across every version of the underlying service api version, resulting in significant bloat as this issue notes. several other cloud provider sdks use runtime client generation from service descriptor files, and as a result are orders of magnitude smaller than azure sdks. attempts at piecemeal fixing this on a one off basis per package seem misguided (in two years two services have been addressed) and is unlikely to resolve anything holistically, as well as likely to regress over time as more service api versions are added back.

is there any appetite for a new azure sdk that moves from static generation to runtime?

[Lawouach]

Hello,

I was struck by the size taken by azure dependencies. I was however pleased to confirm the reduced size of the network package.

updating this issue to say that we have also released a preview version of compute with a much smaller sdk size: https://pypi.org/project/azure-mgmt-compute/29.2.0b2/

By "much smaller", you mean a tiny better smaller right @iscai-msft ? Because it's still more then 225Mb on disk. That's just huge and really a problem from docker images size.

In comparison, my next biggest dependency is botocore and its "only" 82Mb.

[johanste]

There are pros and cons for runtime generation vs. shipping pre-generated code. One of the main benefits of pre-generated code is typing and other IDE integration points. Startup/load performance is generally better as well. One of the main downsides is, as is deservedly called out here, is package size.

We are looking at ways to further reducing size while retaining the value that codegen provides. The work done on the network and compute packages are intermediary steps, but I you are correct, it is primarily targeting the largest packages with the most use.

[iscai-msft]

@Lawouach the beta version. The current public version of 30.0.0 is around 67 MB, while the beta reduced size is 7.7 MB. cc @msyyc for when the compute reduced size package will GA.

I'm not sure what the 225 MB refers to, it might be the size of the entire SDK. Either way agree the entire Azure SDK is still definitely too large and we're actively working on reducing packages sizes for our SDKs, starting with the ones with the largest sizes and working our way down. @msyyc is leading the charge on this.

[Lawouach]

Hi @iscai-msft and @johanste

Thanks for the swift feedback. It's appreciated. To give a bit of context. I'm maintaining the Chaos Toolkit and a variety of its extensions. We have built Reliably on top of these to provide a friendly chaos engineering framework and product.

I therefore bundled these extensions in a container image and it had become quite large once I introduced the chaostoolkit extension which brings the following Azure dependencies:

azure-mgmt-compute>=20.0
azure-mgmt-containerservice>=22.0
azure-mgmt-rdbms>=10.1
azure-mgmt-network>=21.0.1
azure-mgmt-storage>=21.0.0
azure-mgmt-resourcegraph>=8.0
azure-mgmt-web>=3.0
azure-mgmt-netapp>=9.0
azure-identity>=1.6
msrestazure>=0.6
azure-keyvault-secrets

Indeed the 225 MB comes from the space taken by all of these on the image.

I'm quite happy to see the direction of making the packages leaner overall!

[kapilt]

as another view point, azure packages add about 354mb of size to our docker images (which also support 4 other cloud providers in a fraction of that size). the current rate of package size improvements (2 in 2 years), means there is timely no visible path to correction given the number of packages in the azure ecosystem. and as service teams rev versions, these problems are likely to reoccur in the 'corrected' packages.

wrt to packages we're (cncf cloud custodian) currently using

azure-mgmt-authorization
azure-mgmt-advisor
azure-mgmt-apimanagement
azure-mgmt-applicationinsights
azure-mgmt-batch
azure-mgmt-cognitiveservices
azure-mgmt-cosmosdb
azure-mgmt-costmanagement 
azure-mgmt-containerinstance
azure-mgmt-compute
azure-mgmt-cdn 
azure-mgmt-containerregistry
azure-mgmt-containerservice
azure-mgmt-databricks
azure-mgmt-datalake-store
azure-mgmt-datafactory
azure-mgmt-dns
azure-mgmt-eventgrid
azure-mgmt-eventhub
azure-mgmt-hdinsight
azure-mgmt-iothub
azure-mgmt-keyvault
azure-mgmt-managementgroups
azure-mgmt-network
azure-mgmt-redhatopenshift
azure-mgmt-resourcegraph
azure-mgmt-resource
azure-mgmt-rdbms
azure-mgmt-search
azure-mgmt-sql
azure-mgmt-storage
azure-mgmt-subscription
azure-mgmt-web
azure-mgmt-logic
azure-storage-blob
azure-storage-queue
azure-cosmosdb-table
applicationinsights
azure-functions
azure-graphrbac
adal
azure-storage-file
azure-mgmt-policyinsights
azure-mgmt-monitor
azure-cosmos
azure-mgmt-redis
azure-keyvault-secrets
azure-keyvault-keys
azure-keyvault-certificates
azure-identity
azure-keyvault
azure-storage-file-share
azure-mgmt-msi
azure-mgmt-servicefabric
azure-mgmt-trafficmanager
azure-mgmt-frontdoor
azure-mgmt-security
azure-mgmt-servicebus
azure-mgmt-appplatform
azure-mgmt-recoveryservices 

@johanste re dynamic gen, ide integration points can be had via other methods (typing files etc). re startup time, the runtime generation is typically cached, and lazy/amortized. most folks are aren't creating clients at import time versus runtime. the current static generation pattern also feels anti consumption, the more azure you use, the worse the problem is. and its not just package size, its an explosion on the dependency graph and supply chain validation due to the package count as well.

sadly I don't see a runtime style openapi client extant, so the lift on exploring this is larger then I'd like.

[Lawouach]

Following up on this conversation. From https://github.com/Azure/azure-cli/issues/26966#issuecomment-1647141774:

The recommended installation method is to follow the official document https://learn.microsoft.com/en-us/cli/azure/install-azure-cli to install pre-built packages, i.e. MSI, DEB, RPM and docker image, instead of from PyPI or source code.

From my perspective, this approach is broken when it comes to Azure libraries. I build my supply chain around the Python package ecosystem. I can understand the CLI can be installed via native distribution package managers. But libs should always default to being installed in their own right via Pypi.

[johanste]

My apologies for the late response - I was hoping to communicate the official plan from our perspective, but vacations came in the way. This is what I intend for the team to do:

• Expand the "generic" azure management library (this is either going to be a azure-mgmt-resources or a new library) - including documentation/illustrations on how to call arbitrary actions on a resource. This allows a size sensitive application to not include any per-resource-provider metadata (which is what makes the library so large today). In the first phase, it may not have great IDE/typing integration.
• Investigate shipping separate type stubs. Initially for the generic azure management library.

[johanste]

Following up on this conversation. From Azure/azure-cli#26966 (comment):

The recommended installation method is to follow the official document https://learn.microsoft.com/en-us/cli/azure/install-azure-cli to install pre-built packages, i.e. MSI, DEB, RPM and docker image, instead of from PyPI or source code.

From my perspective, this approach is broken when it comes to Azure libraries. I build my supply chain around the Python package ecosystem. I can understand the CLI can be installed via native distribution package managers. But libs should always default to being installed in their own right via Pypi.

Agreed.

[Lawouach]

Hey @johanste and everyone in this thread. I really do appreciate the positive feedback from your side that this is a recognized challenge. It's comforting :)

[sodul]

For the record the overall size seems to have stabilized but due to the azure sdk not explicitly calling older APIs the trimming is less efficient. With azure-cli==2.57.0:

> azure_sdk_trim.py
/Users/stephane/.pyenv/versions/3.12.1/lib/python3.12/site-packages/azure is using 1.2 GB.
Detected az cli with 39 SDKs to keep.
/Users/stephane/.pyenv/versions/3.12.1/lib/python3.12/site-packages/azure is now using 593.0 MB.
Saved 592.3 MB.

We would love to see more progress on controlling the size here.

[dry4ng]

Azure is taking twice the disk size of all our packages combined. 10x the size of AWS libraries.

[github-actions[bot]]

Hi @sodul, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[sodul]

For others we have now published the azure-sdk-trim tool to pypi.org to make it installable with a simple pip install azure-sdk-trim. https://pypi.org/project/azure-sdk-trim/

This tool is NOT affiliated with Microsoft or the Azure SDK maintainers.

With azure-cli==2.59.0 the trimming still helps a lot:

> azure-sdk-trim
/home/user/.pyenv/versions/3.12.3/lib/python3.12/site-packages/azure is using 1.2 GB.
Detected az cli with 39 SDKs to keep.
/home/user/.pyenv/versions/3.12.3/lib/python3.12/site-packages/azure is now using 607.5 MB.

[github-actions[bot]]

Hi @sodul, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[Lawouach]

Thanks @iscai-msft for keeping it opened.

[sodul]

I've noticed a significant improvement with the more recent releases of the SDK. The space used has been pretty much halved from 1.2GB to 600MB and with azure-sdk-trim we went from 600MB to 300MB.

+ azure-sdk-trim
.pyenv/versions/3.12.3/lib/python3.12/site-packages/azure is using 606.9 MB.
Detected az cli with 39 SDKs to keep.
.pyenv/versions/3.12.3/lib/python3.12/site-packages/azure is now using 305.7 MB.
Saved 301.2 MB.

This was with azure-cli==2.60.0.

[iscai-msft]

Amazing, it's still an ongoing process but the sdk and the cli team have both been working on reducing the package size, glad that you're able to see the difference!

[github-actions[bot]]

Hi @sodul, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.