search

dotnet / SqlClient

Microsoft.Data.SqlClient provides database connectivity to SQL Server for .NET applications.
MIT License
851 stars 285 forks source link

Improve binary size when using NativeAOT #1942

Open roji opened 1 year ago

roji commented 1 year ago

As pointed out by @vonzshik in #1941, a minimal SqlClient application weighs 45MB, which is quite a lot. For use in size-sensitive environments, it would be good to reduce this.

Note: this may be related to #1108 - the Azure-specific functionality is currently built-in, making it opt-in would also reduce binary size.

vonzshik commented 1 year ago

Retested with .NET 8 preview 1, the size goes down to 29mb.

kant2002 commented 1 year ago

From what I see there following offenders

image

System.Private.Xml

This is coming from https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs#L2856-L2884

and from this place https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs#L2967-L2978

System.Text.Json

This one is a bit tricky, since it's indirectly referenced

Output of WhyDgml if interesting.

**** Match 1 ****
() System_Text_Json_System_Text_Json_Utf8JsonReader___ctor
  (call) System_Text_Json_System_Text_Json_JsonDocument__Parse
    (call) System_Text_Json_System_Text_Json_JsonDocument__Parse_5
      (call) System_Text_Json_System_Text_Json_JsonDocument__Parse_3
        (call) System_Text_Json_System_Text_Json_JsonDocument__Parse_4
          (call) Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken__ReadToken
            (Instance method on a constructed type) (Tentative instance method: Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken__ReadToken, ??_7Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken@@6B@ constructed)
              (Primary) Tentative instance method: Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken__ReadToken
                (call) Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken___ctor
                  (Instance method on a constructed type) (Tentative instance method: Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken___ctor, ??_7Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken@@6B@ constructed)
                    (Primary) Tentative instance method: Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebToken___ctor
                      (newobj) Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebTokenHandler__ReadJsonWebToken
                        (Instance method on a constructed type) (Tentative instance method: Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebTokenHandler__ReadJsonWebToken, ??_7Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebTokenHandler@@6B@ constructed)
                          (Primary) Tentative instance method: Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebTokenHandler__ReadJsonWebToken
                            (Virtual method) (??_7Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebTokenHandler@@6B@ constructed, VirtualMethodUse [Microsoft.IdentityModel.JsonWebTokens]Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler.ReadJsonWebToken(string))
                              (Primary) ??_7Microsoft_IdentityModel_JsonWebTokens_Microsoft_IdentityModel_JsonWebTokens_JsonWebTokenHandler@@6B@ constructed
                                (newobj) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__ValidateAttestationClaims
                                  (Instance method on a constructed type) (Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__ValidateAttestationClaims, ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider@@6B@ constructed)
                                    (Primary) Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__ValidateAttestationClaims
                                      (call) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__VerifyAzureAttestationInfo
                                        (Instance method on a constructed type) (Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__VerifyAzureAttestationInfo, ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider@@6B@ constructed)
                                          (Primary) Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__VerifyAzureAttestationInfo
                                            (call) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__CreateEnclaveSession
                                              (Instance method on a constructed type) (Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__CreateEnclaveSession, ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider@@6B@ constructed)
                                                (Primary) Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider__CreateEnclaveSession
                                                  (Virtual method) (??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider@@6B@ constructed, VirtualMethodUse [Microsoft.Data.SqlClient]Microsoft.Data.SqlClient.SqlColumnEncryptionEnclaveProvider.CreateEnclaveSession(uint8[],ECDiffieHellman,EnclaveSessionParameters,uint8[],int32,SqlEnclaveSession&,int64&))
                                                    (Primary) ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_AzureAttestationEnclaveProvider@@6B@ constructed
                                                      (newobj) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate__GetEnclaveProvider
                                                        (Instance method on a constructed type) (Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate__GetEnclaveProvider, ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate@@6B@ constructed)
                                                          (Primary) Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate__GetEnclaveProvider
                                                            (call) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate__InvalidateEnclaveSession
                                                              (Instance method on a constructed type) (Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate__InvalidateEnclaveSession, ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate@@6B@ constructed)
                                                                (Primary) Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_EnclaveDelegate__InvalidateEnclaveSession
                                                                  (callvirt) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_SqlCommand__RunExecuteReader_0
                                                                    (Instance method on a constructed type) (Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_SqlCommand__RunExecuteReader_0, ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_SqlCommand@@6B@ constructed)
                                                                      (Primary) Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_SqlCommand__RunExecuteReader_0

Also from authentication here https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ActiveDirectoryAuthenticationProvider.cs#L144-L164

Output of WhyDgml if interesting.


**** Match 1 ****
() ??_7System_Text_Json_System_Text_Json_Utf8JsonWriter@@6B@ constructed
  (newobj) System_Text_Json_System_Text_Json_Nodes_JsonNode__ToString
    (Instance method on a constructed type) (Tentative instance method: System_Text_Json_System_Text_Json_Nodes_JsonNode__ToString, ??_7System_Text_Json_System_Text_Json_Nodes_JsonNode@@6B@ constructed)
      (Primary) Tentative instance method: System_Text_Json_System_Text_Json_Nodes_JsonNode__ToString
        (Virtual method) (??_7System_Text_Json_System_Text_Json_Nodes_JsonNode@@6B@ constructed, VirtualMethodUse object.ToString())
          (Primary) ??_7System_Text_Json_System_Text_Json_Nodes_JsonNode@@6B@ constructed
            (Dictionary dependency) [System.Text.Json]System.Text.Json.Serialization.JsonConverter`1<System.Text.Json.Nodes.JsonNode>.TryRead(Utf8JsonReader&,Type,JsonSerializerOptions,ReadStack&,JsonNode&) backed by System_Text_Json_System_Text_Json_Serialization_JsonConverter_1<System___Canon>__TryRead
              (Generic dictionary dependency) (__GenericDict_System_Text_Json_System_Text_Json_Serialization_JsonConverter_1<System_Text_Json_System_Text_Json_Nodes_JsonNode>, System_Text_Json_System_Text_Json_Serialization_JsonConverter_1<System___Canon>__TryRead)
                (Primary) __GenericDict_System_Text_Json_System_Text_Json_Serialization_JsonConverter_1<System_Text_Json_System_Text_Json_Nodes_JsonNode>
                  (reloc) ??_7System_Text_Json_System_Text_Json_Serialization_Converters_JsonNodeConverter@@6B@ constructed
                    (newobj) System_Text_Json_System_Text_Json_Serialization_Converters_JsonNodeConverter__get_Instance
                      (call) System_Text_Json_System_Text_Json_Serialization_Converters_ObjectConverter__Read
                        (Instance method on a constructed type) (Tentative instance method: System_Text_Json_System_Text_Json_Serialization_Converters_ObjectConverter__Read, ??_7System_Text_Json_System_Text_Json_Serialization_Converters_ObjectConverter@@6B@ constructed)
                          (Primary) Tentative instance method: System_Text_Json_System_Text_Json_Serialization_Converters_ObjectConverter__Read
                            (Virtual method) (??_7System_Text_Json_System_Text_Json_Serialization_Converters_ObjectConverter@@6B@ constructed, VirtualMethodUse [System.Text.Json]System.Text.Json.Serialization.JsonConverter`1<object>.Read(Utf8JsonReader&,Type,JsonSerializerOptions))
                              (Primary) ??_7System_Text_Json_System_Text_Json_Serialization_Converters_ObjectConverter@@6B@ constructed
                                (newobj) System_Text_Json_System_Text_Json_Serialization_Metadata_JsonMetadataServices__get_ObjectConverter
                                  (call) System_Text_Json_System_Text_Json_Serialization_Metadata_DefaultJsonTypeInfoResolver__GetDefaultSimpleConverters
                                    (call) System_Text_Json_System_Text_Json_Serialization_Metadata_DefaultJsonTypeInfoResolver___ctor_0
                                      (Instance method on a constructed type) (Tentative instance method: System_Text_Json_System_Text_Json_Serialization_Metadata_DefaultJsonTypeInfoResolver___ctor_0, ??_7System_Text_Json_System_Text_Json_Serialization_Metadata_DefaultJsonTypeInfoResolver@@6B@ constructed)
                                        (Primary) Tentative instance method: System_Text_Json_System_Text_Json_Serialization_Metadata_DefaultJsonTypeInfoResolver___ctor_0
                                          (newobj) System_Text_Json_System_Text_Json_Serialization_Metadata_DefaultJsonTypeInfoResolver__RootDefaultInstance
                                            (call) System_Text_Json_System_Text_Json_JsonSerializerOptions__InitializeForReflectionSerializer
                                              (Instance method on a constructed type) (Tentative instance method: System_Text_Json_System_Text_Json_JsonSerializerOptions__InitializeForReflectionSerializer, ??_7System_Text_Json_System_Text_Json_JsonSerializerOptions@@6B@ constructed)
                                                (Primary) Tentative instance method: System_Text_Json_System_Text_Json_JsonSerializerOptions__InitializeForReflectionSerializer
                                                  (callvirt) System_Text_Json_System_Text_Json_JsonSerializer__GetTypeInfo
                                                    (call) System_Text_Json_System_Text_Json_JsonSerializer__Deserialize_19<System___Canon>
                                                      (call) Azure_Identity_Azure_Core_Pipeline_ClientDiagnostics__ExtractAzureErrorContent
                                                        (call) Azure_Identity_Azure_Core_Pipeline_ClientDiagnostics__ExtractFailureContent
                                                          (Instance method on a constructed type) (Tentative instance method: Azure_Identity_Azure_Core_Pipeline_ClientDiagnostics__ExtractFailureContent, ??_7Azure_Identity_Azure_Core_Pipeline_ClientDiagnostics@@6B@ constructed)
                                                            (Primary) Tentative instance method: Azure_Identity_Azure_Core_Pipeline_ClientDiagnostics__ExtractFailureContent
                                                              (Virtual method) (??_7Azure_Identity_Azure_Core_Pipeline_ClientDiagnostics@@6B@ constructed, VirtualMethodUse [Azure.Identity]Azure.Core.Pipeline.ClientDiagnostics.ExtractFailureContent(string,ResponseHeaders,IDictionary`2<string,string>&))
                                                                (Primary) ??_7Azure_Identity_Azure_Core_Pipeline_ClientDiagnostics@@6B@ constructed
                                                                  (newobj) Azure_Identity_Azure_Identity_CredentialPipeline___ctor
                                                                    (Instance method on a constructed type) (Tentative instance method: Azure_Identity_Azure_Identity_CredentialPipeline___ctor, ??_7Azure_Identity_Azure_Identity_CredentialPipeline@@6B@ constructed)
                                                                      (Primary) Tentative instance method: Azure_Identity_Azure_Identity_CredentialPipeline___ctor
                                                                        (newobj) Azure_Identity_Azure_Identity_CredentialPipeline__GetInstance
                                                                          (call) Azure_Identity_Azure_Identity_ManagedIdentityCredential___ctor_0
                                                                            (Instance method on a constructed type) (Tentative instance method: Azure_Identity_Azure_Identity_ManagedIdentityCredential___ctor_0, ??_7Azure_Identity_Azure_Identity_ManagedIdentityCredential@@6B@ constructed)
                                                                              (Primary) Tentative instance method: Azure_Identity_Azure_Identity_ManagedIdentityCredential___ctor_0
                                                                                (newobj) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider__AcquireTokenAsync_d__17__MoveNext
                                                                                  (callvirt) S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider__AcquireTokenAsync_d__17>
                                                                                    (call) S_P_CoreLib_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1<System___Canon>__Start<Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider__AcquireTokenAsync_d__17>
                                                                                      (call) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider__AcquireTokenAsync
                                                                                        (Instance method on a constructed type) (Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider__AcquireTokenAsync, ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider@@6B@ constructed)
                                                                                          (Primary) Tentative instance method: Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider__AcquireTokenAsync
                                                                                            (Virtual method) (??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider@@6B@ constructed, VirtualMethodUse [Microsoft.Data.SqlClient]Microsoft.Data.SqlClient.SqlAuthenticationProvider.AcquireTokenAsync(SqlAuthenticationParameters))
                                                                                              (Primary) ??_7Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_ActiveDirectoryAuthenticationProvider@@6B@ constructed
                                                                                                (newobj) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_SqlAuthenticationProviderManager__SetDefaultAuthProviders
                                                                                                  (call) Microsoft_Data_SqlClient_Microsoft_Data_SqlClient_SqlAuthenticationProviderManager___cctor

Ideas

For me, most easier way to reduce size of NativeAOT executable to exclude support for Xml columns by default via AppContextSwitch or at make it opt-out of Xml support. I maybe wrong, but Xml is not that important now for workloads which most likely adopt NativeAOT, but I may be wrong.

If Azure-related stuff can be make opt-in that I can take a look at other places and provide more info since I think there more places which related to Azure which can be rooted-out.

roji commented 1 year ago

For those interested, https://github.com/npgsql/npgsql/issues/4965 tracks one of the main approaches to reduce size in Npgsql. In a nutshell, this leverages the new DbDataSource abstraction from .NET 6.0 to allow users to start with only the basic featureset, and opt into specific required features in granular fashion. The fact that this is done via a new builder maintains 100% backwards compatibility.

kant2002 commented 1 year ago

https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/EnclaveDelegate.Crypto.cs#L98-L102

This class brings System.Runtime.Serialization.Json which to my surprise uses System.Private.Xml.

Also AzureAttestationEnclaveProvider bring dependency on Microsoft.IdentityModel.Json.JsonSerializer which for unkown for me reasons also like to deserialize XML.

Other questionable dependency is ConfigurationManager usage from these places where it can be disabled using AppConfig switch. https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Reliability/AppConfigManager.cs#L22-L30

https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlAuthenticationProviderManager.NetCoreApp.cs#L119-L123

and also hard dependency on ConfigurationManager is here. https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlQueryMetadataCache.cs#L37

Same for EnclaveProviderBase which also has instantiation of MemoryCache. After couple more attemps I see MemoryCache everywhere and I think I should get rid of dependency on ConfigurationManager in that class first. Otherwise nothing really can be done here.

At max I was able to remove 0.5Mb from original 25.8Mb using nighlty .NET 8 ILC when attempt to disable XML support. If I nuke enclave support I would be able reclaim up to 2.7Mb by disabling enclave support here (for experiments only) https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/EnclaveDelegate.Crypto.cs#LL86

roji commented 1 year ago

@kant2002 I think that rather than introducing feature switches to strip out ConfigurationManager (and other similar blocks), a better overall approach would be to introduce a new entry point where ConfigurationManager simply isn't used by default (this is the "slim" data source builder approach I mentioned in my comment above). This means that for regular users not concerned with size, the current APIs continue to work exactly as they do today, but those concerned with perf can use the new entry point, and opt only into what they want.

This kind of approach would also handle things like Azure.Identity, which you can't really put behind a feature switch but still need to solve.

kant2002 commented 1 year ago

@roji I did not ignore your comment about "slim" data source builder. I clearly see other locations where it 100% would be working solution. I'm not so sure about ConfigurationManager. Look at the usages of System.Runtime.Caching.MemoryCache which bring ConfigurationManager dependency internally. https://github.com/search?q=repo%3Adotnet%2FSqlClient%20MemoryCache&type=code All places except first one from addon folder (which use M.E.Caching.Memory) uses that compatibility class.

So I still have questions what to do. Replace S.R.Cachine.MemoryCache usage with M.E.Caching.Memory.MemoryCache?

roji commented 1 year ago

So I still have questions what to do. Replace S.R.Cachine.MemoryCache usage with M.E.Caching.Memory.MemoryCache?

I don't know enough about the differences here, but that definitely could be a way forward... If that eliminates the dependency on ConfigurationManager, that possibly also means that existing settings in App.config (around cache size?) would no longer be respected - I'm guessing all that would have to be clarified.

kant2002 commented 1 year ago

In addition to MemoryCache, ConfigurationManager also used in couple places in codebase https://github.com/search?q=repo%3Adotnet%2FSqlClient+ConfigurationManager+language%3AC%23&type=code&l=C%23 but unfortunately ConfigurationManager still across all samples folder. So changing samples potentially additional work item here.

Wraith2 commented 1 year ago

What are we using MemoryCache for anyway? And how is DataContractSerialization being used?

kant2002 commented 1 year ago

@Wraith2 you can take a look at search results here. It's used in relatively lot of places. Session caches, query metadata caches, key caches, And theoretically it can be configured via app.config/web.config - that's feature of S.R.Caching.MemoryCache.

kant2002 commented 1 year ago

DataContractSerialization is coming from HostGuardianServiceEnclaveProvider here https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/VirtualSecureModeEnclaveProvider.cs#L76

Wraith2 commented 1 year ago

@David-Engel is there a spec that defines what is going to come back from the call made in

https://github.com/dotnet/SqlClient/blob/73c6b2ff3ef3b55431cf187e72d5de918caccf76/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/VirtualSecureModeEnclaveProvider.cs#L74-L78

If it's well defined we can easily write our own deserialization code rather than use the DataContractDeserializer.

David-Engel commented 1 year ago

If it's well defined we can easily write our own deserialization code rather than use the DataContractDeserializer.

@Wraith2 I'm not sure. But maybe this one? https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-hgsa/7730337b-53d6-4f83-bc76-a156c3cfd342