Closed ghost closed 5 years ago
Highly demanded feature!!! Incredible that there's still no support for this in .NET Core, after two years!
As mentioned by @vickyharp previously, it is now planned for 3.0. I have updated the milestone assignment to reflect this.
I see .NET Core 3.0 previews are coming out now. Is this still in scope for the release?
@mmacagno Yes
Man, hopefully AE support gets rolled out soon. We need this big time...
Adding in my vote for this as well. Our project had to switch from .NET Core to .NET Framework over this exact limitation.
Will the 3.0 release support the new Always Encrypted enclaves coming out in SQL 2019? Will there need to be anything done on the .net Core side? I don't think there would be, but if there is -- will we need to wait another 3 years for that support? When will there be previews for AE support that we can start testing?
A workaround is to use ODBC driver to access AE data. It works with .net core 2.x. https://docs.microsoft.com/en-us/sql/connect/odbc/using-always-encrypted-with-the-odbc-driver?view=sql-server-2017. In my project I have 2 connection strings, one SqlServer for non encrypted data and one ODBC for everything that needs to be decrypted. It is a pain but it works well.
A workaround is to use ODBC driver to access AE data. It works with .net core 2.x. https://docs.microsoft.com/en-us/sql/connect/odbc/using-always-encrypted-with-the-odbc-driver?view=sql-server-2017. In my project I have 2 connection strings, one SqlServer for non encrypted data and one ODBC for everything that needs to be decrypted. It is a pain but it works well.
That looks crazy compliated
Is it already available on .net core 3.0 preview ? any preview release targeted for that feature ?
I guess this is still not in any preview? This issue is making porting our website to .Net Core a real pain.
Everyone,
A new out-of-band version of SqlClient was released in preview today to NuGet. It is a new package called Microsoft.Data.SqlClient, and it includes support for Always Encrypted on .NET Core and other enhancements like UTF-8 and data classification support on both .NET Core and .NET Framework.
Initial details about the new package are included in the new .NET Core 3.0 Preview 5 announcement at https://devblogs.microsoft.com/dotnet/announcing-net-core-3-0-preview-5/ and a lot more information can be found in the new repository at https://github.com/dotnet/sqlclient, where the open source codebase for the new provider will be hosted.
In coming weeks, we will move issue tracking to the new repo. In the meantime, issues and PRs that apply to System.Data.SqlClient are still accepted here.
We would like to encourage you to try the new package and give us feedback, especially if you were blocked by Always Encrypted not being available on .NET Core.
cc @David-Engel @vickyharp
@divega thanks for the awesome news!
I just wanted to try to migrate a sample project from .NET Framework 4.7 to .NET Core 2.2 that is using Always Encrypted on a Azure SQL Database and Azure KeyVault. It was originally built from this example.
But I'm failing so far as SqlColumnEncryptionKeyStoreProvider
is not available in Microsoft.Data.SqlClient and neither is SqlConnection.RegisterColumnEncryptionKeyStoreProviders()
Could you maybe please post a complete example using Always Encrypted with KeyVault in the new Microsoft.Data.SqlClient package?
I still have the same issue with Column Encryption Setting in the connection string in my .Net Core app that uses EF. I use all the previews that were released the 6th of may 2019. I have the following related NuGet packages installed:
Microsoft.Data.SqlClient v1.0.19123.2-Preview
But the app (EntitityFrameworkCore) seems to still have a dependency System.Data.SQLClient instead of the new Microsoft.Data.SqlClient this is the Exception :
ArgumentException: Keyword not supported: 'column encryption setting'.
System.Data.SqlClient.SqlConnectionStringBuilder.GetIndex(string keyword)
System.Data.SqlClient.SqlConnectionStringBuilder.set_Item(string keyword, object value)
System.Data.Common.DbConnectionStringBuilder.set_ConnectionString(string value)
Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerConnection.getIsMultipleActiveResultSetsEnabled()
Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Microsoft.EntityFrameworkCore.Storage.IRelationalConnection.UnregisterBufferable(IBufferable bufferable)
Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable
is there something I can do to alleviate the issue?
Maybe worth to note : If I DO NOT use the Column Encryption Setting keyword, I can display the data of unencrypted tables on my page, but not the encrypted table data (throws exception) If I do use it then I get the exception mentioned above on ALL pages.
@Tim-Bijnens your issue as actually a bit different, as the reason for yours is that EF Core does not yet use the new package. The exception is thus thrown by the old System.Data.SqlClient There is already an issue for the update here: https://github.com/aspnet/EntityFrameworkCore/issues/15636
But I'm failing so far as SqlColumnEncryptionKeyStoreProvider is not available in Microsoft.Data.SqlClient and neither is SqlConnection.RegisterColumnEncryptionKeyStoreProviders() Could you maybe please post a complete example using Always Encrypted with KeyVault in the new Microsoft.Data.SqlClient package?
@David-Engel would you be able to answer this? I think it is the second time the idea of having a minimal sample comes up.
I still have the same issue with Column Encryption Setting in the connection string in my .Net Core app that uses EF.,,
@Tim-Bijnens good point. Yes, unfortunately the current preview of EF Core still uses System.Data.SqlClient. We are moving to Microsoft.Data.SqlClient in an upcoming preview. You can follow https://github.com/aspnet/EntityFrameworkCore/issues/15636 to know when the change will make it to nightly builds. At which point I believe we will still have a few rough edges to work out, but to the degree that Always Encrypted is transparent, things should generally work.
@divega It appears that when the AE port to Core was done, the Core ref classes were not updated with the new APIs. Tests were/are run against the non-ref implementation so we didn't catch it. We'll get an update to the package out ASAP.
We pushed an update to the Microsoft.Data.SqlClient 1.0 preview. It should resolve the issues noted around Always Encrypted.
@sebader - I want to note, though, custom key store providers like the Azure Key Vault provider need to be updated to work with Microsoft.Data.SqlClient. Only the built-in providers will work until that happens (hopefully soon!). So the example you were following using Azure Key Vault still won't work. But this example, which uses the built-in Windows Certificate Store provider, should work as expected after changing from System.Data.SqlClient to Microsoft.Data.SqlClient: https://docs.microsoft.com/en-us/azure/sql-database/sql-database-always-encrypted
@divega
thanks for the update @David-Engel ! Too bad though about KeyVault missing yet. I'm using Azure Functions so Windows Cert store is no good for me ;-) Where could we track the progress of their update?
@sebader I'm not sure. We've pinged the team who owns it. I don't think that project is public.
As recently announced in the .NET Blog, focus on new SqlClient features an improvements is moving to the new Microsoft.Data.SqlClient package. For this reason, we are moving this issue to the new repo at https://github.com/dotnet/SqlClient. We will still use https://github.com/dotnet/corefx to track issues on other providers like System.Data.Odbc and System.Data.OleDB, and general ADO.NET and .NET data access issues.
@David-Engel could you recommend the best place to watch for updated Azure Key Vault support? We're using certificate store for the moment but trying to catch all the places that need cert updates is a pain and we'd love to migrate in the near term.
@divega How can this change be incorporated into the EFCore.SqlServer? because System.Data.SqlClient is one of the dependencies of EFCore.SqlServer.
@kedarchinchvalkar There are two relatively small steps:
DbConnection
, constructing instead a Microsoft.Data.SqlClient.SqlConnection
(probably optional).ISqlServerConnection
service with one targeting Microsoft.Data.SqlClient
for both SqlConnection
and SqlConnectionStringBuilder
.I lifted the code from GitHub for the last Microsoft.EntityFrameworkCore.SqlServer 2.x release for SqlServerConnection
and updated it with proper targets as follows:
using Microsoft.Data.SqlClient;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal;
using Microsoft.EntityFrameworkCore.Storage;
using System.Data.Common;
namespace My.Storage
{
/// <summary>
/// This API supports the Entity Framework Core infrastructure.
/// </summary>
public class SqlServerConnection : RelationalConnection, ISqlServerConnection
{
private bool? _multipleActiveResultSetsEnabled;
// Compensate for slow SQL Server database creation
private const int DefaultMasterConnectionCommandTimeout = 60;
/// <summary>
/// This API supports the Entity Framework Core infrastructure and is not intended to be used
/// directly from your code. This API may change or be removed in future releases.
/// </summary>
public SqlServerConnection(RelationalConnectionDependencies dependencies)
: base(dependencies)
{
}
/// <summary>
/// This API supports the Entity Framework Core infrastructure and is not intended to be used
/// directly from your code. This API may change or be removed in future releases.
/// </summary>
protected override DbConnection CreateDbConnection() => new SqlConnection(ConnectionString);
/// <summary>
/// This API supports the Entity Framework Core infrastructure and is not intended to be used
/// directly from your code. This API may change or be removed in future releases.
/// </summary>
public virtual ISqlServerConnection CreateMasterConnection()
{
var connectionStringBuilder = new SqlConnectionStringBuilder(ConnectionString)
{
InitialCatalog = "master"
};
connectionStringBuilder.Remove("AttachDBFilename");
var contextOptions = new DbContextOptionsBuilder()
.UseSqlServer(
connectionStringBuilder.ConnectionString,
b => b.CommandTimeout(CommandTimeout ?? DefaultMasterConnectionCommandTimeout))
.Options;
return new SqlServerConnection(Dependencies.With(contextOptions));
}
/// <summary>
/// This API supports the Entity Framework Core infrastructure and is not intended to be used
/// directly from your code. This API may change or be removed in future releases.
/// </summary>
public override bool IsMultipleActiveResultSetsEnabled
=> (bool)(_multipleActiveResultSetsEnabled
?? (_multipleActiveResultSetsEnabled
= new SqlConnectionStringBuilder(ConnectionString).MultipleActiveResultSets));
/// <summary>
/// Indicates whether the store connection supports ambient transactions
/// </summary>
protected override bool SupportsAmbientTransactions => true;
}
}
Then I updated my service registration for AddDbContext as follows, where I construct a Microsoft.Data.SqlClient.SqlConnection
and SqlServerConnection
is using SqlServerConnection = My.Storage.SqlServerConnection;
:
public void ConfigureServices(IServiceCollection services)
{
services
.AddDbContext<MyContext>(opts =>
opts
.UseSqlServer(new SqlConnection(Configuration.GetConnectionString("DefaultConnection")))
.ReplaceService<ISqlServerConnection, SqlServerConnection>()) // TODO: Remove after transition to Core 3
}
Works like a charm. I even think the step of constructing the DbConnection may be unnecessary and you can use the raw string value for UseSqlServer(), but haven't tried it. Feel free to experiment.
@David-Engel could you recommend the best place to watch for updated Azure Key Vault support? We're using certificate store for the moment but trying to catch all the places that need cert updates is a pain and we'd love to migrate in the near term.
@lsuarez5280 I'm not sure. I'll be sure to update this issue if I hear anything, though.
@divega How can this change be incorporated into the EFCore.SqlServer? because System.Data.SqlClient is one of the dependencies of EFCore.SqlServer.
@kedarchinchvalkar, EF Core 3.0 will switch to depend directly on Microsoft.Data.SqlClient. This change is coming probably in the next preview.
@David-Engel should we have a separate issue to track Azure Key Vault support? I am not very familiar with the feature, but it seems we need to follow up internally with the owners of https://www.nuget.org/packages/Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider.
@divega I've already notified the team that owns the AKV library. Given the priority of AE support in SqlClient on netcore, I'm certainly keeping tabs on their progress. I've filed #111 here for tracking.
@kedarchinchvalkar There are two relatively small steps:
- Register your DB context for DI using the overload provided for specifying the abstract
DbConnection
, constructing instead aMicrosoft.Data.SqlClient.SqlConnection
(probably optional).- Replace the
ISqlServerConnection
service with one targetingMicrosoft.Data.SqlClient
for bothSqlConnection
andSqlConnectionStringBuilder
.I lifted the code from GitHub for the last Microsoft.EntityFrameworkCore.SqlServer 2.x release for
SqlServerConnection
and updated it with proper targets as follows:using Microsoft.Data.SqlClient; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal; using Microsoft.EntityFrameworkCore.Storage; using System.Data.Common; namespace My.Storage { /// <summary> /// This API supports the Entity Framework Core infrastructure. /// </summary> public class SqlServerConnection : RelationalConnection, ISqlServerConnection { private bool? _multipleActiveResultSetsEnabled; // Compensate for slow SQL Server database creation private const int DefaultMasterConnectionCommandTimeout = 60; /// <summary> /// This API supports the Entity Framework Core infrastructure and is not intended to be used /// directly from your code. This API may change or be removed in future releases. /// </summary> public SqlServerConnection(RelationalConnectionDependencies dependencies) : base(dependencies) { } /// <summary> /// This API supports the Entity Framework Core infrastructure and is not intended to be used /// directly from your code. This API may change or be removed in future releases. /// </summary> protected override DbConnection CreateDbConnection() => new SqlConnection(ConnectionString); /// <summary> /// This API supports the Entity Framework Core infrastructure and is not intended to be used /// directly from your code. This API may change or be removed in future releases. /// </summary> public virtual ISqlServerConnection CreateMasterConnection() { var connectionStringBuilder = new SqlConnectionStringBuilder(ConnectionString) { InitialCatalog = "master" }; connectionStringBuilder.Remove("AttachDBFilename"); var contextOptions = new DbContextOptionsBuilder() .UseSqlServer( connectionStringBuilder.ConnectionString, b => b.CommandTimeout(CommandTimeout ?? DefaultMasterConnectionCommandTimeout)) .Options; return new SqlServerConnection(Dependencies.With(contextOptions)); } /// <summary> /// This API supports the Entity Framework Core infrastructure and is not intended to be used /// directly from your code. This API may change or be removed in future releases. /// </summary> public override bool IsMultipleActiveResultSetsEnabled => (bool)(_multipleActiveResultSetsEnabled ?? (_multipleActiveResultSetsEnabled = new SqlConnectionStringBuilder(ConnectionString).MultipleActiveResultSets)); /// <summary> /// Indicates whether the store connection supports ambient transactions /// </summary> protected override bool SupportsAmbientTransactions => true; } }
Then I updated my service registration for AddDbContext as follows, where I construct a
Microsoft.Data.SqlClient.SqlConnection
andSqlServerConnection
isusing SqlServerConnection = My.Storage.SqlServerConnection;
:public void ConfigureServices(IServiceCollection services) { services .AddDbContext<MyContext>(opts => opts .UseSqlServer(new SqlConnection(Configuration.GetConnectionString("DefaultConnection"))) .ReplaceService<ISqlServerConnection, SqlServerConnection>()) // TODO: Remove after transition to Core 3 }
Works like a charm. I even think the step of constructing the DbConnection may be unnecessary and you can use the raw string value for UseSqlServer(), but haven't tried it. Feel free to experiment.
Its giving an error on UserSqlServer() as, The call is ambiguous between the following methods or properties:
'Microsoft.EntityFrameworkCore.SqlServerDbContextOptionsExtensions.UseSqlServer(Microsoft.EntityFrameworkCore.DbContextOptionsBuilder, System.Data.Common.DbConnection, System.Action
Its giving an error on UserSqlServer() as, The call is ambiguous between the following methods or properties: 'Microsoft.EntityFrameworkCore.SqlServerDbContextOptionsExtensions.UseSqlServer(Microsoft.EntityFrameworkCore.DbContextOptionsBuilder, System.Data.Common.DbConnection, System.Action
)' and 'Microsoft.EntityFrameworkCore.SqlServerDbContextOptionsExtensions.UseSqlServer(Microsoft.EntityFrameworkCore.DbContextOptionsBuilder, System.Data.Common.DbConnection, System.Action )'
@kedarchinchvalkar That sounds like you may have reference problems in your project with multiple SQL Server EF Relational packages imported somehow, but is not related to the code. Those method prototypes are exactly the same.
Its giving an error on UserSqlServer() as, The call is ambiguous between the following methods or properties: 'Microsoft.EntityFrameworkCore.SqlServerDbContextOptionsExtensions.UseSqlServer(Microsoft.EntityFrameworkCore.DbContextOptionsBuilder, System.Data.Common.DbConnection, System.Action
)' and 'Microsoft.EntityFrameworkCore.SqlServerDbContextOptionsExtensions.UseSqlServer(Microsoft.EntityFrameworkCore.DbContextOptionsBuilder, System.Data.Common.DbConnection, System.Action )' @kedarchinchvalkar That sounds like you may have reference problems in your project with multiple SQL Server EF Relational packages imported somehow, but is not related to the code. Those method prototypes are exactly the same.
Yes, You're right. There was a duplicate reference due to "Microsoft.AspNetCore.All" package. Thank you so much.
Program now builds perfectly. I have applied column encryption on database table columns. but while reading data I am getting following error,
TState state,
Func<DbContext, TState, TResult> operation,
Func<DbContext, TState, ExecutionResult<TResult>> verifySucceeded)
{
try
{
return operation(Dependencies.CurrentDbContext.Context, state);
}
catch (Exception ex) when (ExecutionStrategy.CallOnWrappedException(ex, SqlServerTransientExceptionDetector.ShouldRetryOn))
{
throw new InvalidOperationException(SqlServerStrings.TransientExceptionDetected, ex);
}
}
Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable
Any updates on above?
@kedarchinchvalkar I'm gonna guess that you haven't actually added Column Encryption Setting=Enabled
in your connection string. At this point, I suggest you don't use this issue to continue the conversation. Either seek community support for utilizing the connector or open a new issue so your questions can be addressed there.
@kedarchinchvalkar I'm gonna guess that you haven't actually added
Column Encryption Setting=Enabled
in your connection string. At this point, I suggest you don't use this issue to continue the conversation. Either seek community support for utilizing the connector or open a new issue so your questions can be addressed there.
@Isuarez5280 : Thank you so much for the suggestion. I will open it as a new issue.
Thank you so much @lsuarez5280 and @divega Your solution works like a charm.
We believe the originally reported issue has been addressed in Microsoft.Data.SqlClient (NetCore), please open new issues for any specific usecase, closing the issue.
When using a DbContext with a ConnectionString like
I get the following error StackTrace
The issue is about adding support for the Column Encryption Setting to the .NET Core ADO.NET SqlClient.