Support arrays of primitive types in in-memory database

[ajcvickers]

See comment https://github.com/aspnet/EntityFrameworkCore/pull/5955#issuecomment-386935828 by @alexzaytsev-newsroomly

This currently does not work because:

• The in-memory type mapper rejects attempts to map, for example int[]
• Allowing mutable primitive collections will require a value comparer as well
• Even when when a converter is created for use with the in-memory database it doesn't get called.

Partial workaround: If the array will not be mutated, then wrapping it in another type should be sufficient. For example

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder
        .Entity<Blog>()
        .Property(e => e.Ints)
        .HasConversion(
            v => new ArrayWrapper(v), 
            v => v.Values);
}

// Note this makes the mapper happy, but won't be called in 2.1.
private struct ArrayWrapper
{
    public ArrayWrapper(int[] values) 
        => Values = values;

    public int[] Values { get; }
}

[roji]

You guys may want to look at what I've done on array comparers in Npgsql, at the very least it would be great to see what you think and where you end up with InMemory.

[ajcvickers]

@roji Yeah, we were discussing again whether these semantics should be built-in to EF. That is, collections of primitives other than byte arrays would be handled as mutable be default with an appropriate comparer used. This would mean that neither Npgsql or in-memory would need to override this.

[roji]

I'd love to get rid of all that code, for sure. The different cases are a bit tricky (array element has a comparer itself, is or isn't equatable...)

[greygreg87]

I 've the same issue as @alexzaytsev-newsroomly in #5955.

@ajcvickers thanks for giving advice. I'm trying to get workaround as you suggest, but error occurs: CS1061 element "PropertyBuilder<int[]>" does not contain a definition for "HasConversion" and no extension method accepting a first argument of type could be found (are you missing a using directive or an assembly reference?)

[ajcvickers]

@greygreg87 You need to be using EF Core 2.1 RC1.

[greygreg87]

Thank you @ajcvickers. I added package Microsoft.EntityFrameworkCore.Relational -V 2.1.0-rc1-final, and properly overrideed OnModelCreating method, as a result of that, all my tests have passed.

I have PostgreSQL database, so I also added Npgsql provider v.2.1.0-rc1, but when I run my application, it returns "NoResponse" (GET request for url .../api/Users), and "InvalidOperationException: The property Users.Tools is of type int[] which is not supported by current database provider" (GET request for url .../api/Users/1)

Fortunately when I comment OnModelCreating method, application works fine, but my tests fails.

I have been working in .NETCore for two months, so I don't know if this feedback is valuable, but I just want to report the errors I received.

[roji]

@greygreg87 for the Npgsql problem, you can open an issue on http://github.com/npgsql/npgsql, but please include the full exception stack trace as well as ideally a small code sample that reproduces it.

[greygreg87]

Hi,

I'm not sure it's Npgsql problem, maybe i don't fulfill all requirements from RC1, escpecially that one:

When updating packages, make sure that all EF Core packages are updated to the RC1 version. Mixing EF Core or infrastructure packages from older .NET Core versions (including previous 2.1 preview bits) will likely cause errors.

I can't update Microsoft.NETCore.App (it's blocking by the project), and Microsoft.AspNetCore.All (package restore error appears) to v.2.1.0-rc1-final.

Despite my suspicions I put here the full exception stack trace, and in meantime I'll try to prepeare a small code sample to reproduce problem:

System.InvalidOperationException: The property 'User.Tools' is of type 'int[]' which is not supported by current database provider. Either change the property CLR type or ignore the property using the '[NotMapped]' attribute or by using 'EntityTypeBuilder.Ignore' in 'OnModelCreating'. at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.PropertyMappingValidationConvention.Apply(InternalModelBuilder modelBuilder) at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnModelBuilt(InternalModelBuilder modelBuilder) at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator) at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Lazy1.CreateValue() at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel() at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model() at lambda_method(Closure , ServiceProviderEngineScope ) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies() at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider() at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies() at Microsoft.EntityFrameworkCore.DbContext.get_Model() at Microsoft.EntityFrameworkCore.Internal.InternalDbSet1.get_EntityType() at Microsoft.EntityFrameworkCore.Internal.InternalDbSet1.get_EntityQueryable() at Microsoft.EntityFrameworkCore.Internal.InternalDbSet1.System.Linq.IQueryable.get_Provider() at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable1 source, Expression expression, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable1 source, LambdaExpression expression, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.SingleOrDefaultAsync[TSource](IQueryable1 source, Expression1 predicate, CancellationToken cancellationToken) at Project.Controllers.UsersController.<GetUser>d__3.MoveNext() in C:\"Full path of the project"\Project\Controllers\UsersController.cs:line 30 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d12.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d10.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.d14.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.d22.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.d17.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.d15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Builder.RouterMiddleware.d4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.d7.MoveNext()

[roji]

@greygreg87 I'd guess that exception is coming from a different provider (since Npgsql does support int[]), as you say a code sample is the best way forward.

[greygreg87]

Hi,

To make sure, that I'm doing everything right, I started from the very beginning. First problem occured when I was trying run "migrations add":

When I commented OnModelCreating method I received:

I suspect that this and my earlier errors occurs as a result of incorrect versions of Microsoft.NETCore.App and Microsoft.AspNetCore.All. See:

I can't update it because:

• package Microsoft.Asp.Net.All 2.1.0-rc1-final need netcoreapp2.1, but in my project I am forced to use netcoreapp2.0,
• package Microsoft.AspNetCore.All is blocking by project,

If you have any suggestions, please write, and if not, it's ok, because solve of this problem isn't very necessary now.

[ajcvickers]

@greygreg87 Mixing the ".All" package 2.0.5 and the EF 2.1 packages will cause problems. You may be able to get it working by making sure all EF packages are listed explicitly as 2.1 versions, but in general if you're using ASP.NET and want 2.1 features, then you need to update to .NET Core SDK 2.1. See this thread: #11704

[efeozyer]

any news?

[ajcvickers]

@efeozyer This issue is in the Backlog milestone. This means that it is not going to happen for the 5.0 release. We will re-assess the backlog following the 5.0 release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources.

[efeozyer]

I agree with you, good luck with it.

[AndriySvyryd]

Part of https://github.com/dotnet/efcore/issues/4179

[smitpatel]

This could work with query using value comparers correctly.

[clintsinger]

I'm currently using CosmosDB for production and In-Memory with unit tests. I ran into the issue with double[] on one of my entities. By doing the ArrayWrapper method I'm able to make it work with In-Memory, but the challenge I'm having is that I'm also using ToJsonProperty with CosmosDB which conflicts with the conversion method.

i.e this doesn't work:

d.Property(p => p.Coordinates)
   .HasConversion(
     v => new ArrayWrapper<double>(v),
     v => v.Values
 ) .ToJsonProperty("coordinates");

Is there some way that I can tell what provider is being used at runtime? Then I could do something like:

if (inmemory) 
{
  d.Property(p => p.Coordinates)
     .HasConversion(
       v => new ArrayWrapper<double>(v),
       v => v.Values)
} 
else 
{
  d.Property(p => p.Coordinates)
     .ToJsonProperty("coordinates");
}

[ajcvickers]

@clintsinger Database.IsInMemory(). However, using the in-memory database for testing is generally not advised.

[clintsinger]

@ajcvickers Thanks for the pointer. In this situation the database isn't important; so in-memory will be fine.

[AndriySvyryd]

@clintsinger Have you considered using Cosmos Emulator for tests?

[clintsinger]

@AndriySvyryd I am doing development in docker containers and up to recently the only way to run cosmos was in windows.

I took a look recently at the current state of the cosmos in docker option but it was a real pain to set up and so I gave up on it for the time being.

[roji]

@clintsinger I work with the Cosmos Emulator on Linux (via docker), and the main hurdle was to make sure the Cosmos client doesn't validate the certificate; beyond that everything does work. It really is discouraged to use InMemory for this (various things that work on Cosmos will fail on InMemory, and also vice versa).

[chrislanzara]

Hi, thanks for all the hard work on this but I need to +1 on this issue. I'm involved in a project where we save an object into CosmosDB using EF Core 6.0.6 that has a List property (so for example List<Guid> Ids { get; set; }). Running my app locally and talking to CosmosDB hosted in Azure works fine (using a ValueConverter to convert List<Guid> to List<string> to store in CosmosDB, which gives me the Json structure in Cosmos that I'm expecting) but when it comes to unit testing, my EF tests fail with The property '<name>' could not be mapped because it is of type 'List<Guid>', which is not a supported primitive type or a valid entity type. In EF Core itself this looks to have been addressed in https://github.com/dotnet/efcore/issues/14762 but I came across this github issue as we are using the InMemory provider for unit testing. I've tried changing the property to List<string> but get the same error, so it looks like the InMemory provider still doesn't support lists of primitive types. Following the previous comments about the Cosmos Emulator, if I've understood correctly, the Windows hosting provider has been deprecated by Microsoft (https://docs.microsoft.com/en-us/azure/cosmos-db/tutorial-setup-ci-cd), and I haven't been able to find a way of getting a Cosmos Emulator running inside an Azure CI pipeline, so I'm a bit stuck. (Full disclosure: I haven't tried running the emulator locally yet as if I can't get it running in the pipeline, there's little point at this juncture.) I've read a few pages that say that using the InMemory provider for unit tests is discouraged - is that still the case? If so, what would be a viable option for going forwards when using Azure CI pipelines with Linux build hosts without (current) support for the Cosmos Emulator? Thanks.

[roji]

Yes, using InMemory for testing is definitely discouraged - it can never match your actual production database, and support for primitive collections for Cosmos is just one example of that. The recommended approach is to test against the Cosmos Emulator.

The page you linked to mentions simply switching to the windows-2019 agent type, which comes pre-installed with the Cosmos Emulator - is that not an option? Note that it's also possible to run Cosmos Emulator via Docker on Linux, so that's also an option.

[clintsinger]

Any chance there can be an In-Memory cosmos emulator? It would be great if one didn't have to go through the entire process of setting up the emulator by installing it, or having to run a container, or really run it anywhere else when all you need to do is unit tests.

[roji]

@clintsinger that's definitely not something we can provide - it would amount to reimplementing the entire Cosmos engine (and with that, all other database engines supported by EF Core).

If you're interested in tests which don't access the database at all (true unit tests), that can be achieved by implementing a repository layer around EF Core; see our testing docs for more details.

However, this isn't something I'd advise as a first approach. Having the Cosmos Emulator built-in to AzDo windows-2019 images should already make it very easy to use (as well as the docker image on Linux).

[chrislanzara]

Yes, using InMemory for testing is definitely discouraged - it can never match your actual production database, and support for primitive collections for Cosmos is just one example of that. The recommended approach is to test against the Cosmos Emulator.

The page you linked to mentions simply switching to the windows-2019 agent type, which comes pre-installed with the Cosmos Emulator - is that not an option? Note that it's also possible to run Cosmos Emulator via Docker on Linux, so that's also an option.

Thanks @roji. We've currently got a dependency on the Linux build agent for some other project dependencies, so can't just switch to a windows build agent unfortunately. Running the emulator in a docker image during a pipeline feels like an awful lot of overhead, not to mention the amount of time it must add to a single pipeline run. However, I'll take it back to the team and see if we can get that working, but it feels far from a seamless process; hence the ask of a +1 on getting this included in.

Thanks.

[roji]

Running the emulator in a docker image during a pipeline feels like an awful lot of overhead, not to mention the amount of time it must add to a single pipeline run.

I'd recommend trying - it's really extremely easy to do (that's what docker was made for after all). In any case, this is a far more reliable way of testing Cosmos than using an in-memory provider which would be more or less compatible with Cosmos.

[ajcvickers]

We recommend against using the in-memory provider for testing--see Testing EF Core Applications. While we have no plans to remove the in-memory provider, we will not be adding any new features to this provider because we believe valuable development time is better spent in other areas. When feasible, we plan to still fix regressions in existing behavior.