Support raw SQL queries without defining an entity type for the result

[anpete]

For Query Types it could be nice to not require a configuration call in OnModelCreating to add the type to the model. Instead, it may be possible to lazily add a type to the model on first access (usually query).

[smitpatel]

Few design notes.

• This will have API with #11624 so will be composable.
• The type of T will be non-scalar type
• We will match the property on the type to column names in projection.
• If T turns out to be an entity type in model then we will throw exception. Users can use DbSet<T>.FromSql to achieve same result. This avoids the ambiguity if the configuration on entity type will be used or not (like custom column names etc) and if the resulting query will be tracked or not.

[stephajn]

@jenergm That will be November 2023.

That's a real shame. For all of the time that EF Core has been out and this feature still isn't there, it makes me sad that it will be put off for yet another year.

[AraHaan]

@jenergm That will be November 2023.

When will be EFCore 10? Also why cant even releases be released on even years and not odd years (for LTS) and odd releases for odd years?

[pantonis]

I think the problem is not the EF Core team (which they do fantastic job) but Microsoft itself. They left the team with very few resources compared with the importance and size of EF Core. I don't know if we can do it but it would be good to open a petition somewhere and ask from Microsoft to allocate much more resources to the EF Core team. Important and very core features are missing or took years to develop whereas if the team had more resources, I strongly believe it they would have been implemented. I have seen numerous issues where people (incl my self) complaining all the time for the lack of core features. So I believe we should stop complaining and open a petition somewhere to help the EF Core team in getting more resources.

[AraHaan]

I think the problem is not the EF Core team (which they do fantastic job) but Microsoft itself. They left the team with very few resources compared with the importance and size of EF Core. I don't know if we can do it but it would be good to open a petition somewhere and ask from Microsoft to allocate much more resources to the EF Core team. Important and very core features are missing or took years to develop whereas if the team had more resources, I strongly believe it they would have been implemented. I have seen numerous issues where people (incl my self) complaining all the time for the lack of core features. So I believe we should stop complaining and open a petition somewhere to help the EF Core team in getting more resources.

Or better yet, work on EFCore as if we were team members but first learn the codebase.

[roji]

When will be EFCore 10? Also why cant even releases be released on even years and not odd years (for LTS) and odd releases for odd years?

@AraHaan releases happen every year around November, with LTS (even version numbers) and non-LTS (odd version numbers) releases alternating. Since 7.0 is being released in 2022, that means that 10.0 will be released in 2025. I'm not sure why it's important for even version numbers to be released on even years - I definitely wouldn't change the whole schedule of .NET just to make that happen.

On a personal note from me... In my view, this feature really is a convenience that is outside EF's core functionality; Dapper was basically built to provide this exact functionality (executing raw SQL and materializing the results as arbitrary .NET POCOs), so you can very easily achieve this in your code today, without needing any special support from EF. I understand that some people want a single package (EF) to take care of all their data needs (or want to use Microsoft-only software), but I personally believe that's not a very sustainable approach in today's software world.

Note that other people in the EF team (possibly most of them) don't agree with me on this point, and we definitely are planning to implement this. But the fact that an alternative route/workaround (Dapper) exists makes this less urgent than some other features being implemented, at least in my view.

[ErikEJ]

I for one disagree on the priority of this and think having a single Microsoft supported solution is important.

But in the meantime I have a Nuget package that enables this scenario: https://www.nuget.org/packages/ErikEJ.EntityFrameworkCore.SqlServer.SqlQuery/6.0.0-preview1

[angelaki]

I for one disagree on the priority of this and think having a single Microsoft supported solution is important.

But in the meantime I have a Nuget package that enables this scenario: https://www.nuget.org/packages/ErikEJ.EntityFrameworkCore.SqlServer.SqlQuery/6.0.0-preview1

As @roji already mentioned, this feels like reinvent the wheel. Sure, if EF offered it, I'd use it. But Dapper is a quite perfect solution for this usecase, worth every of its 200kb ;). So I think this feature can actually be put far in the backlog.

I'd even strongly disagree an having a single Microsoft supported solution is important! That's what they've tried several years until noticing that sharing is the way to go for libs! Anyway, going to off-topic here. Just wanted to praise EF Core's team's work.

[ErikEJ]

I think doing this is very much in line with one of the main principles of EF Core:

Eliminates the need for most of the data-access code that typically needs to be written.

[AraHaan]

Agreed, I think having official code that seems minor to some could be major to users who actually need this trivial feature without them needing to depend on yet another package (that could come with it's own security problems). With Microsoft maintained code, there is an guarantee that if an security issue is ever found, they will patch it right away depending on how significant it may be which for me I find is a much needed bonus over 3rd party libraries that may or may not be maintained anymore and so security issues would go unpatched with no way to patch it (other than forking and maintaining it yourself).

[roji]

@ErikEJ

Eliminates the need for most of the data-access code that typically needs to be written

Interesting coming from the maintainer of the EF Power Tools :wink: Though I'm being somewhat facetious, of course (a UI isn't code). The point is more that doing raw SQL via Dapper is almost no extra code compared to what EF would provide via this issue (a simple extension method over a context could easily implement this).

@AraHaan

I think having official code that seems minor to some could be major to users who actually need this trivial feature without them needing to depend on yet another package [...] With Microsoft maintained code, there is an guarantee [...]

I very strongly believe that in order to thrive, the .NET ecosystem needs to grow beyond this mindset; no other ecosystem out there has a single company such as Microsoft providing so much of its software, and companies/users insistent on using only software from that company. The idea that software is only official, can only be secure or only gets properly maintained if it comes from Microsoft is IMHO problematic and a throwback to the pre-OSS days. Expecting Microsoft to cover any and all software needs out there is unsustainable and unrealistic; people simply have to get used to software coming from other sources, just like the rest of the software world works.

[jez9999]

I very strongly believe that in order to thrive, the .NET ecosystem needs to grow beyond this mindset; no other ecosystem out there has a single company such as Microsoft providing so much of its software, and companies/users insistent on using only software from that company. The idea that software is only official, can only be secure or only gets properly maintained if it comes from Microsoft is IMHO problematic and a throwback to the pre-OSS days. Expecting Microsoft to cover any and all software needs out there is unsustainable and unrealistic; people simply have to get used to software coming from other sources, just like the rest of the software world works.

I don't think this is a given at all. When I dabbled in using JS on the server-side one of the things that overwhelmed me and put me off frankly was that for virtually any task, there were 1000 different libraries offering the functionality and aside from a Google search for general popularity, which tended to leave 5 or 10 main competitors, there was very little way to choose between them. When you did pick one, you'd have trouble getting support if there was a problem because everything was so balkanized. There's something to be said for lots of useful functionality being provided by a central provider that most people tend to use.

[roji]

@jez9999 yes, in a mature ecosystem there are various alternatives to choose from, and a good ranking/usage system is key to helping users making the right decision; choice is a good thing. Saying that one company should provide everything isn't a viable or desirable alternative, IMHO, and stifles ecosystem progress to whatever that company happens to be prioritizing at the moment because of its considerations.

[roji]

@vigouredelaruse raw queries for arbitrary CLR types is something you can already easily do with Dapper; if that's the main thing your application needs to do, then there's really little reason to use EF.

[Mardoxx]

That’s not the point. Many things could be done with external libraries. These community maintained projects are a lot more risky to use from functionality provided in core framework.

On Fri, 18 Nov 2022 at 12:37, Shay Rojansky @.***> wrote:

@vigouredelaruse https://github.com/vigouredelaruse raw queries for arbitrary CLR types is something you can already easily do with Dapper; if that's the main thing your application needs to do, then there's really little reason to use EF.

— Reply to this email directly, view it on GitHub https://github.com/dotnet/efcore/issues/10753#issuecomment-1319886031, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB2QMJSUXJ6GZNQUP6GMHDWI5TALANCNFSM4ENJ43TQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[roji]

@Mardoxx EF Core cannot be a universal answer for every need out there, and treating community projects as inherently risky is IMHO one of the major problems in the .NET ecosystem; other ecosystems (Java, Python) depend on community projects and are doing just fine, whereas in the .NET ecosystem many users are still fixated on using only Microsoft software. I can point to various .NET community projects which are doing better than their Microsoft counterpart, so I'd recommend evaluating which projects you're using based criteria other than just Microsoft or not.

[roji]

@vigouredelaruse I'm not really following... Regardless of the specific scenario you're trying to enable, the discussion here is about executing a SQL query and materializing the result as an arbitrary .NET type. This is something that's already well-supported by Dapper, and this issue tracks adding the same capability to EF.

I can see that making sense if your application generally works with EF since it uses statically-known types, but occasionally needs to do such a dynamic query. However, if your application mainly interacts with dynamic database data via raw SQL queries - as seems to be the case from your description above - then EF is probably not the right tool for the job.

[roji]

they spin up a website that allows users to create and query database entities and use those entities on their content collections without recompiling the project but perhaps by using a 'well known' dto clr poco with a json payload [...] now i need to hydrate arbitrary classes to refer to in controller action methods (and document/operation authorization handlers).

This is specifically not a mode that's well-supported by EF. You can theoretically do completely dynamic model building and add arbitrary user POCO types to the model. But the real question is what querying even looks like in this kind of scenario; sure, you can dynamically construct LINQ expression trees which query your dynamic user POCOs, but there's very little advantage of using EF in this way over just using SQL.

now that efcore supports json things get easier

Not really; EF's JSON support still requires a well-known, non-dynamic model for the stuff you put inside JSON documents - unless you want to just deal with JSON data as strings, and take handle serialization/deserialization yourself (that has always been possible, EF7 doesn't make that mode better).

To summarize, EF (like most other ORMs in the world) simply isn't meant to be used in scenarios where the schema and queries are completely dynamic. Regardless of that, .NET already has good ways of materializing arbitrary SQL results to CLR POCOs - that's exactly what Dapper does. We do intend to provide that ability in EF (that's what this issue is about), but if that's going to be your main use of EF, I'd advise just going with Dapper.

[vigouredelaruse]

ackowledged. apologiess for the distraction

[teliaz]

Planned for EF8 in the meanwhile as a workaround you can create a empty view with the appropriate return types and use FromSqlRaw

Works fine on ef6, ef7.

Postgres example follows bellow:

The view

CREATE OR REPLACE VIEW public.result_function_metrics
AS SELECT 0::double precision AS date_year,
    0::double precision AS date_month,
    0::bigint AS metric,

The function

CREATE OR REPLACE FUNCTION fx_reports_goals (
    _user_id int8, 
    _date_from timestamp with time zone, 
    _date_to timestamp with time ZONE
)
RETURNS SETOF result_function_metrics
AS 
$$
BEGIN

... 

END
$$
LANGUAGE plpgsql;

var query = @"SELECT * FROM fx_reports_goals(
            @pUserId,
            @pDateFrom,
            @pDateTo);";
      var parameters = new NpgsqlParameter[3];

parameters[0] = new NpgsqlParameter("pUserId", NpgsqlTypes.NpgsqlDbType.Bigint);
parameters[0].Value = filter.userID ?? (object)DBNull.Value;
// same for other parameters

var results = await db.ResultFunctionMetric.FromSqlRaw(query, parameters).ToListAsync();

Notice that db contains the dbSet ResultFunctionMetric scaffolded from the empty view.

TL/DR It is easier to set return type from the function or stored procedure add a dummy view with the expected return values and as a bonus you still get that everytime you scaffold (database first approach)

[ErikEJ]

@teliaz this is exactly what EF Core Power Tools does for SQL Server. Wonder if someone could help do the same for Postgres?

[ErikEJ]

Amazing. So this will be in daily build now?

[jenergm]

Thank you @ajcvickers!

I think it is this: public static IQueryable SqlQuery( this DatabaseFacade databaseFacade, [NotParameterized] FormattableString sql) Where calls that: => SqlQueryRaw(databaseFacade, sql.Format, sql.GetArguments()!);

Very good! Did it already have released?

[ajcvickers]

@ErikEJ Should be.

@jenergm As the milestone indicates, it will be released in EF Core 8. However, it should be available now in the daily builds and will be included in preview 1 of EF8.

[goenning]

@ajcvickers just tried this feature today and I love it! This can easily replace Dapper for many and allow us to incrementally adopt EF Core.

One question though: will it support scalar values? This doesn't seem to work (only tested with npgsql).

_db.Database.SqlQuery<string>($"SELECT id FROM environments WHERE key = {appKey}").FirstOrDefault();

[Misiu]

will it support scalar values?

That would be awesome! Waiting for more info.

[ajcvickers]

@goenning @Misiu Scalar values were already supported in EF7, and should still work. If this isn't the case, then please open a new issue and attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.

[Misiu]

@ajcvickers I've created a new issue showing this error: https://github.com/dotnet/efcore/issues/30447 In my case I'm using SqlServer.

The issue is that EF generates this code:

exec sp_executesql N'SELECT TOP(1) [t].[Value]
FROM (
    SELECT name FROM dbo.test_table WHERE entity_id=@p0
) AS [t]',N'@p0 nvarchar(4000)',@p0=N'5A83F3C3-A88F-4A56-934C-FFB8D0E682C1'

instead of:

exec sp_executesql N'SELECT TOP(1) [p].[Name]
FROM (
    SELECT name FROM dbo.test_table WHERE entity_id=@p0
) AS [p]',N'@p0 nvarchar(4000)',@p0=N'5A83F3C3-A88F-4A56-934C-FFB8D0E682C1'

[goenning]

Thanks @Misiu thsts the same error message I get using Postgres

[borisdj]

@ajcvickers I have posted related Q on SO: https://stackoverflow.com/questions/76331657/dynamic-sub-select-using-expression-with-fromsqlraw-in-net-core-v7-with-cast Can it be linked here, and can anyone take a look, or maybe to open separate issue.

[ite-klass]

@borisdj The linked question is confusing overall, but it adds a class definition to the DB context - so no, it's not directly relevant to this ticket - which is specifically about non-entity result types

[pcapozzi]

would it be possible to show an example of how to call a stored procedure with this new functionality?

[ajcvickers]

@pcapozzi See https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-8.0/whatsnew#raw-sql-queries-for-unmapped-types

[pcapozzi]

Thanks for that information. It seems that EF8 will give some exciting new potential. My only comment is that when you mention in the docs ' The code shown here comes from RawSqlSample.cs.'

Although it does seem to be part of NewInEFCore8 project, I have never been able to download a specific project. They all seem to be part of the 'EntityFramework.Docs' solution which is now up to what seems like 100 terabytes. It would be very useful to be able to 'clone' individual projects.

Thanks for your help

Regards,

PatC

On Tue, Oct 3, 2023 at 11:28 AM Arthur Vickers @.***> wrote:

@pcapozzi https://github.com/pcapozzi See https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-8.0/whatsnew#raw-sql-queries-for-unmapped-types

— Reply to this email directly, view it on GitHub https://github.com/dotnet/efcore/issues/10753#issuecomment-1744583518, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABE7UL5F3Q5EEKPPBQNZBLX5PLDHAVCNFSM4ENJ43T2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZUGQ2TQMZVGE4A . You are receiving this because you were mentioned.Message ID: @.***>

[DavidThielen]

This is added, so great. I want to add that this is essential for calls that use FreeTextTable() and ContainsTable() in SQL Server.