Query: Consider translating String.Equals(String, StringComparison) for selected values of StringComparison

[divega]

Currently we support the == operator and the String.Equals(String) methods which both have the same behavior in-memory as String.Equals(String, StringComparison.Ordinal). We could also support the latter without making significant changes and possibility other values of StringComparsion such as OrdinalIgnoreCase by applying LOWER() on both operands.

[TrabacchinLuigi]

Actually I just got a warning about String.Equals (without the third parameter) not being able to be translated and being evaluated locally ...

[JaymzZh]

After upgrade to 1.1.0, I use String.Equals(String, StringComparison.Ordinal) got this error :

InvalidOperationException: variable '__name_0' of type 'System.String' referenced from scope '', but it is not defined

[divega]

@Jeffiy can you please create a new issue and provide a repro? If this is a regression in 1.1 we may want to fix it in 1.1.1.

[akamud]

Could this be implemented using Collation on SQL Server? Using LOWER on indexed columns throws the index away for querying, as far as I know, which could hurt performance for people using this.

We wouldn't know this is happening since this would be translated by EF internally and would get us by surprise. Since I would have no warning about "local evaluation" on this case, I would expect that my ORM is doing the right thing and not creating a performance problem. Doing a LOWER should be an explicit decision, in my opinion.

[Suriman]

@divega Please, it is a must have.

[xperiandri]

Still does not work in 2.0 I was shocked that it doesn't!

[xperiandri]

I used string.Equals(string, string)

[mgolois]

Any update on this?

[mgolois]

Migrating to EFCore is becoming a show stopper for us because this issue. Any update @divega @rowanmiller ?

[divega]

@mgolois This issue is about String.Equals(String, StringComparison). could you please elaborate about what exactly the query looks like that you are trying to port to EF Core?

[divega]

@mgolois to be more precise:

1. EF6 completely ignores the StringComparison argument. Is this the behavior that you expect from EF Core? If yes, why?
2. Also, are you aware that ==, string.Equals(String), and String.Equals(string, string) work? If yes, why is lack of support for String.Equals(String, StringComparison) blocking you?

[xperiandri]

Does equals usage with StringComparison at least produces a warning?

[smitpatel]

@xperiandri - It does evaluation on client to produce correct results.

[divega]

@xperiandri and it produces a warning like this in the log:

warn: Microsoft.EntityFrameworkCore.Query[20500]
      => Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor
      The LINQ expression 'where [p].Name.Equals(__name_0, CurrentCulture)' could not be translated and will be evaluated locally.

Which can optionally be turned into an error as explained in the documentation.

[xperiandri]

Maybe you would add Roslyn analyzer that will display info about that in design time? Because having this warning in runtime is not an excellent experience.

[xperiandri]

Which can optionally be turned into an error as explained in the documentation.

It is very good!

[Bartmax]

so what exactly happens right now if I use Equals with OrdinalIgnoreCase ? does it perform an Equals on the db and then evaluate the ignore case locally or does it gets all records of the db and do the filtering locally ?

[BalassaMarton]

Is there any plan to do this in the near future? I wasn't aware of the fact that it's not supported, saw the warning when I already used this pattern in several queries. Do you recommend changing it to Where(x => x.SomeProperty.ToLower() == parameter.ToLower()) - is string.ToLower translated correctly?

[smitpatel]

@divega - This issue suggest that for OrdinalIgnoreCase we could apply Lower on both sides, but that would mean index cannot be used (as pointed out by @akamud ), but there was no definitive answer which direction we would be taking. Can you clarify?

[divega]

@smitpatel let’s bring this to a design meeting. I would be ok with the overload with StringComparison started to throw in the default implementation in absence of client eval, but I would also be open to grandfathering it (via ignoring the StringComparison argument) if the general feeling is that it is preferable to mitigate the impact of the breaking change.

I think it is preferable not to do the ToLower trick behind the users’ backs. They can still do that themselves explicitly, and there is still a chance that some of them will miss the fact that it isn’t sargable, but at least we didn’t do it without them asking.

[divega]

@smitpatel Besides, I believe many databases will default to case insensitive comparisons anyway, so what is harder to achieve is a case sensitive one.

[divega]

@BalassaMarton if you control the database schema it is preferable to specify a column collation that will compare string according to your preference.

If you have a case insensitive collation and can’t change it but need a comparison that is case sensitive, you can do a regular (==) comparison in the database and then filter out false positives on the client with a case sensitive comparison.

I have tried other workarounds using functions that encapsulate comparisons with explicit collations, but SQL Server actually doesn’t support inline scalar functions, and non-inline functions aren’t sargable, so you end up needing to write a TVF. It is possible but complicated.

[BalassaMarton]

@divega I didn't even think about setting collation, but it could solve my short-term problems, thanks for the tip.

[smarts]

I don't think anyone pointed it out in this issue, but the main reason this is important in my use case is because string equality is [usually] case-insensitive when executed by the database, but when running in unit tests it will not be. Unless there's some configuration for EF Core that intercepts string equality comparisons and makes them case-insensitive then it's necessary for this to be supported in the query abstraction.

[smitpatel]

@smarts - This feature is to influence case sensitivity on server side. It is not for testing. There are always going to be differences between how database works & how linq works. If you want to unit test same query as database, then you need to write unit test which accounts for difference. Or test using a testing database rather than linq.

[smarts]

@smitpatel You're assuming the data in question is returned from the database, rather than used in the query to compute some other value, or simply used in code that is unavailable to the test.

[smitpatel]

Unless there's some configuration for EF Core that intercepts string equality comparisons and makes them case-insensitive

Why do you need to intercept anything and change case sensitivity. Let server handle that. The different arises only when you are computing same equality condition at different places and simple solution is account for it. You really need to give more details why any such interception mechanism should be part of EF query abstraction?

[smarts]

@smitpatel I don't think you understand, so here's a concrete example:

from f in Foos
select new
{
  Status = String.Equals(f.Bar, "bad", StringComparison.OrdinalIgnoreCase) ?
    Status.NotGood :
    ((String.Equals(f.Bar, "ok", StringComparison.OrdinalIgnoreCase) ?
      Status.Ok :
      Status.Good) :
    Status.Unknown)
};

Status is an enum In case it is not clear, I want the equality check to happen server side. Some data in the database for the Bar property/column is UPPERCASE & some is LOWERCASE, so data in the test also should be setup as such. As written it will execute client side, which is undesirable. Changing to use the == operator will work server side, but will cause the test to fail because of the difference in behavior of == for in-memory execution. I understand that for you the main purpose of the feature is to control server side behavior, but it also serves to provide parity between server side (IQueryable<T>) and client side (IEnumerable<T>) execution, which is also very important and one of the things that made EF so good.

[Zetanova]

I am using following for years now (since EF4 or so) and now parting to EF Core

query.Where(n => filter.Name.Equals(n.Name, StringComparison.InvariantCultureIgnoreCase))

Of course it does not work anymore in EF Core.

What i can remember is that the query got resolved depends on the used db-provider. if the server is case-insensitive, it resolved server-side if the server is case-sensitive, it resolved client-side

A .net case-sensitive String.Equals in the LINQ query was a deterministic command.

query.Where(n => filter.Name.Equals(n.Name))

This query was server-side if the server was case-sensitive and ALSO client-side (extra filter) if the server was case-insensitve.

I don't know when and why this behavior got removed for EF. But it should be deterministic, the same linq-filter should produce the same result executed on all db-providers or throw an exception.

[Bartmax]

I don't think it worked as you think. If that were the case, depending on the dbprovider you will retrieve the whole table causing potentially bad performance on huge tables. Not something I personally want to deal with. I believe on 3.0, this client side stuff will throw and I'm happy with it.

[ajcvickers]

@Zetanova I don't remember ever having that behavior in any version of EF. As @Bartmax, client evaluation is not going to be the way we handle this going forward.

[Zetanova]

@ajcvickers My point was not "client and/or server side" I wanted to say that it should be deterministic.

Here some old post from 2009 http://justgeeks.blogspot.com/2009/05/making-linq-to-entities-do-case.html

Because i know that i am using CI comparison server side, i just switched to the single paramter String.Equals method.

[smarts]

Hi, could someone please provide an update? I can't tell whether the proposed functionality is being considered or not…

[ajcvickers]

@smarts This issue is in the Backlog milestone. This means that it is not going to happen for the 3.0 release. We will re-assess the backlog following the 3.0 release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources.

[smarts]

Ok. I guess as long as the Z.EntityFramework.Classic NuGet package exists there's a viable alternative anyway 🤷‍♂

[TanvirArjel]

@xperiandri Still does not work in 3.0 I was shocked that it doesn't! 😂😂

[AllTaken]

I don't understand the "failure mode" of this in 3.0. It used to be that it just evaluated this locally, and performance was the issue. But NOW, for me at least, it does something that I find very detrimental.

With an entity called Currency with a string field called AlphaCode and a record where AlphaCode is "EUR".

If I do the following:

.SingleOrDefault(a =>
                    a.AlphaCode.Equals("EUR", StringComparison.InvariantCultureIgnoreCase)
                );

It just returns null (no error or anything like that)!

If I do:

.SingleOrDefault(a =>
                    a.AlphaCode.ToUpper() == "EUR".ToUpper()
                );

I get the expected entity

And it's doing, what I would consider, strange things on the Database:

With Equals

info: Microsoft.EntityFrameworkCore.Database.Command[20100]
      Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT TOP(2) [c].[CurrencyId], [c].[AlphaCode], [c].[Code], [c].[Description], [c].[SignificantDecimals]
      FROM [Currencies] AS [c]
      WHERE CAST(0 AS bit) = CAST(1 AS bit)

That purposeful 0=1 seems peculiar; why make the round-trip to the database for that? And if it is known that this can never return anything useful why not emit an error?

Without Equals

info: Microsoft.EntityFrameworkCore.Database.Command[20100]
      Executing DbCommand [Parameters=[@__ToUpper_0='EUR' (Size = 4000)], CommandType='Text', CommandTimeout='30']
      SELECT TOP(2) [c].[CurrencyId], [c].[AlphaCode], [c].[Code], [c].[Description], [c].[SignificantDecimals]
      FROM [Currencies] AS [c]
      WHERE (UPPER([c].[AlphaCode]) = @__ToUpper_0) AND @__ToUpper_0 IS NOT NULL

I would really have liked some kind of error in the Equals case. And I would really have like for the ToUpper to have been translated to UPPER on both sides of the = on the database.

[roji]

@AllTaken the issue you're describing with Equals is a bug, filed separately as #18472. Right now the expected behavior would be to throw, since we don't translate that overload of string.Equals (this is what this issue is about). Use the == operator with ToUpper as needed to get the SQL you want.

Note that writing "EUR".ToUpper() isn't necessary since EUR is already an uppercase constant.

[AllTaken]

@roji OK. Great to see that a fix has already been merged.

Note that the call to TopUpper() on a const was merely for illustrative purposes, this is of course in reality a variable.

[wstaelens]

https://stackoverflow.com/questions/58640437/ef-core-3-0-translating-string-equals-ordinalignorecase-correctly

[Suchiman]

[Bartmax]

I think there must be a configurable collation (fluent and/or attribute) that makes it sure the collation is whatever case you want to be. Executing raw sql commands at migrations doesn't seem like good idea. Ef can see changes in collation an make the need for a new migration and everybody's happy.

Something like:

.SetCollation(Collations.SQL_Latin1_General_CP1_CI_AS)

or

SetCollation(charset, codepage, case, accent)

[wstaelens]

I think there must be a configurable collation (fluent and/or attribute) that makes it sure the collation is whatever case you want to be. Executing raw sql commands at migrations doesn't seem like good idea. Ef can see changes in collation an make the need for a new migration and everybody's happy. Something like: .SetCollation(Collations.SQL_Latin1_General_CP1_CI_AS)

or SetCollation(charset, codepage, case, accent)

I didn't know about Collation until I asked on stackoverflow. Yes it would be great to see something like SetCollation to make it friendly for the ones like me that never used this until 3 weeks ago.

[pkanavos]

@wstaelens you really don't want this - your query won't be able to use any indexes that cover that column. Collations are nothing new or unknown either. They affect comparisons and hence, indexing. If you try to use a different collation, the server won't be able to use any existing indexes, scanning all the data instead.

Besides, if the database designer specified that A1 and a1 are different values, why does the client try to treat them as if they were the same? Either the designer got it wrong, or the client got it wrong. In a Name, there's probably no reason for case-sensitive matching. In a column holding Base64 text, casing matters

[TrabacchinLuigi]

@pkanavos he said during a migration, he IS designing the database...

[upsilondynamics]

This needs to be implemented, I am using .Equals("test", StringComparison.InvariantCultureIgnoreCase) and it doesn't work anymore. It's putting a wrench porting my EF Framework app is.

Please prioritize this and get it put in ASAP.

[TrabacchinLuigi]

@upsilondynamics putting a wrench migrating? This wasn't working in ef6 too.. it was compared locally... Put a ToArray() before the comparison and you get the same behaviour... At least this is what I remember... Not in front of a pc right now...

[smarts]

x => x.Foo.Equals("bar", StringComparison.OrdinalIgnoreCase) or maybe x => StringComparer.OrdinalIgnoreCase.Equals(x.Foo, "bar") was working in some version of entity framework prior to EF Core, because it was while converting to EF Core that this problem manifested for me

[upsilondynamics]

@upsilondynamics putting a wrench migrating? This wasn't working in ef6 too.. it was compared locally... Put a ToArray() before the comparison and you get the same behaviour... At least this is what I remember... Not in front of a pc right now...

If it was working in EF6 then why did my LINQ queries work before I went to EF CORE? anyway I got around the issue by simple doing a toLower() and == now seems to work without issue.

[smitpatel]

@dotnet/efcore - We should review if EF6 did any of these translation to server and discuss if we should do it. And then close the issue. Ignoring argument is just a slippery slope which will always be wrong in some cases.

Using Lower create index miss so let users do that explicitly. Collations... let's not do this!