Allow query tags to be forced inside generated SQL to prevent SQL Server from stripping them from "query text entries"

[jzabroski]

https://github.com/aspnet/EntityFramework.Docs/blob/master/entity-framework/core/querying/tags.md

Based on helpful discussion with @ajcvickers

This might be somewhat controversial, but I think many people including @m60freeman and @nickcraver want very similar features: Easy way to scrap boilerplate, reduce dependency on copy-paste errors in log messages, etc.

I think the Query Tags documentation would benefit from a broader example that demonstrates how to use a self-contained data access layer (through a Repository pattern). The exact size of the sample code could start off small as people give their feedback, but the basic goal should be to give DBAs and engineers a language to discuss how to tag queries with pseudo-exact code locations.

My code sample would look something like that following:

class PersonRepository : IRepository<Person> {
  DbContext _dbContext;
  PersonRepository(DbContext dbContext) {
    _dbContext = dbContext;
  }
  private static string GetTypeAndMethodName([CallerMemberName] string callerName = null)
  {
    return $"{typeof(PersonRepository).Name}.{callerName}";
  }
  IQueryableResult<Person> All() { // assume IQueryableResult is an abstraction which blocks adding method chains to an IQueryable<T>
    return DbSet<Person>().TagWith($"{GetTypeAndMethodName()}"); // this will tag the query with "PersonRepository.All"
  }
}

[ajcvickers]

Note from triage: this is not something we plan to implement. The ability to appropriately tag queries is something that needs to be supported by SQL Server without hacks like this.

[m60freeman]

@ajcvickers I agree this would be a hack, but it is the only currently possible solution. Are you going to communicate with the SQL Server team about this in any way or is there a Microsoft policy that prohibits coordination between the product groups? Because they can't produce a feature that your group will use to resolve this serious disconnect without this being a joint effort.

[stap123]

@ajcvickers Is there an open feature request with the SQL team somewhere we can upvote?

[ajcvickers]

@scoriani Any thoughts?

[scoriani]

What’s exactly the ask here? Is it to create an end-to-end mechanism to track and troubleshoot performance of individual EF queries from code to query store? Or is it to just keep comments and spaces before and after a query for sql text in query store?

[NickCraver]

@scoriani I think the base level ask would be "a string of text to go with a query", e.g. think of a comment which is where the query was issued from. For us, it's "Path\Files.cs@129" (path, file, line), but it could be any identifying text. The main purpose is to back track a query in SQL (e.g. something running hot) to where it came from in an application.

[ajcvickers]

@scoriani The requirement here is a way to track a query all the way from the client to places where that query shows up in SQL Server diagnostics. EF would then use this mechanism. Right now, the only thing EF can do is add a comment to the query, but then SQL Server strips those comments out unless they are unnaturally embedded within the SQL somewhere.

[scoriani]

Got it. Haven't investigated on this for query store yet, but i can imagine we're stripping down comments to avoid storing multiple entries for the same sql text that in a busy system can bloat query store tables but, in this specific use case, it is very likely that a given query will be matched 1:1 with a given comment. I'll reach out to some query store folks and ask for more details.

[jzabroski]

@scoriani We're hoping for a corresponding GitHub issue as well, so the trail doesn't run cold.

[scoriani]

@jzabroski what do you mean? Unfortunately SQL Server is not developed in the open using GitHub, and the closest we have is https://aka.ms/sqlfeedback for customer asks, so i'm not promising anything here :)

[jzabroski]

:( I personally wish the heads at Microsoft would just shut down SQL Feedback and Visual Studio feedback and just use codeless GitHub repos for issues. Lots of companies do this already, very successfully. Maybe you are friends with some VIPs at Microsoft and can hint at this. It's shocking how bad the search engines are for VS Issues and SQL Feedback. - I don't even want to use these sites, so I barely bother reporting issues unless someone at Microsoft asks.

[scoriani]

Ahh... that's way above my pay grade :) but a group of us repeatedly proposed that already over time. Things are slow in large orgs, and VIPs are not necessarily the issue, but we do not give up ;-)

[jzabroski]

@scoriani I think my simplest feature request is:

As a database administrator/devops engineer, I want to disable comment stripping from the Query Store, so that I can embed a comment at the beginning of a SQL statement to track queries back to the application they originate from.

@NickCraver I suppose one alternative, that I don't particularly care for, is to use OpenTelemetry .NET's Enrich operation. The main reason I don't like it is more that it looks unnecessarily complex and I have not really played around with it. But there is an EFCore OpenTelemetry plug-in that maybe would be useful to us. https://github.com/open-telemetry/opentelemetry-dotnet/pull/929

[m60freeman]

@scoriani We need a way for EF's .TagWith() to be able to "decorate" generated SQL in a way that will remain intact in Query Store (and hopefully the Plan Cache as well). We have proposed something like "SELECT ColumnName / CommentFromTagWith / FROM TableName" to work around comment stripping (which ignores this comment placement), but the EF Team feels this is too much of a hack and they are looking for some more elegant/official way to accomplish the goal.

The goal is to be able to determine from the SQL text in Query Store where the SQL was generated, whether that be Class.Method or Path\File@LineNumber. A DBA sees the SQL text for something that is performing poorly or producing an incorrect result and wants a developer to change it, but currently the developer has no way to find the code that needs to be changed other than by doing some kind of RegEx search through the entire code base, which is inefficient and error prone.

I am guessing that the EF Team would prefer being able to send something like "/ CommentFromTagWith / SELECT ColumnName FROM TableName", but @ajcvickers would need to clarify that.

Currently in Query Store, we get a value in sys.query_store_query_text.query_sql_text that looks like: "(@ToGuid_0 uniqueidentifier)SELECT ColumnName FROM TableName WHERE (Id = @__ToGuid_0)". Note that the parameter list is "decorating" the SQL. Maybe there could be CommentFromTagWith between another set of delimiters immediately after the parameter list? That might look like "(@ToGuid_0 uniqueidentifier)(CommentFromTagWith)SELECT ColumnName FROM TableName WHERE (Id = @__ToGuid_0)". If there is no such "tag", then omit the empty delimiters the same way the current ones are omitted if there are no parameters. Maybe use [] or {} for this instead of () to differentiate them and to avoid breaking existing scripts that would be broken by having an "(" followed be something other than a parameter declaration at the start of the text when there are no parameters but there is a "decorating" comment.

The SQL Team and the EF Team must collaborate to find a solution that will be acceptable to both and will work in combination. Having the EF Team add comments that the SQL Team strips out is what we want to avoid.

[scoriani]

SQL takes batches of 1 or more statements. The batch does not strip anything, and the sql_handle is effectively an md5 hash of all of the characters (comments included). That's why in plan cache comments are already there and collected by external tools like SolarWind and such.

Query store has a statement sql handle. It is defined based on the first character of the first token of the statement and the last character of the last token in that statement. So, comments between statements are ignored for how the query store defines what is a statement. The statement sql handle is also an md5 hash over the characters. Adding a space or additional characters like comments makes it look completely different to the system from a caching perspective (batch) or plan forcing perspective (statement or batch). So, you don’t want adding text to a comment block to undo all of the history and forcing in the query store, so it isn’t counted. This would have way to many implications and impact on a number of features.

Also, adding multiple entries in sys.query_store_query_text for the same query is not viable as memory management with query store is a major concern for busy systems or in Azure SQL where resources are generally more constrained.

Based on the conversations with some team members, it's very unlikely that this behaviour is going to change. General recommendation would be to inject tags/comments within the statement text instead.

[jzabroski]

@ajcvickers Is this a stalemate?

[ajcvickers]

I think this discussion has become fixated on comments. The requirement as I see it and stated above is:

The requirement here is a way to track a query all the way from the client to places where that query shows up in SQL Server diagnostics. EF would then use this mechanism.

I went on to say:

Right now, the only thing EF can do is add a comment to the query, but then SQL Server strips those comments out unless they are unnaturally embedded within the SQL somewhere.

The fact that SQL Server sees comments as something that its reasonable, and sometimes even necessary, to strip out indicates that comments are not the correct way to do this. There should instead be a different, end-to-end mechanism that allows a query to be traced from source code to diagnostics.

[jzabroski]

@scoriani Is it possible to go back to the team once more and request some end-to-end mechanism that allows a query to be traced from source code to diagnostics? I now see why @NickCraver suggested "a string of text to go with a query" after reading @ajcvickers 's reply directly above.

Also, based on your explanation of the internals, it sounds like it is not even feasible to have a Query Store option to not strip out leading comments, since Query Store only knows sql statement handles. e.g., it sounds like the following would not be possible:

ALTER DATABASE AdventureWorks2014
SET QUERY_STORE = ON   
    (  
    STRIP_COMMENTS = OFF
    );

The reason I was asking is cost to implement a feature. But I agree that "a string of text to go with a query" is probably the best way to provide an "end-to-end mechanism that allows a query to be traced from source code to diagnostics."

[m60freeman]

@scoriani Adding an additional NVARCHAR column to sys.query_store_query_text to hold the TagWith data from EF would be fine with me, but there would have to be a way to EF to send that information in such a way that Query Store would pick it up. Ideally, it would be something that would also get picked up by the plan cache as part of the batch, but if not it would give people another reason to enable Query Store.

Otherwise it looks like the "hack" of having EF inject a comment just after the first token of the statement is the only thing that will work. This would change the md5 hash (once), but I really don't have a problem with that. It would just have to be documented in the new EF feature that enabling it will cause a new query_hash to be generated and there would be no way to link the old and new hashes.

[scoriani]

Sounds to me that there's the need for much more detailed framing/scoping for such a feature, including key use cases and scenarios. Is it a single query that needs to be traced end to end? Or is it a client command/unit of work that may generate multiple queries in a single or multiple batches? Should we trace a single query execution or aggregated metrics? Remember Query Store only aggregated query execution stats over a time period, not individual query executions. Why EF and not underlying client drivers and other data access frameworks? What about integrations with other observability tools/frameworks/solutions like OpenTelemetry, Dynatrace, Datadog, Redgate, etc.? While i can see the value of such a feature, it will impact many areas inside and outside the engine and will need to be properly evaluated and prioritised together with other product improvements.

[m60freeman]

@scoriani The main issue is the EF/LINQ generates queries that are often so long and deeply nested that their text is truncated before even getting to a WHERE clause. Given a FROM and a WHERE, application developers have much less work to do to find where a query is generated. I am not aware of other frameworks that produce code so convoluted and obscured by unhelpful aliases. No one in this thread has asked for anything in the way of metrics other than what is already provided. We just need to know where in the code a query originated.

Having this information available in the plan cache (a new tag in the plan XML and a new column in sys.dm_exec_sql_text?) would make it available to monitoring tools that capture from there and not Query Store.

Speaking for myself, I would much rather see a narrowly-defined feature that can be implemented relatively simply and quickly than something that will handle every possible use case. The information we need in SQL Server is clear from reading this thread. The tricky part is defining how to get it there in a way that the EF team is willing to provide.

[ajcvickers]

@scoriani

Is it a single query that needs to be traced end to end? Or is it a client command/unit of work that may generate multiple queries in a single or multiple batches?

Single query.

Should we trace a single query execution or aggregated metrics? Remember Query Store only aggregated query execution stats over a time period, not individual query executions.

That's a good question--maybe @NickCraver has an idea of what would be best here?

Why EF and not underlying client drivers and other data access frameworks? What about integrations with other observability tools/frameworks/solutions like OpenTelemetry, Dynatrace, Datadog, Redgate, etc.?

This should definitely not be an EF-specific thing. It should be something usable by any client. EF would make use the of the underlying mechanism just like any other client.

While i can see the value of such a feature, it will impact many areas inside and outside the engine and will need to be properly evaluated and prioritised together with other product improvements.

I totally understand that the value of implementing this might not be worth the cost it takes to implement, both in terms of resources, and in terms of impact on the code or performance. That's fine, and a statement as such from the SQL team would put this issue to bed as not something that SQL Server supports.

[jzabroski]

I'd just chime in to say that I think @m60freeman 's point that a lot of the tracing value here relates to impossibly long SQL queries generated by ORMs to be fairly accurate.

Thus, I don't think aggregated metrics matter that much. The only use case for aggregated metrics is when you have nested transactions, because the inner transaction does not record wait types and yet the outer transaction can cause blocking due to session-level resource tracking. Thus, tools like SolarWinds DPA show a 0 cost on things that actually have a high aggregate cost. But, that's what session-based diagnostic tools like SQL Sentry Profiler help with already. Aggregate metrics is more or less a solved problem in my mind. Could it be smoother? Yes. Do I need it to be smoother? No.

User stories I've cared about in the past:

1. SELECT FROM statements without a where clause; root cause is C# developers don't understand LINQ query execution semantics and typically arises when they try joining a SQL table to an in-memory collection. Finding the bad query is actually fairly hard to do.
2. Somebody writes an overly complicated GROUP BY clause that uses object equality, rather than simpler field equality (perhaps taking advantage of a unique constraint) and possibly using a cross apply instead of a group by. I need to find where that query is.
3. SQL Server suddenly chooses a bad query plan for an EF query. I want to be able to peg the old plan ASAP. (This is a really high value item for me. If Postgres had such a feature, I would never go back to SQL Server... especially now that Amazon MariaDB has a T-SQL engine)

For 1 and 2, one way EF Core team could be helpful is in strictly promoting the best practice of repositories that don't leak IQueryable. Code bases have an insane amount of technical debt when you have bad code + layering violations, because it makes it so time consuming to track this non-sense down. As I stated in 2020, I don't leak IQueryable any more because of the maintenance nightmare it becomes tracking down poor performing SQL. To draw an analogy, the ASP.NET Core team outlawed certain types of Dependency Injection patterns, like SimpleInjector/NInject allowing use of Func<T> to create ambient object instances from an implicit "current scope". It would be a net benefit if EF Core query examples didn't just use a DbContext but used a repository abstraction that didn't leak IQueryable, but transformed that into something representing a "promise to return a collection" (VectorResultSet, whatever you want to call it).

[roji]

@jzabroski I suggest keeping the conversation well-scoped: EF documentation guiding against leaking IQueryable (or specific problematic GROUP BY querying patterns) don't seem directly relevant to what is being asked here.

At the end of the day, when using an ORM such as EF, queries are expressed in code via means other than SQL (in the EF case LINQ is used). This raises the problem of correlating those LINQ queries with SQL sent and executed in the database, when is important for tracing problematic SQL queries back to their originating LINQ queries. EF's TagWith feature is currently implemented by injecting a comment into the generated SQL, which allows correlating the SQL in EF's logging, but not in SQL Server logs/diagnostics (since the comment is stripped). Another SQL Server mechanism for providing the tag (AKA correlation info) would allow EF to send that via another mechanism.