Events and interception (aka lifecycle hooks)

[divega]

Done in 2.1

• [x] Add events for entity state changes #10895

Done in 3.1

• [x] Add IDbCommandInterceptor or similar to allow injecting custom query execution service #15066
• [x] IDbConnectionInterceptor and IDbTransactionInterceptor

Done in 5.0

• [x] Implement events for before and after SaveChanges #15910

Done in 6.0

• [x] Interception: raw query vs. LINQ #22899

Done in 7.0

• [x] Implement ObjectMaterialized event #15911
• [x] ChangeTracker: New event for "entity about to get tracked" #27093
• [x] Don't check for a connection string until after ConnectionOpening has been called #23085
• [x] Add interception event to be fired when EF has finished consuming a result set #23535
• [x] Utility of DataReaderDisposing logging/interception #24295
• [x] Please consider adding two methods to the DbConnectionInterceptor Creating/Created #23087
• [x] Consider delaying DbCommandInterceptor.CommandCreated until DbCommand properties are intialized #17261
• [x] Interception for DbUpdateConcurrencyException #28315
• [x] Interception to modify the LINQ expression tree #28505
• [x] Provide event that can be used to get notifications when all property changes have been detected #26506

Backlog

• [ ] Add interception for transaction enlistment #16261
• [ ] Add interception for execution strategies #16257
• [ ] Add interception for batches #16260
• [ ] Implement interceptors for Cosmos #26478
• [ ] Add an interceptor to an existing DbContext instance #20273
• [ ] Ability to register IInterceptor without an IDbContextOptionsExtension #21578
• [ ] Interceptors for SQL generation #19748

• [ ] Interception for model building #31206

Note: below is a copy of a very old EF specification and reflects thinking from several years ago. A lot of things aren't valid anymore.

We define EF Core lifecycle hooks as the general feature that enables an application or library to sign up to be invoked or notified whenever certain interesting conditions or actions occur as part of the lifecycle of entities, properties, associations, queries, context instances, and other elements in the Entity Framework stack.

For example:

1. An application can provide a method that will be invoked automatically whenever an object is about to be saved, or it can subscribe to an event that fires when an object is created and its properties initialized, etc.
2. A framework extension can register an interceptor that gives it an opportunity to rewrite query expression trees before they get translated by EF. This could be used to validate whether a user has access to specific information or to filter query results based on per DbContext filter (see #6440).
3. Execute SQL after a DbConnection is opened (to use features such as SQL Server App Role)

The need for lifecycle hooks

We want to enable customers to write business logic that triggers in the different stages of the lifecycle of these objects, following well factored coding patterns. We also want framework writers to be able to use these hooks to extend EF Core in useful ways.

In previous versions of Entity Framework we already exposed a few lifecycle hooks. For instance, we had the AssociationChanged and ObjectStateManagerChanged events since the first version, and the ObjectMaterialized event was added in EF4. Up until EF6.x many of the existing hooks are not exposed in the DbContext API. In EF6 we also added several low level extensibility points in Interception that can be used too as lifecycle hooks.

There is a continuum of capabilities related and overlapping with lifecycle hooks, e.g.:

• We want to improve everyone’s ability to diagnose functional and performance issues with their code using Entity Framework by recording information about interesting events and conditions in the EF stack, but that is essentially logging (tracked in #218 and other work items). By comparison, lifecycle hooks allows to modify the outcome of these events.
• Also the DI-based architecture of EF Core allows for a very granular capability to wrap and replace individual implementation services, which can potentially be used to extend EF and to execute business logic. By comparison, lifecycle hooks is about adding simple and first class hooks that can easily be used to react to interesting conditions without having to re-implement the complete interface of a service.

  Target customers

  1. Application developers that need to implement business logic that is triggered at certain points in the lifecycle of an application, for instance, before an added or modified object is saved to the database, just before loading the contents of a navigation property, etc. Someone could add an OnValidate method to an entity that gets executed right before the entity is stored.
  2. Framework/tool writers that need to extend the behavior of Entity Framework to support new application scenarios or to integrate it with other products or frameworks can write their own frameworks on top of EF that customize the behavior, collect data about the execution of specific actions, etc. Someone can write a full-fledged profiler taking advantage of the hooks.

     Goals & Principles

  3. Should work well and be consistent with DbContext design principles
  4. Should have a consistent story on when and how it is allowed to re-enter: With a very small set of rules users should be able to predict what is allowed and what is not. The stack should be resilient to reentrancy in some cases and throw good exceptions if reentrancy occurs in other cases. We should prevent doing the same work more than once.
  5. Should have minimal performance impact, especially when hooks are not being used the impact should be insignificant.

     A brief survey of hooking mechanisms

     There are not only different interesting conditions or actions an application may need to listen to, but also different kinds of mechanisms to implement hooks that present distinctive characteristics along the following dimensions:

  6. Compile-time vs. run-time binding
  7. Performance
  8. Cancellability
  9. Override/customization of default behavior
  10. Single vs. multi-cast
  11. Simplicity
  12. Discoverability
  13. Familiarity Note: We are looking for criteria that will help us choose the best type of hook for each extensibility point, as well as for the possibility of defining unified hook mechanisms that we can leverage to support different patterns with the same framework code.

      .NET Events

      Events are the most common hook pattern that almost every API in .NET uses. Events are messages sent by sender object to one or more receiver objects through a multicast delegate that acts as a dispatcher. Among the characteristics of events, they support runtime subscribe/unsubscribe, multiple listeners, and are relatively easy to use. Events can be slower than other hook mechanisms, but they have the advantage of being very discoverable (they are usually public members on the sender object) and familiar to customers. Events also provide a standard way to model cancellable actions with CancelEventArgs.

      Virtual methods

      Virtual methods require the application code to declare a derived type and override the method. Virtual methods provide better performance than events but are slower than regular method invocation. Virtual methods are easy to discover and provide a very nice model for overriding/customizing default behavior and chaining with subsequent derivate types. Visual Studio provides a nice Intellisense experience for virtual methods: when you write the overrides keyword in C#, Intellisense provides the list of all the virtual methods available.

      Delegates

      A more efficient alternative to events, regular (non-multicast) delegates can also be used as a hook. Users can normally provide some implementation of a predefined delegate signature (usually a Func<T…> or Action<T…> that can be implemented as a regular method, and anonymous method or a lambda expression) as a parameter to a framework method, as the return type from a method or as a property. Then the delegate is invoked by the framework at appropriate times. While delegates are often compared to strongly typed function pointers, they in fact are very flexible with regards to the signature (they support variance).

      Partial methods

      Partial methods were introduced in .NET 3.5 as a means to extend generated code in separate partial classes. Partial methods are void methods that are both defined and invoked in the right places in generated code. Users can choose to provide the implementation of partial methods in a separate partial class, and the compiler will resolve the partial method to the implementation provide by the user. When the implementation of a partial method is not provided, all calls to the method and its definition are removed by the compiler. Since partial methods are either turned into regular methods or removed, the mechanism is extremely efficient. Partial methods are discoverable because Visual Studio provides a nice Intellisense experience for them. Similar to virtual methods, once you write the keyword partial inside the partial class, Visual Studio editor will list all the partial method definitions that haven’t been implemented.

      Magic methods

      The concept of magic methods is that the user can write a method that follows a particular naming convention and signature, and a framework component will make sure the method will get invoked automatically at runtime. Usually an expression is compiled at runtime to produce a delegate that be used to invoke the method multiple times very efficiently. There is some runtime overhead in compiling the expression, but this is paid only once. Magic methods can be instance or static method. A common practice for magic methods, when code generation is involved, is to provide the declaration of the magic method as a partial method. Although no actual calls are generated, the partial method that gets an implementation will be compiled into the assembly so that it can be invoked at runtime. The only reason for this is the Intellisense experience you get. Magic methods can be used to override default behaviors but it is necessary to expose a public method that implements the default behavior so that the user has the option to invoke this method from within the magic method if he doesn’t want to completely override it. LINQ to SQL uses this mechanism pervasively.

      Attributed methods

      Similar to magic methods, an instance or static method with the right signature can decorated with a special attribute that specifies a runtime role for this method. The runtime examines types for the presence of these attributes and registers the method to be executed on the occurrence of certain conditions. WCF Data Services uses this pattern pervasively.

      Listener interfaces

      This approach consists on defining a class that implements a custom interface provided by the framework and at a later point register an instance of this class as a listener for particular events. Usually, the interface defines one or more methods with a very specific purpose, but the listener class can be a composite of multiple interfaces. Discoverability can be improved in this programing model with a base listener interfaces and by placing all the related interfaces under the same namespace.

      IObservable<T>

      In many situations a mechanism is required to handle a stream of asynchronous events coming from the same source. IObservable<T> is an analog to IEnumerabe<T> that can be used to represent this kind of source.

      Context hooks vs. entity and property hooks

      DbContext provide a very good place for us to focus when defining extensibility hooks for anything that has to do with the functions that they encompass:

  14. Store Connection
  15. Launchpad for queries
  16. Unit of work
  17. Update adapter But for hooks that are specific to the lifecycle of entity or complex types, properties, etc., other options exist:
  18. Factor business logic into the entity type itself: The typical example of this is the OnValidate method that LINQ to SQL supports: a user can define the method on a particular entity type, and have the framework will automatically invoke the method at appropriate times.
  19. Extend the Code-First API to support hooks: The Code-First API provides a nice central configuration point for entity types, complex types and properties. We could extend the API with methods to register listeners, e.g.:

      mb.Entity<Product>().Notify(new EntityLoadedListener());
      mb.Entity<Customer>().Property(c => c.Orders).Notify(new PropertyLoadedListener());

      The one potential issue with this approach is that the hook configuration would become part of the model and therefore it would not be possible to change it once the context object has been instantiated. This might be an acceptable limitation however, since any necessary changes in behavior can be coded into the listener class itself. Making the configuration of hooks immutable also has the advantage of allowing for compiling the invocations to listeners into efficient delegates.

      EF requirements for a hooking mechanims

      We are trying to find a design that has the following characteristics:

  20. Works well for both app developers and framework developers.
  21. Makes hooks very discoverable.
  22. Invocation is efficient.
  23. Requires only very simple coding patterns to use.
  24. Should allow get the hooks at runtime and from a separate assembly.
  25. Hook handlers can be implemented outside the entity types.
  26. Hook handlers can be implemented as part of the entity types.
  27. Ideally, the way you implement a hook handler in an entity shouldn’t break POCO.
  28. The complexity of supporting multiple hook mechanism should be completely hidden from the sender. This requires further analysis, but it seems that it would be reasonable to support a mix of hook mechanisms (but not too many) to optimize for different scenarios. The current thinking is that we would use a single generic class to represent a hook. The sender should just need to call a method or instantiate a class and call a method, nothing much more complex than the typical “On[EventName]” pattern, and the hook class would make sure all the hook handlers are invoked. We are also considering building a convention system for wiring up hook handlers in the entity types. When designing this we should take advantage of any opportunity to reuse some hook mechanism as building blocks for others. For instance:
  29. If we decide that the basic mechanism is listener interfaces, then we can have context objects implement those interfaces and re-cast the hooks as events.
  30. We can have a single convention system that understands about naming patterns and attributes (similar to how the Code-First convention system understands naming patterns and data annotations as different vocabularies to express the same kind of concepts).  

      Lifecycle hooks list

      The following is an incomplete list that presents various hooks we could consider adding. Intentionally the list does not try to be specific on each hook about certain details:

  31. Hooking mechanisms
  32. Method / argument signatures
  33. All names of new hooks are up for discussion

Name	Pri	Location	Cancel or override	Description & sample scenario
QueryExecuting	0	DbContext	Yes	Query interception, custom query caching.
QueryExecuted	3	DbContext	No	When Execute happened, before the reader is read. Tracing?
QueryCompleted	3	DbContext	No	After DbDataReader is closed. Tracing?
EntityStateChanged	0	DbContext	No	Signals all state changes
EntityStateChanging	3	DbContext	?	Undo changes or change proposed values before they are set?
ConnectionProvisioning	2	DbContext	Yes	Execute additional code to make sure the connection is alive, or do logging
ConnectionReleasing	2	DbContext	Yes	Cleanup something done during Ensure / StartUsingConnection
ConnectionOpened	2		No	More likely for tracing. Since SqlClient has fixed invalid connection pools, then this is lower priority
ConnectionOpening	1	DbContext	Yes	Slightly simpler to use than Ensure/Start, would not require user to check current state. Could also be used for tracing.
ConnectionClosed	1		No
ConnectionClosing	1	DbContext	Yes	Slightly simpler to use than Release/Stop, would not require user to check the initial state. Could also be used for tracing.
OnModelCreating	0	DbContext	Yes	Tweak model before it is cached.
OnModelCreated	1	DbContext	Yes	Signal that the model is done and execute some custom code, possibly related to caching logic. . Issue: do we need this for ObjectContext? Issue: if the user is going to implement his own caching, we should have an abstract class or interface for that.
ModelCacheLookup	2	DbContext	Yes	Implement your own caching logic. Tracing?
ModelCacheHit	2	DbContext	Yes	Execute additional code when the model is found in the cache. Tracing?
EntityLoading	1	DbContext, DbEntityEntry	No	After object instance is created but before its properties are initialized. Can be used to reset a flag that will be set in newly created instances but shouldn’t be set during initialization, i.e. for validation.
EntityLoaded	0	DbContext, DbEntityEntry	No	Can be used to setup anything after an object has been materialized, i.e. event handlers, flags, etc.
CollectionLoading	1	DbContext, DbEntityEntry DbCollectiohnEntry	No	Can be used to setup anything on a collection after it is created but before it is populated. Issue: Could be used to provide your own collection?
CollectionLoading	1	Context, Entity or Collection	No	Can be used to setup anything on a collection after it has been created and populated, i.e. listeners for its changed event.
ObjectTypeResolving	1	Context	Yes	Could be used to specify a different type than the original one, i.e. to implement your own proxy mechanism. It should be per type but could return a Func<T> that returns a new instance and the result could be compiled into materialization delegates.
CollectionTypeResolving	1	Context	Yes	Something similar to ObjectTypeResolving but for collections. Could be used to replace the default collection type with a custom proxy collection with additional functionality (i.e. paging, fine grained lazy load).
Virtual OnSavingChanges	Medium	DbContext	No	Can be used to re-implement SaveChanges but still invoke the existing SavingChanges event
SavedChanges	Low	Context	No	Could be used to execute cleanup code after SaveChanges. For instance, to call AcceptChanges on each STE change tracker. It is lower priority because virtual SaveChanges covers most scenarios.
EntityStateChanging	Low	Context, Entity	Yes	For an entity instance or type in particular we could avoid putting in the modified state. So even if the properties are read-write, the context ignores changes to this entity. Could be also used to suspend fixup on an entity that is being detached.
EntityStateChanged	High	Context, Entity	No	Executes logic after an entity has been put in a certain state. Can be used to setup property values, restore state after the changing event.
PropertyChanging	Low	Context, Entity	Yes	Any time a property is about to be changed by the framework or any party, if notification or interception is enabled by the entity type. Should make original and new value available. Should also work for navigation, scalar and complex types properties. Tracing?
PropertyChanged	High	Context, Entity	No	Any time a change in a property value change is detected.
PropertyLoading	High	Context, Entity, Collection	Yes	Intercepts, overrides de loading of a property. Could be used to support loading of properties using stored procedures.
PropertyLoaded	Medium	Context, Entity, Collection	No	Tracing?
Writetable IsLoaded	High	Context, Entity	Yes	Allows cancelling the loading of a property.
CollectionChanging	Medium	Context	Yes	Any time a collection is about to be changed by the framework or any party, if interception is enabled
CollectionChanged	Medium	Context, Entity	No	Any time a change to a collection has been detected.
AssociationChanging	Low	Context, RelatedEnd	Yes	Can be used to prevent an association from being changed, or to execute business logic when the association is about to change.
AssociationChanged	Medium	Context, RelatedEnd	No	Can be used to execute additional logic after an association is changed, i.e. user can explicitly cascade relationships removals into dependent removals, workaround current databinding shortcomings.
RowValidate , RowValidateAdded, RowValidateModified, RowValidateDeleted	Medium	Context	Yes	Storage level version of ObjectValidate. Tracing?
SavingChanges event	0	DbContext	?	Currently only available on ObjectContext. Should make trigger OnSavingChanges method protected.

Existing hooks

Name	Description & sample scenario
virtual Dispose	This can be used to do additional cleanup, i.e. on entity instances.
virtual SaveChanges	Can be used to execute additional logic before, after or instead of saving changes.

Some open issues:

1. Need to prototype some coding patterns and try them.
2. Is logging and tracing part of this API? It seems that ideally we should have the same level of flexibility for hooking mechanisms with logging and tracing as we end up having with this API.
3. Second level cache should probably expose its hooks through the same mechanisms.
4. Should we provide low level query interception points with the same mechanisms, i.e. as command tress and store commands? Should we do the same for CUD store commands? Would need to make sure those work well with caching.
5. Can we get some level of support for async execution of queries with this hook?
6. Should we provide enough lifecycle hooks to implement custom fixup logic?
7. Areas of overlap with other extensibilities: read and write properties in object mapping, proxy type creation (can be imperative vs. event driven), equality and snapshot comparisons for change tracking extensibility.
8. What about customizing identity resolution?
9. Is Logging and Tracing part of the lifecycle hooks
10. Is Query interception part of the lifecycle hooks
11. Even without query interception we should expose when we are about to execute (imagine a profiling tool that measures how much query compilation costs).
12. Is ContinueOnConflict part of the lifecycle hooks
13. How do we improve diagnostics? Can we have OnError
14. Need to do prioritization, costing and scoping
15. Should we have a fine grained version of CollectionAdding / CollectionRemoving with support for magic methods on the entities to enable collection patterns? We would need a pattern for Contains checks also.
16. There is a conversation about splitting AssociationChanged this event into properties and collection changes. However, there should be a way to tell the difference between a scalar property change and a nav prop. Should we make AssociationChanged more accessible and add AssociationChanging? This would provide a way to intercept changes in associations independently of cardinality, constraints, etc.
17. AssociationChanging would need the entity and collection types to collaborate to avoid changes from being made to the graph.

[ajcvickers]

@sidshetye Thanks for the feedback--I will certainly pass it on to those who make the calls on the release dates.

[ajcvickers]

For anyone following this issue. in order to help us prioritize work can you let us know which hooks are important to you and what you intend to do with the hooks?

[dazinator]

@ajcvickers During a save operation of a business object, my parent / root business object typically calls SaveChanges() but only after passing the DbContext instance to all its child business objects in the graph, so that they can add / update / delete entities on the DbContext in order to take part in the transaction. Once the entire business object graph has worked with the DbContext instance, the root business object will call SaveChanges(). At that point, the database can generate values for all of those persisted entities. I would like for my Child business objects in the graph to be able to register their interest in entities that they have added / updated, so that after SaveChanges is called, they can get notified, and get access the new state of the entity now that it has any database generated values (such as primary keys etc) in order to update their own state to match.

[popcatalin81]

A very useful use case of Lifecycle hooks is to implement Revisions (I've used this in several applications with EF6)

How it works: when a user edits an entity IE: Post, during save the EntityState is changed from Modified to Added and the RevisionNr is increased. This way as simple workflow, Load, Edit, Commit is used (that does not require copying object values and adding them as new) to create a Revisions based system that always inserts new revisions (instead of updating the original entity).

A second use case is to compute complex properties eagerly. Like decode a binary blob and update a specific graph (In cases where this cannot be done lazily), IE: loading complex scheduling rules and updating a scheduling graph (this will be done eagerly)

A third use case is Detailed Change Auditing (usually the more enterprisey the project, the, more this is required)

[nphmuller]

Our use case is to implement global query caching. Basically we'd need a hook to capture a read query just before its executed. If the key does not exist in the cache the query will be executed on the database and the result will be cached.

The other hook we'd need is when a transaction is committed (or rolled back), so we can invalidate the proper items in the cache. Overriding SaveChanges() will probably also work instead of this hook.

See also: https://github.com/aspnet/EntityFrameworkCore/issues/5858#issuecomment-306441806

In EF6 we used https://github.com/moozzyk/EFCache and we implemented our own Redis provider for this.

[markusschaber]

We have some "StatusService", where clients can subscribe (via .NET events or WebSocket) so they get notified when something changes. This includes some hysteresis / rate limiting of notifications. Part of the changes are changes to (a subset of) our database entities, so they can react accordingly (e. G. update their UI, or schedule synchronization of some data, etc...). So an "this database entity has changed" event is what we'd need. (Currently, we do this via overwriting SaveChanges[Async]() Methods which triggers events on an injected singleton service, and subscribers use the dbcontext ChangeTracker to check which entities they're interested.)

[codetuner]

We have several uses for them: 1) In the OnSaving event (which we trigger from an overloaded SaveChanges) we update simple tracking information (UpdatedBy, UpdatedWhen,...). 2) A more advanced version of the tracking logs old and new values of changed properties in new entities. 3) Some objects have complex graphs of subobjects associated which we prefer to store as a JSON string in a single database field. We then use both OnObjectMaterialized and OnSaving to deserialize/serialize the JSON string and populate the entity. I have described this solution in detail on https://www.codeproject.com/Articles/1210354/Hybrid-Storage-relationalplusdocument-with-Entity 4) Though there should probably be a better solution for this one, but we capture the OnSaving event to throw an exception when an entity(type) is considered read-only and has a Changed state.

[markusschaber]

@codetuner For your use case 3 (JSON): https://github.com/aspnet/EntityFrameworkCore/issues/4021 may help there, and I could also envision a different implementation (with conversion logic in the Entity) which works without overriding those methods or LifeCycle hooks.

[michaelaird]

We have 2 cases

1. Database.Connection.StateChange - we hook into Connection.Open to issue a command to set the current TenantId for multi-tenancy.
2. ObjectContext.ObjectMaterialized - we hook into the loading of certain object types so we can call DateTime.SpecifyKind to specify that datetimes are in UTC (instead of Unspecified)

1 is really our primary concern, 2 is a nice to have.

[sjb-sjb]

Per my posts of Jul 13 and Dec 4, I am using hooks on the DbContext to detect entity state transitions. The ask is to include ILocalViewListener or similar in the public API.

I use the hooks primarily for two purposes

(a) to inform an observable list of entities of a change in entity state, so that the list can be maintained appropriately (e.g. entities added or removed from the selected set of entities). The observable list is being used as a view model, more or less like a customized LocalView but not attached to a DbContext. It is a free-standing list of entities that is being accessed by the end user. Individual units of work are sent in from various parts of the program and the list provides an integrated view of the resulting entities.

(b) to inform the view model of an individual entity that there has been a change of entity state. This is being used in an entity view model system along the lines of the one in Template 10 but including entity state information. When a property is changed in an entity, a unit of work is opened and the view model reflects a modified status for the property; when SaveChanges is used to complete the unit of work, the entity state transition hook is used to inform the view model that the property is no longer dirty (i.e. the value has been updated). This is very convenient because the callback provides exact information about the specific entity that was changed (or whose changes were rolled back), and there is no tight coupling between the part of the program that submitted the SaveChanges and the part of the program that is using the entity view model.

Broadly speaking, hooking into the entity state change provides a way of driving view model updates and maintaining a single consistent internal state (a) without having to explicitly tie together the receiving view and view model with the UI or other program code that was used to generate and submit the entity changes and (b) without having to reproduce the logic that EF uses to track state and to commit or rollback changes.

These uses are somewhat similar to the objective that @markusschaber cited ("changes to (a subset of) our database entities, so they can react accordingly e.g. update their UI, or schedule synchronization of some data") and to the use that @codetuner described ("logs old and new values of changed properties in new entities") although we are updating UI instead of logging.

Note: in this application, concurrency contention is low so it is OK for a copy of the data to stay around in the app for a while.

[SidShetye]

We need these/equivalents from ObjectContext ...

1. ObjectMaterialized
2. SavingChanges

And preferably another hook after SavingChanges completes (maybe SavedChanges ?) Hope to see these soon!

[markusschaber]

@michaelaird I think a general option to specify whether Timestamp columns are UTC or local would be a better alternative here, what do you think? (Maybe this should be discussed somewhere else?)

[michaelaird]

I think they should deprecate DateTime and ‘bless’ NodaTime (the way they did with Json.net). But that is definitely a discussion for somewhere else.

That said we jump through hoops to translate nodatime types to datetime for storage so a mechanism to define those translations at the dbcontext would be hugely useful.

On Fri, Jan 19, 2018 at 3:02 AM Markus Schaber notifications@github.com wrote:

@michaelaird https://github.com/michaelaird I think a general option to specify whether Timestamp columns are UTC or local would be a better alternative here, what do you think? (Maybe this should be discussed somewhere else?)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFrameworkCore/issues/626#issuecomment-358893510, or mute the thread https://github.com/notifications/unsubscribe-auth/AANX6YLjFey2wq0V4r-xY9rCQwM30d9qks5tMEwegaJpZM4Cesel .

[javiercampos]

@michaelaird you can workaround point 2 with something like:

public static class DateTimeMapper
{
    public static DateTime SetUtc(DateTime value)
    {
        return DateTime.SpecifyKind(value, DateTimeKind.Utc);
    }
}
public class MyMaterializerSource : EntityMaterializerSource
{
    private static readonly MethodInfo SetUtcMethod = typeof(DateTimeMapper).GetTypeInfo().GetMethod(nameof(DateTimeMapper.SetUtc));
    public override Expression CreateReadValueExpression(Expression valueBuffer, Type type, int index, IProperty property = null)
    {
        if (type == typeof(DateTime))
        {
            return Expression.Call(
                SetUtcMethod,
                base.CreateReadValueExpression(valueBuffer, type, index, property)
            );
        }
        return base.CreateReadValueExpression(valueBuffer, type, index, property);
    }
}

And when setting your DbContextOptions, on your DbContextOptionsBuilder:

options.UseSqlServer(/* ... */);
options.ReplaceService<IEntityMaterializerSource, MyMaterializerSource>();

I agree it could be more user friendly with a hook like ObjectMaterialized, but it works.

For point 1, you can pass your own connection if you don't want the DbContext to create it for you and handle StateChange on it (for example, with UseSqlServer, using this overload), for example:

var myConnection = new SqlConnection(connectionString);
myConnection.StateChange += (s,e) => { /* */ }
/* then, in your options builder */
options.UseSqlServer(myConnection);

[javiercampos]

@sidshetye just out of curiosity, what would you need on a SavingChanges hook that can't be done by overriding SaveChanges/SaveChangesAsync?

You could implement that easily, with something like:

public class MyDbContext : DbContext
{
    public event EventHandler SavingChanges;
    public event EventHandler SavedChanges;

    public override int SaveChanges(bool acceptAllChangesOnSuccess)
    {
        SavingChanges?.Invoke(this, EventArgs.Empty);
        var returnValue = base.SaveChanges(acceptAllChangesOnSuccess);
        SavedChanges?.Invoke(this, EventArgs.Empty);
        return returnValue;
    }

    public override async Task<int> SaveChangesAsync(bool acceptAllChangesOnSuccess, CancellationToken cancellationToken = default(CancellationToken))
    {
        SavingChanges?.Invoke(this, EventArgs.Empty);
        var returnValue = await base.SaveChangesAsync(acceptAllChangesOnSuccess, cancellationToken);
        SavedChanges?.Invoke(this, EventArgs.Empty);
        return returnValue;
    }
}

Maybe there's something I'm missing?

[popcatalin81]

@javiercampos I'm thinking a lifecycle hook inside EF could potentially allow access to the Update pipeline allowing you to modify the resulted SQL, or modify the way store generated values are read back, making this feature much more powerful and making more scenarios possible than overriding SaveChanges. For example instead of single Update statement, you could send multiple statements interleaved with other updates. You cannot do this by overriding SaveChanges.

[javiercampos]

@popcatalin81 oh yes, I definitely agree as many "advanced scenarios" lifecycle hooks as possible should be implemented (and in a user-friendly way if possible), that's why I'm following this thread. Was just curious about @sidshetye's request (he was asking for the equivalent to ObjectContext.SavingChanges, and a possible SavedChanges event which are simple enough to implement yourself, not to overload the busy team :smile: ).

[sjb-sjb]

@javiercampos , the challenge with just overriding SaveChanges and SaveChangesAsync is that you then have to keep track of the entities that were changed. While it is true that you can copy this information from ChangeTracker before the save and then iterate through it again after the save, it seems more natural to me to have the system notify you of the changes in each entity. In addition this approach will not notify you of changes that are not due to SaveChanges, e.g. due to a property assignment, a revert changes (state = Unmodified) change, etc. Again you can build this yourself but, given that the ChangeTracker is already figuring all this out, it makes more sense to simply have the system notify you when there is a change.

About 'overloading the busy team', I agree they are busy (!!) and doing a fantastic job. On the other hand, this is a hook that is already there (in the ILocalViewListener service) that just needs to be included in the public API.

[javiercampos]

@sjb-sjb again, yes, but that's not what ObjectContext.SavingChanges did, and that's what I was replying to (@sidshetye specifically asked for the equivalent to ObjectContext.SavingChanges)

[SidShetye]

@javiercampos our data encryption platform (see http://www.crypteron.com) supports C# and within C#, we also support Entity Framework. We need SavedChanges to make it user friendly for our common Entity Framework customers. Because we can then directly plug into the hooks EF provides (plug and play) rather than us telling them "... and don't forget this snippet of custom code" - which busy developers always seem to forget! Specifically, we need this hook to cryptographically verify data integrity after saving to storage (SQL in EF's case). For contrast, the Java Persistence API (JPA) has such hooks, so our guidance there is very simple and I'd like to keep my C# customers equally happy! I raised this since adding a hook is simpler when already adding other hooks and seems the implementation could be straight forward.

About ObjectContext: We're not married to this specific interface but it's what we've used for our EF6 integrations, so we're going by that. I checked with my team and we're using the following API's. A same/improved API would be desired

• ((ObjectContext)sender).ObjectStateManager.GetObjectStateEntries(/* Added or Modified */)
• ((ObjectContext)sender).ObjectStateManager.ChangeObjectState(/* obj to modified or unchanged */)

In addition we're using these APIs DbContext.Configuration.AutoDetectChangesEnabled DbContext.Entry(..) DbEntityEntry.OriginalValues DbEntityEntry.CurrentValues

[javiercampos]

@sidshetye I see, that makes sense... in the meantime, one possible workaround (which involves work from the customer though) would be making a base DbContext from which they must inherit to use your framework. Not ideal on your specific case, yes, but better than having your customers override SaveChanges themselves

[ajcvickers]

Thanks for all the feedback!

Notes for triage:

• Suggest for 2.1 we do:
  • Entity state changes
  • Saving/SavedChanges
  • Possibly ObjectMaterialized

[ajcvickers]

Decision from triage: we will attempt to get state change events in for 2.1 as a stretch goal. Moving other events out for now.

[dazinator]

Just sharing a quick and dirty implementation to get the type of notification I needed for grabbing ID values after a save:

   // DbContext declared as partial class so implementation not lost if rescaffolding from db.
    public partial class FooDbContext 
    {
        private readonly List<Action> _notifySavedList = new List<Action>();

        protected void NotifySavedChanges()
        {
            foreach (var item in _notifySavedList)
            {
                item();
            }
            _notifySavedList.Clear();
        }  

        public void OnAfterSaveChanges(Action p)
        {
            _notifySavedList.Add(p);
        }

        public override int SaveChanges()
        {
            var result = base.SaveChanges();
            NotifySavedChanges();
            return result;
        }

        public override int SaveChanges(bool acceptAllChangesOnSuccess)
        {
            var result = base.SaveChanges(acceptAllChangesOnSuccess);
            NotifySavedChanges();
            return result;
        }

        public override async Task<int> SaveChangesAsync(bool acceptAllChangesOnSuccess, CancellationToken cancellationToken = default(CancellationToken))
        {
            var result = await base.SaveChangesAsync(acceptAllChangesOnSuccess, cancellationToken);
            NotifySavedChanges();
            return result;
        }

        public override async Task<int> SaveChangesAsync(CancellationToken cancellationToken = default(CancellationToken))
        {
            var result = await base.SaveChangesAsync(cancellationToken);
            NotifySavedChanges();
            return result;
        }

       public override void Dispose()
        {           
            _notifySavedList.Clear();
            base.Dispose();
        }

}

Use case looks something like this:

public class FooChildBusinessObject
{         
           public int Id { get; private set; }      
           public string Name { get; set; }      

           public bool IsNew { get; set; }     // Whatever logic you use to track inserts vs updates.

           public void Update(DbContext dbContext)
           {
               if(IsNew)
               {
                        var fooEntity = new Foo();
                        fooEntity.Name = Name;
                        dbContext.Foo.Add(fooEntity);
                        dbContext.OnAfterSaveChanges(() =>
                        {
                             // grab the database generated ID..
                             Id = fooEntity.Id;
                        });
               }
              else
              {
                        var fooEntity = dbContext.Foo.Single((f)=>f.Id == Id);
                        fooEntity.Name = Name;
              }       
          }
}

public class RootBusinessObject 
{
    public FooChildBusinessObject Foo { get; set; }   

    public void Save()
    {
          // This is some parent / root object saving children in its object graph.
          // It calls SaveChanges() after all objects in the graph have added their changes to the dbcontext.
         using (var dbContext = GetDbContext())
         {            
              Foo.Update(dbContext);
              dbContext .SaveChanges();       
              // Foo.Id should now be populated!      
         } 
    }     

}          

[SidShetye]

@ajcvickers Is there documentation for committed public API for entity state changes? EntityFrameworkCore/src/EFCore/ChangeTracking/ChangeTracker.cs seems to be evolving on which APIs are internal and which ones are public. We're investigating if we can make any progress till SavingChanges and ObjectMaterialized (and hopefully SavedChanges) are implemented ... hopefully not too far away!

[ajcvickers]

@sidshetye Docs are not done yet--tracking issue is https://github.com/aspnet/EntityFramework.Docs/issues/661. The public surface as it is now is what we plan to ship for 2.1, unless we get significant feedback from the prereleases that cause us to re-think.

[sjb-sjb]

Thanks for this!

A few comments based on looking at ChangeTracker.cs and also based on the experience I've had with the earlier, nonpublic version (LocalViewListener).

• It might be worth cautioning in the documentation that during the state change callback, one should be careful not to re-enter into another state change on the same entity (for example by assigning the state or by calling SaveChanges). I had a case where due to some very indirect interactions I was reverting the state from Added --> Detached during an Add, i.e. during a Detached --> Added state change callback. The result was that after the revert, subsequent continued processing of the Detached --> Added callback looked like the transition was from Detatched --> Detached. In the new interface I believe the effect would be that NewState would still be Added but this would no longer be equal to the new State of the EntityEntry, which would be Detached -- certainly a confusing state of affairs. Incidentally it would be useful to document that NewState is equal to the State of the entry at the time the state transition event is raised (assuming I understand it correctly).

• I am not clear on the reason for the Tracked event. Is this not redundant with entity state transitions from the Detached state to one of the other states?

• I am using these callback to, essentially, construct a view model which includes information about the entity states and entity values. As part of this I frequently would like to test whether or not there are any attached entries with certain states. For example, to enable UI for saving changes, one should test among other things whether or not there are any entities in Added, Modified, or Deleted states. This can be easily accomplished using the ChangeTracker.Entries().Where( ee => ee.State == X). However I am not very comfortable with that approach because it does not seem efficient to loop over all the attached entities simply to find the few that have a specific state. Consequently I ended up keeping my own tables for each of the states that I am interested in. I would prefer that there be a more efficient way.

[ajcvickers]

@sjb-sjb Thanks for the comments--I referenced them from the docs issue.

With regard to the Tracked event, that was the result of a long discussion on whether or not something being tracked for the first time is just a state change from Detached to something else, or whether it is a distinct event. Specifically, are there common scenarios where they would be used independently. My initial thought was that it was all just one state change event, but the ultimate outcome of the discussion was a team decision to create two different events. So you will never get state changed events from Detached to something else--those will instead be Tracked events.

Can you file a new issue for your last point? I think this is useful information for EF to provide. However, keep in mind that when normal non-notifying entity types are used, then it requires DetectChanges to run to determine if there are any entities in a given state, and this always involves scanning all entities, so this may never be super-performant unless notifying entities are being used.

[markusschaber]

Is there any docs on "notifying entities"? This is the first time I read about them...

[ajcvickers]

@markusschaber Docs are being tracked as https://github.com/aspnet/EntityFramework.Docs/issues/531

In a nutshell, have your entities implement INotifyPropertyChanging and INotifyPropertyChanged, and then use ObservableCollection for collection navigation properties. There are a quite a few places in the tests that you could look at.

[sjb-sjb]

@ajcvickers thanks for the comments.

I think that the FromQuery flag in the tracked event is useful. It looks like this will let you know whether the tracking is due to materialization or not. I could imagine this being extended to indicate whether, for example, tracking was caused by (a) an explicit assignment to the EntityEntry.State or by being the head entity in an Add, Update, etc, or (b) was due to the entity being reachable from a head entity that was tracked due to Add, Update, etc.

It is a little bit odd that we have a NewState in the entity transition event but do not have it in the Tracked event. Presumably the state of a newly tracked entity could be either Unchanged or Added, for example. If the user prevents entity state changes during the transition event and during the tracked event, then there is no need for NewState since it will remain equal to the EntityEntry.State. On the other hand since this locking is not built into EF there is always the possibility that the state could change. In conclusion it would make sense to me to add NewState to the tracked event. Caveat: I'm doing some guessing here since I haven't tried the new interface.

I will create the issue as you requested. As relates to the efficiency of state testing, as a matter of fact I am using notifying entities. One reason in fact for using notifying entities is to avoid the inefficiency of DetectChanges. Another reason, however, is the same as the reason for using EF state callbacks, namely to build a view-model system. If you are going to implement a view-model system then you have to notify property changes anyway, so you may as well leverage these notifications for use by EntityFramework tracking. Concerning the state change, I attach the entity to a context the first time a property is assigned and then the state callbacks are used to manage the entity's presence and sorting in observable collections supporting the interfaces required by ListView (this is a view-model activity and, I might say, not a very simple one -- especially if the collection is paged).

I'll tell you an overall comment. I realize that EF is focused on data access, that is it's raison d'etre. At the same time I don't think MS should ignore the fact that data access operates within UI programs (and within web services, etc.) As a consumer of MS products I think MS has done a fantastic job on the data access part, but it is really surprising to me that little has been done on integrating this into a coherent program-building paradigm. Specifically I feel MS should put effort into building a really professional view-model layer that fits on top of EF. Many people are out there cooking up their own, it is very duplicative and I imagine in most cases leading to not nearly as robust a solution.

[Wain123]

Are you planning to have an event after entities are created and the context is available to be used again?

After I load an entity from the DB, I want to initalize it, which would look something like this:

ctx.ChangeTracker.Tracked += (sender, e) => {
    ((BOBase)e.Entry.Entity).Init();
};

The problem is, if Init() attempts to execute a DB query, it will throw

System.InvalidOperationException: 'A second operation started on this context before a previous
operation completed. Any instance members are not guaranteed to be thread safe.'

because the Tracked event is invoked while the context is still busy with an operation.

If a query starts tracking multiple entities, I'd want an interception point after the whole query is completed (so, the context can be used again), not after each individual entity is loaded.

[ajcvickers]

@Wain123 We discussed such an event back in the EF4 days, but it is very problematic to implement because it would have to fire only after each entire query has been enumerated, which is something that is controlled by the caller. So at that time we decided that it was better for the application to define and control such events. That way the application can know when it has finished executing whatever queries it needs, even if that is several, and then fire and event to other parts of the application that should respond.

[Wain123]

@ajcvickers Ok, that's inconvenient, maybe there's another approach - is there a way to not get the InvalidOperationException when executing a DB query inside the ChangeTracker.Tracked event? The context becomes available very soon after the Tracked event (it's available inside the loop where I'm enumerating the query), so could this Tracked event be sent slightly later to allow me to Init each entity right as it becomes tracked?

[javiercampos]

Did any of this got through into 2.1? If so, any docs available?

I'm particulary interested on an ObjectMaterialized replacement... I see you have now the Tracked event on the ChangeTracker, but I'm afraid that will not do if you make a query with AsNoTracking().

Is there any "official" hook (apart from EntityMaterializerSource which, although it's what we are using, is internal and has actually changed from 2.0 to 2.1) to get when entities are created?

[ajcvickers]

@javiercampos The only new hooks added in 2.1 were for the Tracked and StateChanged events. There is no ObjectMaterialized yet--the main reason being that the code and patterns in this area are changing with the introduction of constructor binding and factory support. However, how all the API hooks up and when to use the event over a binding, if ever, is till to be determined. See #10789

[AFDevMike]

The InternalEntityEntry received via IEntityStateListener's StateChanged doesn't seem to contain the values set by the database itself such as those which are set by .ValueGeneratedOnAddOrUpdate().

Should they? (No I learned)

Update - This was a fundamental misunderstanding on my part. Entities are not updated from the Db after SaveChanges[Async]() call unless they are specific properties such as Primary Keys or RowVersion(I don't know the full extent but certainly Primary Keys). Therefore properties which are not the above and are .ValueGeneratedOnAddOrUpdate() cannot be expected to be current.

[dazinator]

@jdevmike - I worked around this using the hand rolled solution I posted above. If I should be replacing this with a new EF mechanism please let me know.

[SidShetye]

@ajcvickers when can we expect ObjectMaterialized and SavingChanges equivalents to be released? Waiting but hoping it's not a long wait 🤞

[tourili]

Hi @divega

What about intercepting connection like we used to do on EF6 using IDbConnectionInterceptor. Interception is not for logging only!

My asp core app requires executing session context EXEC sp_set_session_context ... on the server each time database connection is opened.

My only workaround now is to hook on the event Database.GetDbConnection().StateChange. which is my last option.

Thanks for help

[SidShetye]

@ajcvickers Following up on the April 7th comment; our proof-of-concept with EF Core 2.1 confirms that ChangeTracker.Tracked and ChangeTracker.StateChanged events in a DbContext are insufficient for us to get the job done on EF Core 2.1. Coming from EF 6.0, we did have this hunch. But can now confirm that we are indeed blocked from supporting EF Core 2.x till SavingChanges and ObjectMaterialized (and hopefully SavedChanges) make the release. Would appreciate you sharing your roadmap on this so we can adjust ours accordingly.

[divega]

@SidShetye re Tracked and StateChanged, are they insufficient because they don't trigger when non-tracked objects are materialized or because you need to be able to perform operations that can potentially use the context to execute queries while processing the event?

[SidShetye]

@divega for us they are insufficient because there are no event hooks at the moment of entity materialization from the db (like ObjectMaterialized) as well as at the moment just prior to the entity hitting the network (like SavingChanges). A 3rd event hook that fired after an entity (e.g. SavedChanges) has been written to the network would also be extremely helpful.

[sjb-sjb]

@SidShetye concerning the 3rd event hook, given that before the SaveChanges the state will be Modified, Added or Deleted and afterward it will be Unchanged or Detached, can you not use the StateChanged event?

[dazinator]

@sjb-sjb Just to play devils advocate, given you can detach and attach entities, and change the state of entities yourself, a state changing event is not necessarily any guarantee that the entity was actually written to the network right?

[SidShetye]

@sjb-sjb : Our reason is the same as @dazinator i.e. the state transitions

(Modified | Added | Deleted ) => (Unchanged | Detached)

could happen for certain permutations of user-level usage patterns even when the entity doesn't actually hit the network.

[sjb-sjb]

Yeah, I agree if you let the user do anything then state transitions won't correspond to writes. I could see it happening with a library intended to complement EF for example.

In my case I just don't let them do anything weird, and if they do then they pretty much just get what they deserve 😏.

[juanmarl]

@Wain123 When you materialize a query with many entities you get a bunch of ChangeTracker.Tracked events, one per each entity in the query result. With this code you can call Init() when the object graph is completely materialized:

(You still can't use the object context to make further queries, but you can eager load everything you need in the same graph and then Initialize it on due time. Another option would be to use directly a DbCommand or another instance of the context for these queries).

        private void ChangeTracker_Tracked(object sender, EntityTrackedEventArgs e)
        {
            if (!e.FromQuery)
                return;

            var entity = e.Entry.Entity;

            if (entity is IInitializable initializable)
                _initializableEntities.Add(initializable); //keep until materialization completes and then call Init()

            if (_dbConnection.State == ConnectionState.Closed) //last entity in graph
            {
                 e.Entry.State = EntryState.Detached;
                 e.Entry.State = EntryState.Unchanged; //force change tracker to fix all entry navigations

                 _initializableEntities.ForEach(x => x.Init());
                 _initializableEntities.Clear();
            }
        }

[sjb-sjb]

Just a comment based on some recent experience and reviewing the various comments above.

Many of the comments / issues raised are connected to the fact that Tracked / StateChanged events occur during materialization or during SaveChanges. For example we see a number of comments seeking initialization hooks, which would be addressed if we knew that state transitions occurred after completion of queries or completion of saves (writes).

What I recently realized is that the processing of state changes during the process of materialization or during the process of SaveChanges can lead to inconsistent application state because the various entities in the application (for example a set of entities currently attached to a context and being saved) will be at different stages of being processed by EF. The example I ran into was a SaveChanges of a graph of entities in the context. Some of the entities had been saved and their store-generated ID's set, while other dependent entities had foreign keys (and navigation properties) onto these saved entities but the foreign keys had not yet been updated to match the store-generated id's. That is, the foreign key was a negative number (EF-generated placeholder) while the referenced entity pointed to by the navigation property had a positive id (store-generated). Clearly a bad situation.

Thinking about this, I wonder if I need to put the Tracked / StateChanged events into a producer/consumer queue and pull them off asynchronously for processing after the materialization or SaveChanges is complete. The asynchrony could also help to avoid accidental re-entrancy while processing the state changes. This leads back to the question of having a hook for when materialization or SaveChanges is complete.

If people feel it would be useful to have these events be processed asynchronously after the materialization or save is complete then maybe it could be considered for inclusion as an option in EF.

[sjb-sjb]

On the other hand it is also true that sometime you do want the synchronous invocation. For example this is useful if you are trying to track state information that is synchronized with EF state information.