Determine source of truth for an entity `LastSeen` timestamp

[mercul3s]

For keepalive events, the entity's last_seen time is updated in the store to the current time by keepalived and added to the event when it is processed. Eventd does not have the same behavior. Depending on where the entity is viewed (by itself, as part of a keepalive event, or a check event), the last_seen time may differ. We can update the last_seen time any time we see a check or keepalive event, but that would double the amount of writes to etcd. If we want to go that route, we should try to do the update in a transaction.

[jamesdphillips]

Drive-by.

I think there might be an argument that etcd is not well suited for storing something like an entity's presence. We easily could find that the value is inconsistent, writing every time we receive an event is likely costly and could lead to races (unless we implement with locks which increases complexity.)

[mercul3s]

I'm concerned about writes here too, it could be very costly depending on the number of entities and checks. Now that I've had a night's sleep on this, I'm thinking that keepalives should probably remain the source of truth for an entity's last seen time.

[portertech]

One reason we may want to update Entity last_seen on new relevant event is to enable de-registration for Proxy Entities (those that do not have an Agent and therefore keepalives), see https://github.com/sensu/sensu/issues/1477

While this is rad, I do not see us doing this until we're not writing events to etcd (down the road).

[annaplotkin]

Agent is the source of truth. Closing.