alohaninja
commented
6 years ago Known workarounds (Option 2)
So I found another workaround that doesn't require re-creating the entire lease collection. If you identify the documents in your lease collection that contain state=1 and Owner=null, you should set the ContinuationToken=null. Here is the before/after and the final state once the lease continued to get out of the errored state. Unsure why the ContinuationToken gets out of whack.
Lease Collection Document (Error State)
Partition change feed trigger stops sending input triggers to azure function. state=1 and Owner=null.
{
"state": 1,
"PartitionId": "7",
"Owner": null,
"ContinuationToken": "\"245895\"",
"SequenceNumber": 4726
}Lease Collection Document (Modified State)
Update the state=2 (healthy) and ContinuationToken=null.
{
"state": 2,
"PartitionId": "7",
"Owner": null,
"ContinuationToken": null,
"SequenceNumber": 4726
}Lease Collection Document (Healthy State)
ContinuationToken=<new_value>, Owner=<new_guid> and SequenceNumber increments.
{
"state": 2,
"PartitionId": "7",
"Owner": "e40a9970-6f13-4fe3-9581-7e5306d982e5",
"ContinuationToken": "\"246479\"",
"SequenceNumber": 4727
}After modifying the lease collection partition document, change feed events are no longer hung. Hoping someone can shed some light on why this happens. Seen this with numerous change feed functions we are using.
Downside is that any events that occurred when the hung state occurred are lost (since we reset the continuation token to get back to a healthy state). This is a major reliability hurdle for us (and the 2nd such critical issue we've identified with cosmos triggers)
We have numerous change feeds 10+ for various eventing and ALL of them failed at the same ContinuationToken for the same partition. We had to perform the above workaround 10x for each lease collection.
christopheranderson
ealsur
mkolt
syedhassaanahmed
pvmraghunandan
paulbatum
ryancrawcour
Investigative information
Please provide the following:
Repro steps
Seems that after our change feeds have been running for awhile they intermittently just stop working. It seems it could be related to RU/s being exhausted, but there is no recovery process. There are 429s that were logged in the DB being monitored as it was under occasional load.
Expected behavior
Even when the DB being monitored is under load or receives 429s, the lease for the change feed should not ever get into a hung state. It should recover when the lease gets frozen. The input trigger should never stop firing.
Actual behavior
The input trigger stops firing and no new change feed docs are processed despite there still being traffic/activity in the lease db collection.
Known workarounds
The only known workaround I've been able to identity is to delete the lease collection and restart the function host. Restarting the function host recreates the lease collection (assuming
createLeaseCollectionIfNotExists=trueinfunction.json) and input triggers begin flowing in again. This is a very weak solution as we lose any change feed activity since the feed stopped responding.Related information
Language: Javascript AzureWebJobsHostLogs: stops writing trigger events when change feed is hung/expires, even though there are change feed events occurring and the lease collection has activity. The function host just stops checking for new events or is hung. Restarting the function app has no effect.
Lease Collection Partition State
Upon further inspection of the lease collection, all other partition documents contain
state=2and contain a validOwnerguid. One partition does not contain an owner and hasstate=1.Invalid Lease Collection Document
Other partitions are sending triggers to the function app, just not this partition stuck in
state=1. Appears this issue is related to a partition state being stuck.