WatchAssignments efficiency

[andrewgrundy]

Is your feature request related to a problem? Please describe. The FrontendService method WatchAssignments polls the database waiting for the assignment object to change. The backoff.Retry is set to have a ConstantBackOff (a backoff policy that always returns the same backoff delay). The ConstantBackOff is configured to have backoff.initialInterval delay between reads which defaults to 100ms.

Describe the solution you'd like Rather than polling the database, potentially hundreds of times between submitting a ticket and getting an assignment, WatchAssignments could subscribe to PubSub events related to the ticket id in question and the director could publish once an assignment has been made.

Describe alternatives you've considered None, but happy to discuss other solutions to not polling the database.

Additional context One of the issues highlighted by load testing has been that Redis has been a bottleneck, this will not be helping.

[syntxerror]

We discussed in the contributors call that there is no immediate negatives to this pattern. There is a cap on the number of connected subscribers but is configurable if it is an issue. It can offered as a feature flag or can be turned on or off. We can explore (run load tests) against this and see the pros/cons. We can provide multiple configurations to test (alpha, beta, stable/release).

[scosgrave]

@syntxerror - I was talking with @andrewgrundy and I think your comment above made perfect sense to those who were in the meeting when you wrote it, but reading it again here from the perspective of someone not in the meeting, we can probably add some clarifications. Please feel free to correct me if I'm wrong on any of these. I'll try and fill in some details related to your comment below.

We discussed in the contributors call that there is no immediate negatives to this pattern.

The pattern you were talking about is the one @andrewgrundy is proposing, not the pattern of querying Redis on a short interval.

There is a cap on the number of connected subscribers but is configurable if it is an issue.

I believe this maxclients setting on Redis is what you are referring to: https://github.com/redis/redis/blob/6.2/redis.conf#L965

It can offered as a feature flag or can be turned on or off.

The new pub-sub notification feature can be flagged on or off as desired.

[syntxerror]

@scosgrave thanks for adding detail for the community. I forget that not all folks tune in, read notes, and the calls are no longer recorded