[Notifier] Use the Postgres notifier's connection to deliver notifications.
The Postgres notifier holds a single connection for listening and relaying messages. However, it wasn't used to dispatch messages; that was left to queries through the Ecto pool. Those queries were noisy and put unnecessary load on the pool, particularly from insert notifications.
Now notifications are delivered through the notifier's connection—they don't require a pool checkout, and they won't clutter Ecto logs or telemetry.
[Engine] Emit insert trigger notification directly from Engine callbacks.
Notifications are now sent from the engine, within the insert_* telemetry block, so the timing impact is visible. In addition, notifications aren't emitted for scheduled jobs, as there's nothing ready for producers to fetch.
Bug Fixes
[Notifier] Track and compare sonar pings using the correct time unit.
The notifier's status tracker pruned stale nodes using mismatched time units, causing constant status change events despite nothing changing. This ensures the recorded and compared times are both milliseconds, not a mixture of seconds and native time.
[Cron] Retain @reboot cron entries until node becomes leader.
With rolling deploys it is frequent that a node isn't the leader the first time cron evaluates. However, @reboot expressions were discarded after the first run, which prevented reboots from being inserted when the node acquired leadership.
[Oban] Require Ecto v3.10 to support materialized flag added in the previous patch.
The materialized option wasn't supported by Ecto until v3.10. Compiling with an earlier version causes a compilation error.
v2.17.4
Enhancements
[Notifier] Add Notifier.status/1 to expose cluster connectivty status for diagnostics.
Notifier gains status/1, backed by an internal Sonar process, for access to cluster diagnostics. This will help identify whether the current node can communicate with other nodes for vital functionality such as staging jobs.
A new [:oban, :notifier, :switch] telemetry event is emitted for diagnostic instrumentation and switch events are displayed by the default logger.
[Stager] Use notifier status and leadership checks to switch between global and local mode.
The sonar-backed notifier status is more accurate than the ad-hoc stager checks previously used to check connectivity. Now the staging mode is determined by a combination of sonar and leadership—isolated follower nodes will correctly switch to :local (polling) mode.
The new detection mechanism prevents an infrequent and difficult to spot bug where isolated follower nodes wouldn't process jobs.
[Backoff] Catch GenServer timeouts in with_retry/2.
Backoff's with_retry/2 function previously rescued and retried known database exceptions, and it didn't catch exits. Now it also catches timeout errors to accommodate synchronized call based acks like those used in Pro's Smart engine in sync mode.
[Stager] Avoid transactions on follower nodes.
Only the leader needs to execute a transaction while staging. This change prevents pointless empty transactions from each follower node every second. Now there are 60 staging transactions per minute across the whole cluster, rather than 60 from each node.
[Notifier] Use the Postgres notifier's connection to deliver notifications.
The Postgres notifier holds a single connection for listening and relaying messages. However, it
wasn't used to dispatch messages; that was left to queries through the Ecto pool. Those queries
were noisy and put unnecessary load on the pool, particularly from insert notifications.
Now notifications are delivered through the notifier's connection—they don't require a pool
checkout, and they won't clutter Ecto logs or telemetry.
[Engine] Emit insert trigger notification directly from Engine callbacks.
Notifications are now sent from the engine, within the insert_* telemetry block, so the timing
impact is visible. In addition, notifications aren't emitted for scheduled jobs, as there's
nothing ready for producers to fetch.
Bug Fixes
[Notifier] Track and compare sonar pings using the correct time unit.
The notifier's status tracker pruned stale nodes using mismatched time units, causing constant
status change events despite nothing changing. This ensures the recorded and compared times are
both milliseconds, not a mixture of seconds and native time.
[Cron] Retain @reboot cron entries until node becomes leader.
With rolling deploys it is frequent that a node isn't the leader the first time cron evaluates.
However, @reboot expressions were discarded after the first run, which prevented reboots from
being inserted when the node acquired leadership.
[Oban] Require Ecto v3.10 to support materialized flag added in the previous patch.
The materialized option wasn't supported by Ecto until v3.10. Compiling with an earlier
version causes a compilation error.
v2.17.4 — 2024-02-16
Enhancements
[Notifier] Add Notifier.status/1 to expose cluster connectivty status for diagnostics.
Notifier gains status/1, backed by an internal Sonar process, for access to cluster
diagnostics. This will help identify whether the current node can communicate with other nodes
for vital functionality such as staging jobs.
A new [:oban, :notifier, :switch] telemetry event is emitted for diagnostic instrumentation
and switch events are displayed by the default logger.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bumps oban from 2.13.5 to 2.17.5.
Release notes
Sourced from oban's releases.
... (truncated)
Changelog
Sourced from oban's changelog.
... (truncated)
Commits
febfc63
Release v2.17.5a3e8a99
Emit insert notification directly from Enginefb87366
Use notifier connection to deliver notifications3d2e1fb
Use :connected flag in isolated notifierd65ceec
Retain reboot cron entries until node is leader89eb348
Use configurable agent for isolated/disabled peercf01577
Fix typo in peer docs (#1046)45e5ef8
Improve docs on retrying in case of an uncaught exception, exit or raise (#1044)f8d3478
Require Ecto v3.10 to support materialized cte6b9bf1d
Correctly use :millisecond instead of :second unitDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase
.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show