A new queue shutdown event, [:oban, :queue, :shutdown], is emitted by each queue when it terminates. The event originates from the watchman process, which tracks the total ellapsed time from when termination starts to when all jobs complete or the allotted period is exhausted.
Any jobs that take longer than the :shutdown_grace_period (by default 15 seconds) are brutally killed and left as orphans. The ids of jobs left in an executing state are listed in the event's orphaned meta.
This also adds queue:shutdown logging to the default logger. Only queues that shutdown with orphaned jobs are logged, which makes it easier to detect orphaned jobs and which jobs were affected:
[
message: "jobs were orphaned because they didn't finish executing in the allotted time",
queue: "alpha",
source: "oban",
event: "queue:shutdown",
ellapsed: 500,
orphaned: [101, 102, 103]
]
🚚 Distributed PostgreSQL Support
It's now possible to run Oban in distributed PostgreSQL databases such as Yugabyte. This is made possible by a few simple changes to the Basic engine, and a new unlogged migration option.
Some PostgreSQL compatible databases don't support unlogged tables. Making oban_peers unlogged isn't a requirement for Oban to operate, so it can be disabled with a migration flag:
defmodule MyApp.Repo.Migrations.AddObanTables do
use Ecto.Migration
def up do
Oban.Migration.up(version: 12, unlogged: false)
end
end
🧠Job Observability
Job stop and exception telemetry now includes the reported memory and total reductions from the job's process. Values are pulled with Process.info/2 after the job executes and safely fall back to 0 in the event the process has crashed. Reductions are a rough proxy for CPU load, and the new measurements will make it easier to identify computationally expensive or memory hungry jobs.
In addition, thanks to the addition of Process.set_label in recent Elixir versions, the worker name is set as the job's process label. That makes it possible to identify which job is running in a pid via observer or live dashboard.
v2.18.0 — 2024-07-26
Enhancements
[Job] Support simple unique: true and unique: false declarations
[Job] Support simple unique: true and unique: false declarations
Uniqueness can now be enabled with unique: true and disabled with unique: false from job
options or a worker definition. The unique: true option uses all the standard defaults, but
sets the period to :infinity for compatibility with Oban Pro's new simple unique mode.
[Cron] Remove forced uniqueness when inserting scheduled jobs.
Using uniqueness by default prevents being able to use the Cron plugin with databases that don't
support uniqueness because of advisory locks. Luckily, uniqueness hasn't been necessary for safe
cron insertion since leadership was introduced and scheduling changed to top-of-the-minute
many versions ago.
The check_available/1 callback allows engines to customize the query used to find jobs in the
available state. That makes it possible for alternative engines, such Oban Pro's Smart engine,
to check for available jobs in a fraction of the time with large queues.
[Peer] Add Oban.Peer.get_leader/2 for checking leadership
The get_leader/2 function makes it possible to check which node is currently the leader
regardless of the Peer implementation, and without having to query the database.
[Producer] Log a warning for unhandled producer messages.
Some messages are falling through to the catch-all handle_info/2 clause. Previously, they were
silently ignored and it degraded producer functionality because inactive jobs with dead pids
were still tracked as running in the producer.
[Oban] Use structured messages for most logger warnings.
A standard structure for warning logs makes it easier to search for errors or unhandled messages
from Oban or a particular module.
Bug Fixes
[Job] Include all fields in the unique section of Job.t/0.
The unique spec lacked types for both keys and timestamp keys.
[Basic] Remove materialized option from fetch_jobs/3.
The MATERIALIZED clause for CTEs didn't make a meaningful difference in job fetching accuracy.
In some situations it caused a performance regression (which is why it was removed from Pro's
Smart engine a while ago).
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
Bumps the production-dependencies group with 8 updates:
3.2.6
3.3.2
1.3.8
1.4.1
2.0.4
2.1.1
2.1.15
2.1.17
0.2.0
0.3.0
1.4.3
1.4.4
2.17.12
2.18.0
3.20.0
3.20.1
Updates
ash
from 3.2.6 to 3.3.2Changelog
Sourced from ash's changelog.
... (truncated)
Commits
89200d5
chore: release version v3.3.2ea7abca
remove IO.inspect266958d
improvement: add--timestamps
argument toash.gen.resource
db54a65
fix: properly key nested calculations and add additional tests2843f8d
fix: pass relationship path down when merging query loads8f865c1
fix: don't set--name nil
when calling codegen tasks77d12dc
docs: Add section regarding strict-loading for manual relationships. (#1359)8a368c0
fix: fix behavior of synthesized joins across data layers26f4a15
chore: use.has_validate?
on conditions, nothas_batch_change?
f521069
chore: update changelogUpdates
ash_json_api
from 1.3.8 to 1.4.1Changelog
Sourced from ash_json_api's changelog.
Commits
f4a4f30
chore: release version v1.4.166249be
improvement: properly install domain withModule.concat
in AshJsonApi routerb729210
chore: update igniter dependency104b8c1
improvement: include multipart parser in installer49da26c
chore: release version v1.4.09ccef6b
chore: handleallow_nil_input
in open api schema as well05d909b
fix: reject allow_nil_input fields in required_write_attributes (#219)7633fce
docs: Update documentation with what Igniter generates (#216)beaa320
feat: Ash.Type.File support (#214)a5c3fef
Docs: Use correct command in moduledoc (#215)Updates
ash_phoenix
from 2.0.4 to 2.1.1Changelog
Sourced from ash_phoenix's changelog.
Commits
9b37c31
chore: release 2.1.192674c5
improvement: raise an error on usage of old option nameee2626d
chore: release version v2.1.03f6c72f
chore(deps): bump ash in the production-dependencies group (#228)b8a0bb8
chore(deps-dev): bump mix_audit in the dev-dependencies group (#225)1a925c6
chore(deps): bump ash in the production-dependencies group (#224)cb5a50f
chore(deps): bump ash in the production-dependencies group (#222)5500c69
chore(deps-dev): bump ex_doc in the dev-dependencies group (#223)4f4ea9b
fix: Use :public? instead of :private? (#221)5c5ee5a
docs: Use AshPhoenix.Form.submit/2 instead of calling context functions (#220)Updates
ash_postgres
from 2.1.15 to 2.1.17Changelog
Sourced from ash_postgres's changelog.
... (truncated)
Commits
b84713c
chore: release version v2.1.17fb5e57c
improvement: update ash & ash_sql for various fixes793d27d
test: replicate type mismatch bug (#364)07d84d4
chore: release version v2.1.16ac08104
fix: don't overwrite non-updated fields on updatebe7329d
docs: Update migrations-and-tasks.md (#363)2b4befe
chore: addigniter.install
version of getting started guidee49b678
fix: ensure app is compiled before using repo modulesUpdates
flame
from 0.2.0 to 0.3.0Changelog
Sourced from flame's changelog.
Commits
1de41ef
Release 0.3.0dab6f73
Add FLAME.Trackable (#44)ed91149
Dereference symlinkse8a52dd
Fix body type855540b
Kill use of anons for config3b99fb6
Feat/erlang json parser (#45)9ee8fd8
Support specifying multiple GPUs for a Fly runner (#48)be9d6d9
Fix code snippet (#47)6c4e155
Update missing comma to code example (#43)cf65727
Use default args for callUpdates
jason
from 1.4.3 to 1.4.4Changelog
Sourced from jason's changelog.
Commits
926d2ac
Bump 1.4.46c1b99e
Raise if trying to decode decimals without decimal9517f56
Remove unneeded workarounds for xref warnings433f93d
Fix warnings by conditionally compiling Decimal supporteb1e92a
Bump 1.4.33ffa13a
Fix derive with _ keyUpdates
oban
from 2.17.12 to 2.18.0Release notes
Sourced from oban's releases.
... (truncated)
Changelog
Sourced from oban's changelog.
... (truncated)
Commits
baf6851
Release v2.18.00ef8021
Update testing_queues.md to be consistent with installation guide (#1121)78d9ee7
Accept enumerable output instead of exact streamb6d83e6
Include process memory and reductions in telemetry7601b8e
Add Oban.Peer.get_leader/2 for checking leadershipb36b963
Use structured messages for most logger warningsd8b5fee
Log a warning for unhandled producer messagese63d60d
Remove materialized option from Basic.fetch_jobs/300ddc10
Remove forced unique from cron plugin4d43326
Merge oban_jobs table migrationsUpdates
open_api_spex
from 3.20.0 to 3.20.1Release notes
Sourced from open_api_spex's releases.
Changelog
Sourced from open_api_spex's changelog.
Commits
7d388cd
Release version 3.20.15ec452f
Update Schema.example/2 typespec to allow references19ffc49
Support custom error messages in custom validators (#621)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase
.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show