oxidecomputer / omicron

Omicron: Oxide control plane
Mozilla Public License 2.0
252 stars 40 forks source link

test failed in CI: `db-dev::test-db-dev test_db_killed` #7141

Open iliana opened 12 hours ago

iliana commented 12 hours ago

This test failed on a CI run on "main":

https://github.com/oxidecomputer/omicron/runs/33299021748

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JD6AMYY5FWJG8QEE5YV9VDMR/HviS2fo6F628FaTMfPIrWwuLVKy3mGtea6bgJKpuqOVsvMYg/01JD6AN89MD4FZ5MNX4QJ3PX8X#S5268

Excerpt from the log showing the failure:

subproc stdout: WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
subproc stdout: WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpB1d7nX"
subproc stdout:     If you would like to access the database for debugging, run the following:
subproc stdout:
subproc stdout:     # Run the database
subproc stdout:     cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpB1d7nX/data"
subproc stdout:     # Access the database. Note the port may change if you run multiple databases.
subproc stdout:     cockroach sql --host=localhost:32221 --insecure
subproc stdout: Error: populating database
subproc stdout:
subproc stdout: Caused by:
subproc stdout:     0: populate
subproc stdout:     1: populating Omicron database
subproc stdout:     2: db error: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout:        --
subproc stdout:        Try setting a higher --max-sql-memory.
subproc stdout:        --
subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout:        Manual inspection may be required to determine the actual state of the database.
subproc stdout:        --
subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061
subproc stdout:     3: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout:        HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout:        --
subproc stdout:        Try setting a higher --max-sql-memory.
subproc stdout:        --
subproc stdout:        Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout:        Manual inspection may be required to determine the actual state of the database.
subproc stdout:        --
subproc stdout:        See: https://github.com/cockroachdb/cockroach/issues/42061

--max-sql-memory is supposed to be 25% of system max memory, but this is not implemented on illumos. It appears to default to 128 MiB when system memory information is unavailable.

iliana commented 12 hours ago

(I thought I had seen this twice in CI on main, and said so in chat earlier today, but I was mistaken.)