subproc stdout: WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
subproc stdout: WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpB1d7nX"
subproc stdout: If you would like to access the database for debugging, run the following:
subproc stdout:
subproc stdout: # Run the database
subproc stdout: cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpB1d7nX/data"
subproc stdout: # Access the database. Note the port may change if you run multiple databases.
subproc stdout: cockroach sql --host=localhost:32221 --insecure
subproc stdout: Error: populating database
subproc stdout:
subproc stdout: Caused by:
subproc stdout: 0: populate
subproc stdout: 1: populating Omicron database
subproc stdout: 2: db error: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout: HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout: --
subproc stdout: Try setting a higher --max-sql-memory.
subproc stdout: --
subproc stdout: Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout: Manual inspection may be required to determine the actual state of the database.
subproc stdout: --
subproc stdout: See: https://github.com/cockroachdb/cockroach/issues/42061
subproc stdout: 3: ERROR: transaction committed but schema change aborted with error: (53200): failed to ingest index entries during backfill: not enough memory available to create a BulkAdder: root: memory budget exceeded: 33554432 bytes requested, 101902336 currently allocated, 134217728 bytes in budget
subproc stdout: HINT: Consider increasing --max-sql-memory startup parameter.
subproc stdout: --
subproc stdout: Try setting a higher --max-sql-memory.
subproc stdout: --
subproc stdout: Some of the non-DDL statements may have committed successfully, but some of the DDL statement(s) failed.
subproc stdout: Manual inspection may be required to determine the actual state of the database.
subproc stdout: --
subproc stdout: See: https://github.com/cockroachdb/cockroach/issues/42061
--max-sql-memory is supposed to be 25% of system max memory, but this is not implemented on illumos. It appears to default to 128 MiB when system memory information is unavailable.
This test failed on a CI run on "main":
https://github.com/oxidecomputer/omicron/runs/33299021748
Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01JD6AMYY5FWJG8QEE5YV9VDMR/HviS2fo6F628FaTMfPIrWwuLVKy3mGtea6bgJKpuqOVsvMYg/01JD6AN89MD4FZ5MNX4QJ3PX8X#S5268
Excerpt from the log showing the failure:
--max-sql-memory
is supposed to be 25% of system max memory, but this is not implemented on illumos. It appears to default to 128 MiB when system memory information is unavailable.