Admin terminated by SIGABRT - Can't DROP INDEX `PRIMARY`

karolszklarski commented 10 months ago

Steps To Reproduce

Install bitwarden image and standalone mariadb database (a service, not a docker image).

bitwarden/self-host b11825cadef4 (old dev) works fine

bitwarden/self-host 74ccd545df9c (current dev) and e1a1833fcc06 (current beta) - admin module enters a crash loop.

gitHash cf4d8a4f is working, ea162d53 is not

Expected Result

Bitwarden should start normally.

Actual Result

Admin enters a crash loop

root@helios4:/home/bitwarden/prod# docker-compose logs
Attaching to bitwarden-prod
bitwarden-prod | Adding group `bitwarden' (GID 2009) ...
bitwarden-prod | Done.
bitwarden-prod | Adding user `bitwarden' ...
bitwarden-prod | Adding new user `bitwarden' (2009) with group `bitwarden' ...
bitwarden-prod | Not creating home directory `/home/bitwarden'.
bitwarden-prod | 2024-01-03 10:35:37,349 INFO Included extra file "/etc/supervisor.d/admin.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,350 INFO Included extra file "/etc/supervisor.d/api.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,352 INFO Included extra file "/etc/supervisor.d/events.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/icons.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/identity.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/nginx.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/notifications.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/scim.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,354 INFO Included extra file "/etc/supervisor.d/sso.ini" during parsing
bitwarden-prod | 2024-01-03 10:35:37,379 INFO RPC interface 'supervisor' initialized
bitwarden-prod | 2024-01-03 10:35:37,379 CRIT Server 'unix_http_server' running without any HTTP authentication checking
bitwarden-prod | 2024-01-03 10:35:37,380 INFO supervisord started with pid 1
bitwarden-prod | 2024-01-03 10:35:38,386 INFO spawned: 'identity' with pid 68
bitwarden-prod | 2024-01-03 10:35:38,450 INFO spawned: 'admin' with pid 69
bitwarden-prod | 2024-01-03 10:35:38,461 INFO spawned: 'api' with pid 70
bitwarden-prod | 2024-01-03 10:35:38,475 INFO spawned: 'icons' with pid 71
bitwarden-prod | 2024-01-03 10:35:38,489 INFO spawned: 'nginx' with pid 72
bitwarden-prod | 2024-01-03 10:35:38,505 INFO spawned: 'notifications' with pid 73
bitwarden-prod | 2024-01-03 10:35:54,206 INFO success: identity entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,208 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,208 INFO success: api entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,209 INFO success: icons entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,209 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:35:54,210 INFO success: notifications entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:36:37,760 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-prod | 2024-01-03 10:36:38,767 INFO spawned: 'admin' with pid 172
bitwarden-prod | 2024-01-03 10:36:54,568 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:37:32,493 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-prod | 2024-01-03 10:37:33,509 INFO spawned: 'admin' with pid 216
bitwarden-prod | 2024-01-03 10:37:48,535 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-prod | 2024-01-03 10:38:24,773 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-prod | 2024-01-03 10:38:25,778 INFO spawned: 'admin' with pid 253
bitwarden-prod | 2024-01-03 10:38:40,800 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)

Admin logs

root@helios4:/home/bitwarden/prod# cat log/bitwarden/admin.log
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (17ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      CALL POMELO_BEFORE_DROP_PRIMARY_KEY(NULL, 'Grant');
      ALTER TABLE `Grant` DROP PRIMARY KEY;
Unhandled exception. MySqlConnector.MySqlException (0x80004005): Can't DROP INDEX `PRIMARY`; check that it exists
   at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 43
   at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 130
   at MySqlConnector.MySqlDataReader.CreateAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 468
   at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList`1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56
   at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 296
   at MySqlConnector.MySqlCommand.ExecuteNonQuery() in /_/src/MySqlConnector/MySqlCommand.cs:line 107
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQuery(RelationalCommandParameterObject parameterObject)
   at Microsoft.EntityFrameworkCore.Migrations.MigrationCommand.ExecuteNonQuery(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQuery(IEnumerable`1 migrationCommands, IRelationalConnection connection)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration)
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade)
   at Bit.MySqlMigrations.MySqlDbMigrator.MigrateDatabase(Boolean enableLogging, CancellationToken cancellationToken) in /source/util/MySqlMigrations/MySqlDbMigrator.cs:line 30
   at Bit.Admin.HostedServices.DatabaseMigrationHostedService.StartAsync(CancellationToken cancellationToken) in /source/src/Admin/HostedServices/DatabaseMigrationHostedService.cs:line 29
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Bit.Admin.Program.Main(String[] args) in /source/src/Admin/Program.cs:line 9

Screenshots or Videos

No response

Additional Context

docker image tag bitwarden/self-host:beta bitwarden/self-host:prod

docker-compose.yml

version: "3.8"

services:
  bitwarden:
    container_name: bitwarden-prod
    env_file:
      - settings.env
    image: bitwarden/self-host:prod
    restart: always
    networks:
      - bitwarden-prod
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 20489:8443
    volumes:
      - /home/bitwarden/prod/etc:/etc/bitwarden
      - /home/bitwarden/prod/log/bitwarden:/var/log/bitwarden
      - /home/bitwarden/prod/log/nginx:/var/log/nginx

networks:
  bitwarden-prod:
    driver: bridge

settings.env

BW_DOMAIN=xxx.com
BW_ENABLE_SSL=true
globalSettings__baseServiceUri__vault=https://xxx.com:20489

globalSettings__disableUserRegistration=true

BW_DB_PROVIDER=mariadb
BW_DB_SERVER=host.docker.internal
BW_DB_DATABASE=bitwarden
BW_DB_USERNAME=bitwarden
BW_DB_PASSWORD=xxx

BW_INSTALLATION_ID=xxx
BW_INSTALLATION_KEY=xxx

adminSettings__admins="xxx@gmail.com"

globalSettings__mail__smtp__host=smtp.gmail.com
globalSettings__mail__smtp__port=587
globalSettings__mail__smtp__ssl=true
globalSettings__mail__smtp__username=xxx
globalSettings__mail__smtp__password=xxx

PUID=2009
PGID=2009

Githash Version

919d759f-dirty

Environment Details

Linux helios4 6.1.63-current-mvebu #1 SMP Mon Nov 20 10:52:19 UTC 2023 armv7l GNU/Linux
Marvell Armada 388 (88F6828) ARM Cortex-A9
ARMv7 32-bit
Dual Core 1.6 Ghz
2 GB RAM
Docker version 20.10.5+dfsg1, build 55c4c88

Database Image

mysql Ver 15.1 Distrib 10.5.21-MariaDB, for debian-linux-gnueabihf (armv7l) using EditLine wrapper

user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
#datadir        = /var/lib/mysql
datadir         = /home/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#bind-address           = 127.0.0.1
bind-address            = 0.0.0.0

Issue-Link

https://github.com/bitwarden/server/issues/2480

Issue Tracking Info

[X] I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

karolszklarski commented 10 months ago

gitHash cf4d8a4 is working, ea162d5 is not

That commit modifies GrantRepository, and logs say something about 'grant' table

withinfocus commented 10 months ago

@karolszklarski would you be able to upgrade your MariaDB instance to the latest and try again? We aren't testing with Maria and also can't replicate this on MySQL 8. The Pomelo library that's used to perform these migrations reported such an issue but it was resolved ~3 years ago and we are suspecting that this has something to do with Maria installations.

karolszklarski commented 10 months ago

I upgraded to the latest available release in debian "testing" repository Ver 15.1 Distrib 10.11.6-MariaDB but still get the same error.

Isn't it the same issue as #3635 ?

withinfocus commented 10 months ago

Yes, same issue most likely even though the error messages for the migration aren't included.

I see you're still on Maria 10.x -- would you be willing to try using 11.x, even though it's not LTS? Or MySQL 8?

JoeGTN1 commented 10 months ago

I have upgraded to Maria 11.0, same issue persists.

withinfocus commented 10 months ago

@karolszklarski can you pull the latest from main and give this a try? We made a modification to the database migration.

dk-armada commented 10 months ago

I'm running a MariaDB backend also. Seeing this exact issue with the latest Docker image bitwarden/self-host:beta.

etaggart65 commented 10 months ago

Same exact issue using 10.11.2-MariaDB backend. Had to revert to 2024.1.0-beta.

withinfocus commented 10 months ago

2024.1.0 had this issue present so you'd be experiencing it on that version, but the latest images have resolved this.

etaggart65 commented 10 months ago

That's not what I'm seeing. If I run bitwarden/self-host:beta or specifically 2024.1.1-beta I get the error.

2024/01/22 09:10:44 | stdout | 2024-01-22 09:10:44,894 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)

I can run 2024.1.0-beta without any issues.

withinfocus commented 10 months ago

That error may be due to something else? There are admin logs that would reveal more.

etaggart65 commented 10 months ago

Here is the entry from the admin log when running 2024.1.1-beta

fail: Microsoft.EntityFrameworkCore.Database.Command[20102] Failed executing DbCommand (9ms) [Parameters=[], CommandType='Text', CommandTimeout='30'] CALL POMELO_BEFORE_DROP_PRIMARY_KEY(NULL, 'Grant'); ALTER TABLEGrantDROP PRIMARY KEY; Unhandled exception. MySqlConnector.MySqlException (0x80004005): Can't DROP INDEXPRIMARY; check that it exists at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 43 at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 130 at MySqlConnector.MySqlDataReader.CreateAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlDataReader.cs:line 468 at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList`1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/Core/CommandExecutor.cs:line 56 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlCommand.cs:line 296 at MySqlConnector.MySqlCommand.ExecuteNonQuery() in //src/MySqlConnector/MySqlCommand.cs:line 107 at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQuery(RelationalCommandParameterObject parameterObject) at Microsoft.EntityFrameworkCore.Migrations.MigrationCommand.ExecuteNonQuery(IRelationalConnection connection, IReadOnlyDictionary2 parameterValues) at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQuery(IEnumerable1 migrationCommands, IRelationalConnection connection) at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration) at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade) at Bit.MySqlMigrations.MySqlDbMigrator.MigrateDatabase(Boolean enableLogging, CancellationToken cancellationToken) in /source/util/MySqlMigrations/MySqlDbMigrator.cs:line 30 at Bit.Admin.HostedServices.DatabaseMigrationHostedService.StartAsync(CancellationToken cancellationToken) in /source/src/Admin/HostedServices/DatabaseMigrationHostedService.cs:line 29 at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host) at Bit.Admin.Program.Main(String[] args) in /source/src/Admin/Program.cs:line 9`

etaggart65 commented 10 months ago

My docker file..

`version: "3.8"

services: bitwarden: container_name: bitwarden hostname: bitwarden env_file:

settings2.env environment:
PUID=1034
PGID=100
TZ=America/New_York image: bitwarden/self-host:beta restart: unless-stopped network_mode: synobridge ports:
"9890:8080" volumes:
/volume1/docker/bitwarden/data:/etc/bitwarden
/volume1/docker/bitwarden/logs:/var/log/bitwarden`

withinfocus commented 10 months ago

https://github.com/bitwarden/server/issues/3651#issuecomment-1901004782 may be what you need.

etaggart65 commented 10 months ago

Ok thanks @withinfocus! that worked.

karolszklarski commented 10 months ago

I confirm, it works :)

stop bitwarden
login to database
execute ALTER TABLE Grant DROP COLUMN Id;
execute ALTER TABLE Grant ADD PRIMARY KEYPK_Grant (Key);
exit database, start bitwarden

bitwarden / server