Fails to start - (code=exited, status=101)

jeffshead commented 4 years ago

Subject of the issue

Bitwardenrs fails to start at server boot. However, it seems to run just fine if I run the "systemctl restart bitwardenrs" command after the server boots. I do not get any errors after restarting the service.

I don't have much Linux experience so please take that into consideration when asking for more info.

Your environment

Bitwarden_rs version: 2.12.0 (from the web login page) I could not find any other version #'s in the log or anywhere else.
Install method: Package (https://github.com/MrMEEE/bitwarden_rs_rpm)
Clients used: Web, Android, Windows
Reverse proxy and version: nginx/1.16.1
Version of mysql/postgresql: 5.5.64-MariaDB MariaDB Server
Other relevant information: CentOS 7, kernel 5.4.6-1.el7.elrepo.x86_64. I do NOT have Docker installed.

Steps to reproduce

Try to start bitwarden_rs at server boot via it's service. I currently have only root user on the server so that is what I have specified for user and group in the service file. I have everything set to defaults in \etc\bitwarden_rs\bitwarden-rs.conf except for DATABASE_URL which is properly set for the correct database. ### Expected behaviour

bitwarden_rs should start without error.

Actual behaviour

Fails to start with errors.

Relevant logs

Below is the service status after CentOS boots:

[root@vault ~]# systemctl status bitwarden_rs
 bitwarden_rs.service - Bitwarden Server (Rust Edition)
   Loaded: loaded (/usr/lib/systemd/system/bitwarden_rs.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2019-12-27 17:16:56 EST; 40s ago
     Docs: https://github.com/dani-garcia/bitwarden_rs
  Process: 1166 ExecStart=/opt/bitwarden_rs/server/bin/bitwarden_rs (code=exited, status=101)
 Main PID: 1166 (code=exited, status=101)

Dec 27 17:16:56 vault.mysite.com bitwarden_rs[1166]: at src/libstd/panic.rs:394
Dec 27 17:16:56 vault.mysite.com bitwarden_rs[1166]: 21:     0x555f8a7f0ea0 - std::rt::lang_start_internal::hcc4f56ca099ff096
Dec 27 17:16:56 vault.mysite.com bitwarden_rs[1166]: at src/libstd/rt.rs:51
Dec 27 17:16:56 vault.mysite.com bitwarden_rs[1166]: 22:     0x555f8a1370d2 - main
Dec 27 17:16:56 vault.mysite.com bitwarden_rs[1166]: 23:     0x7f085265a505 - __libc_start_main
Dec 27 17:16:56 vault.mysite.com bitwarden_rs[1166]: 24:     0x555f89f72ddb - <unknown>
Dec 27 17:16:56 vault.mysite.com bitwarden_rs[1166]: 25:                0x0 - <unknown>
Dec 27 17:16:56 vault.mysite.com systemd[1]: bitwarden_rs.service: main process exited, code=exited, status=101/n/a
Dec 27 17:16:56 vault.mysite.com systemd[1]: Unit bitwarden_rs.service entered failed state.
Dec 27 17:16:56 vault.mysite.com systemd[1]: bitwarden_rs.service failed.
[root@vault ~]# reboot

Below is the status after restarting the service and successfully logging in to BW:

[root@vault ~]# systemctl status bitwarden_rs
 bitwarden_rs.service - Bitwarden Server (Rust Edition)
   Loaded: loaded (/usr/lib/systemd/system/bitwarden_rs.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-12-27 18:29:30 EST; 1h 22min ago
     Docs: https://github.com/dani-garcia/bitwarden_rs
 Main PID: 1713 (bitwarden_rs)
   CGroup: /system.slice/bitwarden_rs.service
           ??1713 /opt/bitwarden_rs/server/bin/bitwarden_rs

Dec 27 18:47:37 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:37][lettre::smtp][INFO] d3br28f4-3br1-4v8e-b3w2-d38y48476df2: to=<me@mysite.com>
Dec 27 18:47:37 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:37][lettre::smtp][INFO] d3br28f4-3br1-4v8e-b3w2-d38y48476df2: conn_use=1, status=sent (2.6.0 Ok, message save...localhost>>)
Dec 27 18:47:37 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:37][bitwarden_rs::api::identity][INFO] User me@mysite.com logged in successfully. IP: 192.168.1.123
Dec 27 18:47:37 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:37][response][INFO] POST /identity/connect/token (login) => 200 OK
Dec 27 18:47:37 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:37][request][INFO] POST /notifications/hub/negotiate
Dec 27 18:47:37 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:37][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK
Dec 27 18:47:38 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:38][request][INFO] GET /api/sync
Dec 27 18:47:38 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:47:38][response][INFO] GET /api/sync?<data..> (sync) => 200 OK
Dec 27 18:49:55 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:49:55][request][INFO] POST /notifications/hub/negotiate
Dec 27 18:49:55 vault.mysite.com bitwarden_rs[1713]: [2019-12-27 18:49:55][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK
Hint: Some lines were ellipsized, use -l to show in full.
[root@vault ~]#

mprasil commented 4 years ago

It seems that relevant part of the error message is missing from the status output. Can you provide full failed start log? You can see service logs with journalctl -u bitwarden_rs.

If I had to guess right now, it seems like it might start too early and some of the dependencies (network, filesystem mount,...) are not yet ready. But full log will tell us more.

jeffshead commented 4 years ago

[root@vault ~]# journalctl -u bitwarden_rs -- Logs begin at Sat 2019-12-28 03:14:20 EST, end at Sat 2019-12-28 03:14:37 EST. -- Dec 28 03:14:24 vault.mysite.com systemd[1]: [/usr/lib/systemd/system/bitwarden_rs.service:21] Failed to parse protect system value, ignoring: strict Dec 28 03:14:29 vault.mysite.com systemd[1]: Started Bitwarden Server (Rust Edition). Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: /--------------------------------------------------------------------\ Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: | Starting Bitwarden_RS | Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: |--------------------------------------------------------------------| Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: | This is an *unofficial* Bitwarden implementation, DO NOT use the | Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: | official channels to report bugs/features, regardless of client. | Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: | Report URL: https://github.com/dani-garcia/bitwarden_rs/issues/new | Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: \--------------------------------------------------------------------/ Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: thread 'main' panicked at 'Can't connect to DB: BadConnection("Can\'t connect to local MySQL server through socket \'/var/lib/mysql/mysql.sock Dec 28 03:14:29 vault.mysite.com bitwarden_rs[1164]: stack backtrace: Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 0: 0x559dd55f90b4 - backtrace::backtrace::libunwind::trace::h71335cfc631605fe Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 1: 0x559dd55f90b4 - backtrace::backtrace::trace_unsynchronized::h040535d06686dfd9 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 2: 0x559dd55f90b4 - std::sys_common::backtrace::_print_fmt::h72dcc396a6104258 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/sys_common/backtrace.rs:84 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 3: 0x559dd55f90b4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he3d4a17dc3fffc1b Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/sys_common/backtrace.rs:61 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 4: 0x559dd562166c - core::fmt::write::ha1a9a6794f43be2d Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libcore/fmt/mod.rs:1025 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 5: 0x559dd55f1e17 - std::io::Write::write_fmt::hc8ddc8bf09df58a0 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/io/mod.rs:1426 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 6: 0x559dd55fb26e - std::sys_common::backtrace::_print::hd857dc29f8bac85f Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/sys_common/backtrace.rs:65 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 7: 0x559dd55fb26e - std::sys_common::backtrace::print::h233c9aaa2ba179c4 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/sys_common/backtrace.rs:50 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 8: 0x559dd55fb26e - std::panicking::default_hook::{{closure}}::h0db8ece81993eee1 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/panicking.rs:193 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 9: 0x559dd55faf61 - std::panicking::default_hook::hbc2e7eb45ed63cb8 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/panicking.rs:210 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 10: 0x559dd55fb94b - std::panicking::rust_panic_with_hook::ha0a47579a1c5ce17 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/panicking.rs:471 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 11: 0x559dd55fb4fe - rust_begin_unwind Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/panicking.rs:375 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 12: 0x559dd561dc0e - core::panicking::panic_fmt::hfecda217c72674fe Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libcore/panicking.rs:84 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 13: 0x559dd561dd07 - core::result::unwrap_failed::h60fdd8c5f8b4e2c5 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libcore/result.rs:1189 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 14: 0x559dd4f3a45b - bitwarden_rs::main::hf45141613e78909c Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 15: 0x559dd4ff3673 - std::rt::lang_start::{{closure}}::h16268225ee6ead92 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 16: 0x559dd55fb393 - std::rt::lang_start_internal::{{closure}}::h3ce0d67d2233e4ab Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/rt.rs:52 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 17: 0x559dd55fb393 - std::panicking::try::do_call::hbe2d73f88b7d35e5 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/panicking.rs:292 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 18: 0x559dd56054fa - __rust_maybe_catch_panic Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libpanic_unwind/lib.rs:78 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 19: 0x559dd55fbea0 - std::panicking::try::h785e7263aeda5240 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/panicking.rs:270 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 20: 0x559dd55fbea0 - std::panic::catch_unwind::h94ff5d315ff1583e Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/panic.rs:394 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 21: 0x559dd55fbea0 - std::rt::lang_start_internal::hcc4f56ca099ff096 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: at src/libstd/rt.rs:51 Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 22: 0x559dd4f420d2 - main Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 23: 0x7feafbe81505 - __libc_start_main Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 24: 0x559dd4d7dddb - <unknown> Dec 28 03:14:30 vault.mysite.com bitwarden_rs[1164]: 25: 0x0 - <unknown> Dec 28 03:14:30 vault.mysite.com systemd[1]: bitwarden_rs.service: main process exited, code=exited, status=101/n/a Dec 28 03:14:30 vault.mysite.com systemd[1]: Unit bitwarden_rs.service entered failed state. Dec 28 03:14:30 vault.mysite.com systemd[1]: bitwarden_rs.service failed.

jeffshead commented 4 years ago

I think I fixed it.

In _\usr\lib\systemd\system\bitwarden_rs.service_, I added the bold text below:

... [Unit] Description=Bitwarden Server (Rust Edition) Documentation=https://github.com/dani-garcia/bitwarden_rs After=network.target mariadb.service Requires=mariadb.service

[Service] #The user/group bitwarden_rs is run under. the working directory (see below) should allow write and read access to this user/group User=root Group=root #The location of the .env file for configuration ...

mprasil commented 4 years ago

Glad you fixed it. I think you might want to undo the changes you did and instead do:

systemctl edit bitwarden_rs

And then add these two lines:

After= mariadb.service
Requires=mariadb.service

This way the override is added to separate file /etc/systemd/system/bitwarden_rs.service.d/override.conf that won't be overwritten by rpm package update.

mqus commented 4 years ago

I'll try to add some more info to the wiki.

dani-garcia / vaultwarden