search

dani-garcia / vaultwarden

Unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs
GNU Affero General Public License v3.0
38.63k stars 1.87k forks source link

rocket segfault make vaultwarden exit #3912

Closed Asutorufa closed 1 year ago

Asutorufa commented 1 year ago

Subject of the issue

rocket segfault make vaultwarden exit

[5620919.655731] rocket-worker-t[2551213]: segfault at 0 ip 00007f1d72b0b404 sp 00007f1d6f858388 error 4 in vaultwarden[7f1d7108a000+23d9000]
[5620919.660905] Code: 00 00 0f 1f 40 00 48 85 ff 74 03 8b 07 c3 b8 ff ff ff ff c3 66 90 48 85 ff 74 1b 85 f6 78 17 39 37 7e 13 48 8b 47 08 48 63 f6 <48> 8b 04 f0 c3 0f 1f 80 00 00 00 00 31 c0 c3 66 66 2e 0f 1f 84 00
[6587176.877824] rocket-worker-t[2996487]: segfault at 43054270 ip 00007f4e729065ff sp 00007f4e6eebae28 error 4 in vaultwarden[7f4e70aee000+23d9000]
[6587176.886577] Code: 80 7f fc 00 74 12 85 d2 74 01 f4 48 63 57 f8 81 fa ff ff 00 00 7f 01 f4 89 d0 c1 e0 04 48 98 48 29 c7 48 8b 47 f0 48 8d 77 f0 <48> 39 70 10 74 01 f4 40 8a 70 20 83 e6 1f 39 ce 7d 01 f4 8b 70 18
[7220307.815947] traps: rocket-worker-t[3262850] general protection fault ip:7fe609a57625 sp:7fe6058efe58 error:0 in vaultwarden[7fe607c3f000+23d9000]

Deployment environment

Your environment (Generated via diagnostics page)

Config (Generated via diagnostics page)

Show Running Config **Environment settings which are overridden:** ADMIN_TOKEN ```json { "_duo_akey": null, "_enable_duo": false, "_enable_email_2fa": false, "_enable_smtp": true, "_enable_yubico": true, "_icon_service_csp": "", "_icon_service_url": "", "_ip_header_enabled": true, "_smtp_img_src": "cid:", "admin_ratelimit_max_burst": 3, "admin_ratelimit_seconds": 300, "admin_session_lifetime": 20, "admin_token": "***", "allowed_iframe_ancestors": "", "attachments_folder": "data/attachments", "auth_request_purge_schedule": "30 * * * * *", "authenticator_disable_time_drift": false, "data_folder": "data", "database_conn_init": "", "database_max_conns": 10, "database_timeout": 30, "database_url": "***************", "db_connection_retries": 15, "disable_2fa_remember": false, "disable_admin_token": false, "disable_icon_download": false, "domain": "****://*********", "domain_origin": "****://*********", "domain_path": "", "domain_set": true, "duo_host": null, "duo_ikey": null, "duo_skey": null, "email_attempts_limit": 3, "email_expiration_time": 600, "email_token_size": 6, "emergency_access_allowed": true, "emergency_notification_reminder_schedule": "0 3 * * * *", "emergency_request_timeout_schedule": "0 7 * * * *", "enable_db_wal": true, "event_cleanup_schedule": "0 10 0 * * *", "events_days_retain": null, "extended_logging": true, "helo_name": null, "hibp_api_key": null, "icon_blacklist_non_global_ips": true, "icon_blacklist_regex": null, "icon_cache_folder": "data/icon_cache", "icon_cache_negttl": 259200, "icon_cache_ttl": 2592000, "icon_download_timeout": 10, "icon_redirect_code": 302, "icon_service": "internal", "incomplete_2fa_schedule": "30 * * * * *", "incomplete_2fa_time_limit": 3, "invitation_expiration_hours": 120, "invitation_org_name": "Vaultwarden", "invitations_allowed": true, "ip_header": "X-Real-IP", "job_poll_interval_ms": 30000, "log_file": null, "log_level": "Info", "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f", "login_ratelimit_max_burst": 10, "login_ratelimit_seconds": 60, "org_attachment_limit": null, "org_creation_users": "", "org_events_enabled": false, "org_groups_enabled": false, "password_hints_allowed": true, "password_iterations": 600000, "push_enabled": false, "push_installation_id": "***", "push_installation_key": "***", "push_relay_uri": "https://push.bitwarden.com", "reload_templates": false, "require_device_email": false, "rsa_key_filename": "data/rsa_key", "send_purge_schedule": "0 5 * * * *", "sendmail_command": null, "sends_allowed": true, "sends_folder": "data/sends", "show_password_hint": false, "signups_allowed": false, "signups_domains_whitelist": "", "signups_verify": false, "signups_verify_resend_limit": 6, "signups_verify_resend_time": 3600, "smtp_accept_invalid_certs": false, "smtp_accept_invalid_hostnames": false, "smtp_auth_mechanism": null, "smtp_debug": false, "smtp_embed_images": true, "smtp_explicit_tls": null, "smtp_from": "", "smtp_from_name": "Vaultwarden", "smtp_host": null, "smtp_password": null, "smtp_port": 587, "smtp_security": "starttls", "smtp_ssl": null, "smtp_timeout": 15, "smtp_username": null, "templates_folder": "data/templates", "tmp_folder": "data/tmp", "trash_auto_delete_days": null, "trash_purge_schedule": "0 5 0 * * *", "use_sendmail": false, "use_syslog": false, "user_attachment_limit": null, "web_vault_enabled": true, "web_vault_folder": "web-vault/", "websocket_address": "0.0.0.0", "websocket_enabled": false, "websocket_port": 3012, "yubico_client_id": null, "yubico_secret_key": null, "yubico_server": null } ```

Steps to reproduce

random

Expected behaviour

normal running

Actual behaviour

random exit

Troubleshooting data

$ sudo dmesg
[5620919.655731] rocket-worker-t[2551213]: segfault at 0 ip 00007f1d72b0b404 sp 00007f1d6f858388 error 4 in vaultwarden[7f1d7108a000+23d9000]
[5620919.660905] Code: 00 00 0f 1f 40 00 48 85 ff 74 03 8b 07 c3 b8 ff ff ff ff c3 66 90 48 85 ff 74 1b 85 f6 78 17 39 37 7e 13 48 8b 47 08 48 63 f6 <48> 8b 04 f0 c3 0f 1f 80 00 00 00 00 31 c0 c3 66 66 2e 0f 1f 84 00
[6587176.877824] rocket-worker-t[2996487]: segfault at 43054270 ip 00007f4e729065ff sp 00007f4e6eebae28 error 4 in vaultwarden[7f4e70aee000+23d9000]
[6587176.886577] Code: 80 7f fc 00 74 12 85 d2 74 01 f4 48 63 57 f8 81 fa ff ff 00 00 7f 01 f4 89 d0 c1 e0 04 48 98 48 29 c7 48 8b 47 f0 48 8d 77 f0 <48> 39 70 10 74 01 f4 40 8a 70 20 83 e6 1f 39 ce 7d 01 f4 8b 70 18
[7220307.815947] traps: rocket-worker-t[3262850] general protection fault ip:7fe609a57625 sp:7fe6058efe58 error:0 in vaultwarden[7fe607c3f000+23d9000]
BlackDex commented 1 year ago

Are there any logs of Vaultwarden it self? The current log's posted are not useful for us to even start figuring out what could be the issue.

Asutorufa commented 1 year ago

I see no panic in log, it just exit and then i restart it. This behavior only in the 1.29.2, 1.29.1 everything is ok.

[2023-09-22 19:00:32.840][response][INFO] (config) GET /api/config => 200 OK
[2023-09-22 19:00:55.905][request][INFO] PUT /api/ciphers/b30c7dbb-d9e7-4233-800c-c7e77fe76df7
[2023-09-22 19:00:55.907][response][INFO] (put_cipher) PUT /api/ciphers/<uuid> => 200 OK
[2023-09-22 19:04:28.215][request][INFO] PUT /api/ciphers/b30c7dbb-d9e7-4233-800c-c7e77fe76df7
[2023-09-22 19:04:28.217][response][INFO] (put_cipher) PUT /api/ciphers/<uuid> => 200 OK
[2023-09-22 19:09:48.291][request][INFO] PUT /api/ciphers/b30c7dbb-d9e7-4233-800c-c7e77fe76df7
[2023-09-22 19:09:48.293][response][INFO] (put_cipher) PUT /api/ciphers/<uuid> => 200 OK
[2023-09-22 19:18:12.888][request][INFO] PUT /api/ciphers/b30c7dbb-d9e7-4233-800c-c7e77fe76df7
[2023-09-22 19:18:12.891][response][INFO] (put_cipher) PUT /api/ciphers/<uuid> => 200 OK
                                          ...
[2023-09-22 20:01:41.119][request][INFO] GET /icons/passport.yandex.com/icon.png
[2023-09-22 20:01:41.120][response][INFO] (icon_internal) GET /icons/<domain>/icon.png => 200 OK
[2023-09-22 20:01:41.121][response][INFO] (icon_internal) GET /icons/<domain>/icon.png => 200 OK
/--------------------------------------------------------------------\
|                        Starting Vaultwarden                        |
|                           Version 1.29.2                           |
|--------------------------------------------------------------------|
| This is an *unofficial* Bitwarden implementation, DO NOT use the   |
| official channels to report bugs/features, regardless of client.   |
| Send usage/configuration questions or feature requests to:         |
|   https://github.com/dani-garcia/vaultwarden/discussions or        |
|   https://vaultwarden.discourse.group/                             |
| Report suspected bugs/issues in the software itself at:            |
|   https://github.com/dani-garcia/vaultwarden/issues/new            |
\--------------------------------------------------------------------/

[INFO] Using saved config from `data/config.json` for configuration.

[WARNING] The following environment variables are being overriden by the config.json file.
[WARNING] Please use the admin panel to make changes to them:
[WARNING] ADMIN_TOKEN

[NOTICE] You are using a plain text `ADMIN_TOKEN` which is insecure.
Please generate a secure Argon2 PHC string by using `vaultwarden hash` or `argon2`.
See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token

[2023-09-22 20:01:41.662][start][INFO] Rocket has launched from http://0.0.0.0:80
BlackDex commented 1 year ago

And what about the testing-alpine tagged image?

Asutorufa commented 1 year ago

OK, I'll try.

mirolm commented 1 year ago

I saw this several times when clients fetch alot of icons from added entries. Sometimes it crashes 2-3 times before client gives up and leaves the site without icon. Below are some of the sites that have missing icons after such refreshes:

https://bulbankonline.bg/ https://brunata.bg/

Not sure this is the exact cause tho, but hope this helps. The above two sites had icons before the new icon code was introduced.

BlackDex commented 1 year ago

@mirolm, which deployment do you use? Docker? Debian, Alpine? Which architecture? Which version are you using? Is it the current stable or testing?

mirolm commented 1 year ago

I use standalone install with executables extracted from testing-alpine running on debian 11 x64. Build my client shows is 1.29.2-bbd630f1.

BlackDex commented 1 year ago

I use standalone install with executables extracted from testing-alpine running on debian 11 x64. Build my client shows is 1.29.2-bbd630f1.

@mirolm could you please try the build from yesterday? It has some updated libraries.

mirolm commented 1 year ago

Now my client reports build 1.29.2-bc26bfa5. It works so far without crashes, but is there a way to invalidate the server icon cache so i can test my guess?

I noticed the server is pretty rock solid in it's daily operation. The only times i noticed it crash is when it pulls tons of icons all at once.

BlackDex commented 1 year ago

@mirolm this can be done in two ways.

  1. Set a lower cache time. https://github.com/dani-garcia/vaultwarden/blob/bc26bfa589c007da9b9be37e1172060f38a948b9/.env.template#L63..L66
  2. Remove the files from the cache folder.

The first one setting to something like 1 or 2 second would probably work.

mirolm commented 1 year ago

Yea it crashed after i cleared the browser cache too. Last rows of the log are:

Sep 23 13:24:09 altair vaultwarden[1522133]: [2023-09-23 13:24:09.942][request][INFO] GET /icons/brunata.bg/icon.png Sep 23 13:24:10 altair systemd[1]: vaultwarden.service: Main process exited, code=killed, status=11/SEGV

Sep 23 13:24:10 altair systemd[1]: vaultwarden.service: Failed with result 'signal'.

[578896.207012] traps: rocket-worker-t[1522143] general protection fault ip:7f04448f3da4 sp:7f0441a4bdf0 error:0 in vaultwarden[7f0442a72000+244f000] [578977.951013] traps: rocket-worker-t[1522351] general protection fault ip:7f384ac6cc05 sp:7f3847f81a58 error:0 [579160.640653] traps: rocket-worker-t[1522448] general protection fault ip:7f63c4dec8ff sp:7f63c1b3ebd8 error:0 in vaultwarden[7f63c2f6b000+244f000]

Sooo i guess this site makes it crash for me.

BlackDex commented 1 year ago

Ill see of i can replicate it.

BlackDex commented 1 year ago

I can replicate it. It seems to be an Alpine thingy only. It doesn't break on the Debian container for me. Ill have to check it out.

BlackDex commented 1 year ago

So, 1.29.1 seems to work just fine. Everything after that fails. It could be multiple items.

BlackDex commented 1 year ago

Found the issue. It seems to be a combination of the OpenSSL version and the OpenSSL Crate which doesn't seem to use v3.x.x OpenSSL version. This probably causes some API calls to be incompatible and break.

BlackDex commented 1 year ago

Hmmm, thought it was that, but for some reason it broke again. I now see that if i increase ROCKET_WORKERS it doesn't fail. So it must be something with threading.

sidewinder94 commented 1 year ago

@BlackDex This is probably the issue I had and asked about on the matrix channel (Irwene). Feel free to ask if you want me to try things on my end

BlackDex commented 1 year ago

I have figured it out already. And can reproduce it. Now, i only need to fix it fully. Running into some compile issues.

BlackDex commented 1 year ago

Ok, i didn't knew me merging my PR on my Repo would close the issue in this repo :). But technically it is fixed, though no new release yet of Vaultwarden which contains this fix.

bo0tzz commented 1 year ago

Should this issue be reopened until the fix is in Vaultwarden itself?

BlackDex commented 1 year ago

No, not really. Since there is nothing to fix in Vaultwarden. We should only trigger a new build, for which i am also working on some changed.

Crashdummyy commented 1 year ago

@BlackDex So how can I handle this in the meantime ?

I disabled icon downloads and increased the WorkerPool to 30 it however still crashes about 200 times a day

BlackDex commented 1 year ago

Revert to v1.29.1 or use debian instead of alpine.

Crashdummyy commented 1 year ago

Revert to v1.29.1 or use debian instead of alpine.

Thanks for the quick reply. I am in somewhat the same situation as this issuer.

I'm on postgres and when I change the base image from testing-alpine to testing I cant start the server. ./vaultwarden: error while loading shared libraries: libmariadb.so.3: cannot open shared object file: No such file or directory

Is there a way to supress this or do I need to install the mariadb

BlackDex commented 1 year ago

say what? That is on the current testing?

Crashdummyy commented 1 year ago

say what? That is on the current testing?

on the non testing as well. I deploy the binaries like this.

docker pull vaultwarden/server:testing
docker create --name vaultwardenTemp vaultwarden/server:testing
docker cp vaultwardenTemp:/vaultwarden .
docker cp vaultwardenTemp:/web-vault .

before that I always used server:alpine-testing which can be started but crashes for now

BlackDex commented 1 year ago

Ah, you extract the binary. That will not work with the debian based containers, as they are linked to other shared libraries. I'm working on the rebuild of the building right now.

One other thing you could do is build the container your self right now and extract it from there. That will use the new Alpine build containers and solve your issue.

Crashdummyy commented 1 year ago

Ah, you extract the binary. That will not work with the debian based containers, as they are linked to other shared libraries. I'm working on the rebuild of the building right now.

One other thing you could do is build the container your self right now and extract it from there. That will use the new Alpine build containers and solve your issue.

I guess Ill do that for now unitl the new release is available. Thanks a lot for your help :)

BlackDex commented 1 year ago

The problem is, if i trigger a new testing now, it will still have a broken debian build for armv6. I want to have both solved instead of creating two different testing releases. That only makes it confusing.