search

dani-garcia / vaultwarden

Unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs
GNU Affero General Public License v3.0
38.95k stars 1.89k forks source link

Some website icons won't download - 404 error in logs #1540

Closed FloatingMilkshake closed 3 years ago

FloatingMilkshake commented 3 years ago

Subject of the issue

Some website icons won't download (namely those for dell.com and guilded.gg)

Deployment environment

Your environment (Generated via diagnostics page)

Config (Generated via diagnostics page)

{
  "_duo_akey": "***",
  "_enable_duo": true,
  "_enable_email_2fa": false,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_ip_header_enabled": true,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_max_conns": 10,
  "database_url": "****/**.*******",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://*********.*****************.*******.***",
  "domain_origin": "*****://*********.*****************.*******.***",
  "domain_path": "",
  "domain_set": true,
  "duo_host": "api-ea899181.duosecurity.com",
  "duo_ikey": "DI2J7V68VV1DN6ZFIXZH",
  "duo_skey": "***",
  "email_attempts_limit": 3,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "enable_db_wal": true,
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "invitation_org_name": "Bitwarden_RS",
  "invitations_allowed": true,
  "ip_header": "X-Real-IP",
  "log_file": null,
  "log_level": "Info",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "org_attachment_limit": null,
  "org_creation_users": "",
  "password_iterations": 500000,
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "show_password_hint": true,
  "signups_allowed": false,
  "signups_domains_whitelist": "",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_explicit_tls": false,
  "smtp_from": "",
  "smtp_from_name": "Bitwarden_RS",
  "smtp_host": null,
  "smtp_password": null,
  "smtp_port": 587,
  "smtp_ssl": true,
  "smtp_timeout": 15,
  "smtp_username": null,
  "templates_folder": "data/templates",
  "use_syslog": false,
  "user_attachment_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": true,
  "websocket_port": 3012,
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

Steps to reproduce

The only different parameters that I can remember are those for global Duo Security settings, the setting to disable signups, and the admin token. Other than that I'm not doing anything differently. I'm starting bitwarden_rs the same way I usually do. When using Bitwarden's official server (vault.bitwarden.com), the icons load fine.

Expected behaviour

All icons load

Actual behaviour

Icons for dell.com and guilded.gg are missing, instead the placeholder icon shows

Troubleshooting data

Logs when bitwarden_rs tries to fetch the icons: dell.com:

[2021-03-26 11:29:33.694][request][INFO] GET /icons/www.dell.com/icon.png
[2021-03-26 11:29:33.694][response][INFO] GET /icons/<domain>/icon.png (icon) => 404 Not Found

You can see that it fails with a 404 error. But with guilded.gg:

[2021-03-24 15:59:09.051][request][INFO] GET /icons/www.guilded.gg/icon.png
[2021-03-24 15:59:09.068][request][INFO] GET /icons/imgur.com/icon.png
[2021-03-24 15:59:09.068][response][INFO] GET /icons/<domain>/icon.png (icon) => 200 OK

...it seems to move on to the next icon without showing a response? The line after these three is just another request for a different site.

BlackDex commented 3 years ago

Thx for the report. I see that upstream is able to download the icon got at least dell. I will see what i can do to fix this. But some sites are hard to extract icons from because of some security measures.

BlackDex commented 3 years ago

Dell should work, but they do have some site protection which could cause some issues. The other one was because it enforced gzip compression but that feature wasn't enabled, after i did that it worked.

FloatingMilkshake commented 3 years ago

After pulling the latest image, something strange happened. It seems like bitwarden_rs is trying to download all icons at once and is failing for all of them. While it repeatedly tries and fails, my vault is totally inaccessible - "waiting for \<domain>...". I had to edit config.json and disallow icon downloads and restart the container to be able to access my vault again. Now no icons are visible while downloads are disabled and there are several .miss files in my icon cache folder.

So whatever changes were made in that PR apparently broke all icon downloads? I haven't changed anything on my side other than making my icon cache no longer temporary (before it was mounted to a non-persistent volume, now it's back in the default location inside bw-data/icon_cache).

BlackDex commented 3 years ago

And which version are you running? What tag are you using? If all are miss, then it could also be that the instance didn't had access to the internet.

FloatingMilkshake commented 3 years ago

And which version are you running?

1.20.0

What tag are you using?

:latest

it could also be that the instance didn't had access to the internet.

Looks like it doesn't? The bitwarden_rs admin diagnostics page shows no internet access. It had internet access before...how can I fix that? The PC it's running on has internet access, so why wouldn't bitwarden_rs?