search

bitwarden / server

Bitwarden infrastructure/backend (API, database, Docker, etc).
https://bitwarden.com
Other
15.1k stars 1.26k forks source link

Resource temporarily unavailable #456

Closed blissend closed 5 years ago

blissend commented 5 years ago

I am trying to setup bitwarden to utilize an external SMTP service that uses my domain. I've edited the global.override.env file like so...

globalSettings__mail__replyToEmail=no-reply@bitwarden.example.com
globalSettings__mail__smtp__host=smtp.thirdparty.com
globalSettings__mail__smtp__username=postmaster@example.com
globalSettings__mail__smtp__password=###############-######
globalSettings__mail__smtp__ssl=false
globalSettings__mail__smtp__port=587
globalSettings__mail__smtp__useDefaultCredentials=false
globalSettings__disableUserRegistration=true
adminSettings__admins=example@blah.com

After restarting with ./bitwarden.sh restart and sending verification email for the only registered user or setting up user at /admin I get this in the docker logs for the bitwarden-api container...

fail: Bit.Core.Services.SmtpMailDeliveryService[0]
      Mail send failed.
System.Net.Mail.SmtpException: Failure sending mail. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: Resource temporarily unavailable
   at System.Net.Dns.InternalGetHostByName(String hostName)
   at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
   at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.DoDnsCallback(IAsyncResult result, MultipleAddressConnectAsyncResult context)
   at System.Net.Sockets.Socket.DnsCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
   at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.InitializeConnectionCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.End(IAsyncResult result)
   at System.Net.Mail.SmtpTransport.EndGetConnection(IAsyncResult result)
   at System.Net.Mail.SmtpClient.ConnectCallback(IAsyncResult result)
   --- End of inner exception stack trace ---

The email is never sent and unsure what this error means or what to do. I have tested on the same server via swaks command to make sure SMTP works (it does).

Bitwarden (web 2.8.0 tag with everything else at 1.29.0)

kspearrin commented 5 years ago

Try changing globalSettings__mail__smtp__ssl to true

blissend commented 5 years ago

When setting ssl to true for smtp (then restarting) I get the same error unfortunately.

kspearrin commented 5 years ago

I am not sure then. Is there a firewall or something preventing outbound traffic from the docker containers? Can you ping the SMTP server from inside a container?

blissend commented 5 years ago

Yes there is a firewall but nothing specific to the containers. Ping isn't available within the containers (forgive my ignorance if there is a way) but I can use curl from bitwarden-api...

# curl -v -Is --connect-timeout 5 smtp.mailgun.org
* About to connect() to smtp.mailgun.org port 80 (#0)
*   Trying 35.172.4.119...
* After 2498ms connect time, move on!
*   Trying 52.205.27.158...
* After 1248ms connect time, move on!
*   Trying 52.45.97.210...
* After 623ms connect time, move on!
* Failed connect to smtp.mailgun.org:80; Operation now in progress
* Closing connection 0
# docker exec -it bitwarden-api /bin/bash
root@e0183555dbf5:/app# curl -v -Is --connect-timeout 5 smtp.mailgun.org
* Rebuilt URL to: smtp.mailgun.org/
* Could not resolve host: smtp.mailgun.org
* Closing connection 0

I'm using firewall-cmd and currently this is it...

# firewall-cmd --permanent --list-ports
32400/tcp 22/tcp 80/tcp 1900/udp 3005/tcp 5353/udp 32410-32414/udp 32469/tcp 443/tcp 53/tcp 53/udp 8001/tcp 587/tcp

Containers list if it helps...

# docker ps -a
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS                 PORTS                                                    NAMES
077cff11017c        bitwarden/notifications:1.29.0   "/entrypoint.sh"         5 minutes ago       Up 5 minutes           5000/tcp                                                 bitwarden-notifications
e0183555dbf5        bitwarden/api:1.29.0             "/entrypoint.sh"         5 minutes ago       Up 5 minutes           5000/tcp                                                 bitwarden-api
2e4fccb6162b        bitwarden/identity:1.29.0        "/entrypoint.sh"         5 minutes ago       Up 5 minutes           5000/tcp                                                 bitwarden-identity
85e6b026cf1f        bitwarden/mssql:1.29.0           "/entrypoint.sh"         5 minutes ago       Up 5 minutes           1433/tcp                                                 bitwarden-mssql
3d42b68435bb        bitwarden/web:2.8.0              "/entrypoint.sh"         5 minutes ago       Up 5 minutes           5000/tcp                                                 bitwarden-web
56cee782dd98        bitwarden/icons:1.29.0           "/entrypoint.sh"         5 minutes ago       Up 5 minutes           5000/tcp                                                 bitwarden-icons
a05a619068d9        bitwarden/attachments:1.29.0     "/entrypoint.sh"         5 minutes ago       Up 5 minutes           5000/tcp                                                 bitwarden-attachments
91c8470f4e6c        bitwarden/admin:1.29.0           "/entrypoint.sh"         5 minutes ago       Up 5 minutes           5000/tcp                                                 bitwarden-admin
21a6f181f781        bitwarden/nginx:1.29.0           "/entrypoint.sh"         5 minutes ago       Up 5 minutes           0.0.0.0:8080->8080/tcp, 80/tcp, 0.0.0.0:8443->8443/tcp   bitwarden-nginx
29f364e4f0a3        nginx:stable                     "nginx -g 'daemon of…"   13 days ago         Up 2 days                                                                       nginx
b1537a5cefed        plexinc/pms-docker:plexpass      "/init"                  3 weeks ago         Up 3 weeks (healthy)                                                            plex
caaddda5eb9f        tautulli/tautulli                "/init"                  8 months ago        Up 3 weeks             0.0.0.0:8181->8181/tcp                                   tautulli
kspearrin commented 5 years ago

Can you try temporarily disabling the firewall to see if that makes any difference?

blissend commented 5 years ago

To disable I just need to stop the firewalld service and delete the bridge created for docker (shown in ip a command). After doing that I rebuild bitwarden and start up again. Here's the status of firewall...

# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-03-14 17:14:49 EDT; 13min ago
     Docs: man:firewalld(1)
  Process: 823 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 823 (code=exited, status=0/SUCCESS)

Mar 14 16:52:33 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION-STAGE-1 -i br-76d7e8c3bd84 ! -o br-76d7e8c3bd84 -j DOCKER-ISOLATION-STAGE-2' failed: iptable...tch by that name.
Mar 14 16:52:33 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION-STAGE-2 -o br-76d7e8c3bd84 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Mar 14 16:52:34 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -p tcp -d 0/0 --dport 8443 -j DNAT --to-destination 172.18.0.3:8443 ! -i br-76d7e8c3bd84' failed: iptable...tch by that name.
Mar 14 16:52:34 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER ! -i br-76d7e8c3bd84 -o br-76d7e8c3bd84 -p tcp -d 172.18.0.3 --dport 8443 -j ACCEPT' failed: iptables:... in that chain?).
Mar 14 16:52:34 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -p tcp -s 172.18.0.3 -d 172.18.0.3 --dport 8443 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Mar 14 16:52:34 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 172.18.0.3:8080 ! -i br-76d7e8c3bd84' failed: iptable...tch by that name.
Mar 14 16:52:34 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER ! -i br-76d7e8c3bd84 -o br-76d7e8c3bd84 -p tcp -d 172.18.0.3 --dport 8080 -j ACCEPT' failed: iptables:... in that chain?).
Mar 14 16:52:34 mantisshrimp firewalld[823]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -p tcp -s 172.18.0.3 -d 172.18.0.3 --dport 8080 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Mar 14 17:14:48 mantisshrimp systemd[1]: Stopping firewalld - dynamic firewall daemon...
Mar 14 17:14:49 mantisshrimp systemd[1]: Stopped firewalld - dynamic firewall daemon.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mantisshrimp bitwarden]# iptables -L -v -n
Chain INPUT (policy ACCEPT 39082 packets, 41M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  442  556K DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  442  556K DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  406  554K ACCEPT     all  --  *      br-c1781e6b18a0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   36  2160 DOCKER     all  --  *      br-c1781e6b18a0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br-c1781e6b18a0 !br-c1781e6b18a0  0.0.0.0/0            0.0.0.0/0
   36  2160 ACCEPT     all  --  br-c1781e6b18a0 br-c1781e6b18a0  0.0.0.0/0            0.0.0.0/0
   82 75056 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
   85  9902 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 35406 packets, 26M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           tcp dpt:8181
    0     0 ACCEPT     tcp  --  !br-c1781e6b18a0 br-c1781e6b18a0  0.0.0.0/0            172.18.0.4           tcp dpt:8443
    0     0 ACCEPT     tcp  --  !br-c1781e6b18a0 br-c1781e6b18a0  0.0.0.0/0            172.18.0.4           tcp dpt:8080

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  br-c1781e6b18a0 !br-c1781e6b18a0  0.0.0.0/0            0.0.0.0/0
   85  9902 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
  609  641K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      br-c1781e6b18a0  0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
   85  9902 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
  609  641K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Running verification email again gives same error in bitwarden-api logs.

kspearrin commented 5 years ago

I am not sure. Do you have another SMTP server you could try as a test, like gmail or sendgrid?

blissend commented 5 years ago

So I noticed running curl within bitwarden-api container failed to resolve my external SMTP host unless I used the IP address. This prompted me to run docker run --rm busybox nslookup google.com which failed too until I provided the dns option docker run --rm --dns <routerDNS_IP> busybox nslookup google.com. Knowing that worked all I had to do was change the docker-compose YAML file to have a DNS option under each container settings or add { "dns": ["X.X.X.X", "XX.XX.XX.XX"] } to /etc/docker/daemon.json for system wide fix. After a bitwarden restart all is well.

Why this works I'm not certain but it has something to do with running an internal DNS (BIND9 on CentOS 7) on the server this runs on. My router DNS is just forwarding to internal DNS server so it's weird it can't resolve with local IP found in /etc/resolv.conf but can with router DNS. Confusing but good to know BIND9 can screw up local docker DNS resolutions.