2024.4.2 update broke my docker-compose.yml

SynVisions commented 5 months ago

Steps To Reproduce

edit: I believe I have the root cause (and thus steps to reproduce) here: https://github.com/bitwarden/server/issues/4078#issuecomment-2105534684 -- it seems that docker-compose 1.24.1 does not work with bitwarden-server 2024.4.2 despite current documentation stating that it should work.

I'm not sure if this is easily reproducible as I've been running the same self-hosted Bitwarden instance for years now, but this is what occurred in my environment. After updating from self-hosted bitwarden 2024.4.1 to 2024.4.2, my docker-compose.yml is now invalid, and bitwarden will no longer start.

The update was applied in this fashion:

./bitwarden.sh updateself
./bitwarden.sh update

Expected Result

When running ./bitwarden.sh start after updating I would expect to see something like this (except replace 2024.4.1 with 2024.4.2):

Apr 21 02:01:12 bitwarden bitwarden.sh[1746]:  _     _ _                         _
Apr 21 02:01:12 bitwarden bitwarden.sh[1746]: | |__ (_) |___      ____ _ _ __ __| | ___ _ __
Apr 21 02:01:12 bitwarden bitwarden.sh[1746]: | '_ \| | __\ \ /\ / / _` | '__/ _` |/ _ \ '_ \
Apr 21 02:01:12 bitwarden bitwarden.sh[1746]: | |_) | | |_ \ V  V / (_| | | | (_| |  __/ | | |
Apr 21 02:01:12 bitwarden bitwarden.sh[1746]: |_.__/|_|\__| \_/\_/ \__,_|_|  \__,_|\___|_| |_|
Apr 21 02:01:12 bitwarden bitwarden.sh[1747]: Open source password management solutions
Apr 21 02:01:12 bitwarden bitwarden.sh[1747]: Copyright 2015-2024, 8bit Solutions LLC
Apr 21 02:01:12 bitwarden bitwarden.sh[1747]: https://bitwarden.com, https://github.com/bitwarden
Apr 21 02:01:12 bitwarden bitwarden.sh[1747]: ===================================================
Apr 21 02:01:12 bitwarden bitwarden.sh[1745]: bitwarden.sh version 2024.4.1
Apr 21 02:01:12 bitwarden bitwarden.sh[1752]: Docker version 24.0.5, build 24.0.5-0ubuntu1~22.04.1
Apr 21 02:01:13 bitwarden bitwarden.sh[1759]: docker-compose version 1.24.1, build 4667896b
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling mssql         ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling web           ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling attachments   ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling api           ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling identity      ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling sso           ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling admin         ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling icons         ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling notifications ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling events        ...
Apr 21 02:01:15 bitwarden bitwarden.sh[1777]: Pulling nginx         ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1777]: [2.8K blob data]
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating network "docker_default" with the default driver
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating network "docker_public" with the default driver
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-identity ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-attachments ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-web         ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-icons       ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-events      ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-mssql       ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-notifications ...
Apr 21 02:01:19 bitwarden bitwarden.sh[1793]: Creating bitwarden-api           ...
Apr 21 02:01:18 bitwarden bitwarden.sh[1793]: Creating bitwarden-sso           ...
Apr 21 02:01:20 bitwarden bitwarden.sh[1793]: [91B blob data]
Apr 21 02:01:21 bitwarden bitwarden.sh[1793]: [533B blob data]
Apr 21 02:01:25 bitwarden bitwarden.sh[1793]: [55B blob data]
Apr 21 02:01:25 bitwarden bitwarden.sh[3050]: 2024.4.1: Pulling from bitwarden/setup
Apr 21 02:01:25 bitwarden bitwarden.sh[3050]: Digest: sha256:b69e7a14ccbc1675128eba8bf5f0db519b83e26913efa830c20d8102306342d1
Apr 21 02:01:25 bitwarden bitwarden.sh[3050]: Status: Image is up to date for bitwarden/setup:2024.4.1
Apr 21 02:01:25 bitwarden bitwarden.sh[3050]: docker.io/bitwarden/setup:2024.4.1
Apr 21 02:01:54 bitwarden bitwarden.sh[3068]: Bitwarden is up and running!
Apr 21 02:01:54 bitwarden bitwarden.sh[3068]: ===================================================
Apr 21 02:01:54 bitwarden bitwarden.sh[3068]: visit https://bitwarden.dhcp
Apr 21 02:01:54 bitwarden bitwarden.sh[3068]: to update, run `./bitwarden.sh updateself` and then `./bitwarden.sh update`

Actual Result

What I see now:

sean@bitwarden:~$ ./bitwarden.sh start
 _     _ _                         _
| |__ (_) |___      ____ _ _ __ __| | ___ _ __
| '_ \| | __\ \ /\ / / _` | '__/ _` |/ _ \ '_ \
| |_) | | |_ \ V  V / (_| | | | (_| |  __/ | | |
|_.__/|_|\__| \_/\_/ \__,_|_|  \__,_|\___|_| |_|

Open source password management solutions
Copyright 2015-2024, 8bit Solutions LLC
https://bitwarden.com, https://github.com/bitwarden

===================================================

bitwarden.sh version 2024.4.2
Docker version 24.0.5, build 24.0.5-0ubuntu1~22.04.1
docker-compose version 1.24.1, build 4667896b

ERROR: The Compose file '/home/sean/bwdata/docker/docker-compose.yml' is invalid because:
Unsupported config option for services: 'notifications'
Unsupported config option for networks: 'public'
ERROR: The Compose file '/home/sean/bwdata/docker/docker-compose.yml' is invalid because:
Unsupported config option for services: 'identity'
Unsupported config option for networks: 'public'

I get different sets of errors each time I run it, for example the next time I got these:

ERROR: The Compose file '/home/sean/bwdata/docker/docker-compose.yml' is invalid because:
Unsupported config option for services: 'api'
Unsupported config option for networks: 'default'
ERROR: The Compose file '/home/sean/bwdata/docker/docker-compose.yml' is invalid because:
Unsupported config option for networks: 'default'
Unsupported config option for services: 'web'

Screenshots or Videos

No response

Additional Context

This is my docker-compose.yml:

#
# Useful references:
# https://docs.docker.com/compose/compose-file/
# https://docs.docker.com/compose/reference/overview/#use--f-to-specify-name-and-path-of-one-or-more-compose-files
# https://docs.docker.com/compose/reference/envvars/
#
#########################################################################
# WARNING: This file is generated. Do not make changes to this file.    #
# They will be overwritten on update. If you want to make additions to  #
# this file, you can create a `docker-compose.override.yml` file in the #
# same directory and it will be merged into this file at runtime. You   #
# can also manage various settings used in this file from the           #
# ./bwdata/config.yml file for your installation.                       #
#########################################################################

services:
  mssql:
    image: bitwarden/mssql:2024.4.2
    container_name: bitwarden-mssql
    restart: always
    stop_grace_period: 60s
    volumes:
      - ../mssql/data:/var/opt/mssql/data
      - ../logs/mssql:/var/opt/mssql/log
      - ../mssql/backups:/etc/bitwarden/mssql/backups
    env_file:
      - mssql.env
      - ../env/uid.env
      - ../env/mssql.override.env

  web:
    image: bitwarden/web:2024.4.2
    container_name: bitwarden-web
    restart: always
    volumes:
      - ../web:/etc/bitwarden/web
    env_file:
      - global.env
      - ../env/uid.env

  attachments:
    image: bitwarden/attachments:2024.4.2
    container_name: bitwarden-attachments
    restart: always
    volumes:
      - ../core/attachments:/etc/bitwarden/core/attachments
    env_file:
      - global.env
      - ../env/uid.env

  api:
    image: bitwarden/api:2024.4.2
    container_name: bitwarden-api
    restart: always
    volumes:
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/api:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  identity:
    image: bitwarden/identity:2024.4.2
    container_name: bitwarden-identity
    restart: always
    volumes:
      - ../identity:/etc/bitwarden/identity
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/identity:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  sso:
    image: bitwarden/sso:2024.4.2
    container_name: bitwarden-sso
    restart: always
    volumes:
      - ../identity:/etc/bitwarden/identity
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/sso:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  admin:
    image: bitwarden/admin:2024.4.2
    container_name: bitwarden-admin
    restart: always
    depends_on:
      - mssql
    volumes:
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/admin:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  icons:
    image: bitwarden/icons:2024.4.2
    container_name: bitwarden-icons
    restart: always
    volumes:
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/icons:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
    networks:
      - default
      - public

  notifications:
    image: bitwarden/notifications:2024.4.2
    container_name: bitwarden-notifications
    restart: always
    volumes:
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/notifications:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  events:
    image: bitwarden/events:2024.4.2
    container_name: bitwarden-events
    restart: always
    volumes:
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/events:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  nginx:
    image: bitwarden/nginx:2024.4.2
    container_name: bitwarden-nginx
    restart: always
    depends_on:
      - web
      - admin
      - api
      - identity
    ports:
      - '80:8080'
      - '443:8443'
    volumes:
      - ../nginx:/etc/bitwarden/nginx
      - ../letsencrypt:/etc/letsencrypt
      - ../ssl:/etc/ssl
      - ../logs/nginx:/var/log/nginx
    env_file:
      - ../env/uid.env
    networks:
      - default
      - public

networks:
  default:
    internal: true
  public:
    internal: false

Build Version

2024.4.2

Environment

Self-Hosted

Environment Details

OS: Ubuntu 22.04.4 LTS
Docker
Hardware: VM with 4 CPU cores and 8GiB of memory

Issue Tracking Info

[X] I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

SynVisions commented 5 months ago

I was able to fix the issue by installing the latest docker-compose and deleting the old one (which was not installed via apt). So one bug seems to be that the documentation here says that these are the requirements:

Minimum	Recommended
Engine 19+ and Compose 1.24+	Engine 19+ and Compose 1.24+

As the output in the original issue report shows, I was previously running Docker Engine 24.0.5 and Compose 1.24.1. I think updating the documentation is prudent, but it would also be nice I think to fail the upgrade with a precondition check on the docker-compose version, as this upgrade broke my bitwarden installation.

Ayitaka commented 5 months ago

@djsmith85 please note this is NOT a duplicate of #4059 and should not be closed as such the way #4067 and #4068 were incorrectly closed.

This error is due to many systems not updating to the latest composer version via normal update channels (apt) and so they still default to version: "1", which obviously lacks some of the config options used in later versions (e.g. 3).

Ayitaka commented 5 months ago

I was able to fix the issue by installing the latest docker-compose and deleting the old one (which was not installed via apt).

@SynVisions Would you mind sharing the steps you took to updating your docker-compose, please?

tobiasbaehr commented 5 months ago

Had the same problem. Just delete old docker-compose binary.

The bitwarden.sh script use the old docker-compose by default.

if command -v docker-compose &> /dev/null
then
    dccmd='docker-compose'
else
    dccmd='docker compose'
fi

SynVisions commented 5 months ago

I was able to fix the issue by installing the latest docker-compose and deleting the old one (which was not installed via apt).

@SynVisions Would you mind sharing the steps you took to updating your docker-compose, please?

First I followed the steps here (you may need to use different steps depending upon your OS and distro): https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

This did not fully solve the problem because as @tobiasbaehr calls out, the shell script looks for the existence of docker-compose (which is the old version) first. I don't remember how that was installed originally, but I simply did a sudo rm /usr/local/bin/docker-compose (path where it was in my case) and re-ran ./bitwarden.sh start and it then started using the new version of docker compose.

LufoX11 commented 5 months ago

Ubuntu 18.04 here. Solution for me was a mix of everything.

First I upgraded docker via PPA:

~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
~# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
~# apt update
~# apt-cache policy docker-ce
~# apt install docker-ce -y

Check docker is properly running with systemctl status docker.

Install docker-compose-plugin and remove docker-compose. docker-compose-plugin is the replacement for docker-compose binary so, from now on, ./docker-compose will be invoked as docker compose:

~# apt remove docker-compose
~# apt install docker-compose-plugin

(you may not need this). After running sudo -u bitwarden /opt/bitwarden/bwdata/scripts/updatebw.sh, it failed with
```
error getting credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
```
So I had to delete golang-docker-credential-helpers:

~# apt remove golang-docker-credential-helpers

Then I run the update script again and all came back to normal. As mentioned by @tobiasbaehr the start script will try docker compose (which is the new way to do it) if docker-compose is not available, so it's crucial to remove the old package.

spencer2785 commented 5 months ago

This did not fully solve the problem because as @tobiasbaehr calls out, the shell script looks for the existence of docker-compose (which is the old version) first. I don't remember how that was installed originally, but I simply did a sudo rm /usr/local/bin/docker-compose (path where it was in my case) and re-ran ./bitwarden.sh start and it then started using the new version of docker compose.

I needed to use ./bitwarden.sh rebuild after deleting docker-compose an then start ./bitwarden.sh start. It would be nice if the script checks that the requirement for docker-compose are right before starting with an error.

tferic commented 5 months ago

Ran into the same issue. Deleting docker-compose manually fixed the issue. Thanks. The Bitwarden software really should be changed to default to docker compose nowadays.

Greenderella commented 4 months ago

The 2024.5.0 image of the setup container now contains a fix for this issue. Please make sure you are running the latest version of Docker and update Bitwarden again to apply the fix.

Thank you for everybody's patience.

zatmania commented 4 months ago

Hi, i try to update bitwarden on Linux with 2024.5.0, same problem. I have update my system before to have latest version of docker.

and then with ./bitwarden.sh update, i got this error :

ERROR: The Compose file '/opt/bitwarden/bwdata/docker/docker-compose.yml' is invalid because: Unsupported config option for services: 'nginx' Unsupported config option for networks: 'public' ERROR: The Compose file '/opt/bitwarden/bwdata/docker/docker-compose.yml' is invalid because: Unsupported config option for services: 'notifications' Unsupported config option for networks: 'public'

MatthieuBarthel commented 4 months ago

@zatmania I got a similar issue, you need to remove the old docker-compose command (which docker-compose and remote it), and install the new compose plugin if you don't already have it (docker compose version to check)

zatmania commented 4 months ago

@MatthieuBarthel I have already do this, but no success

Ayitaka commented 4 months ago

@MatthieuBarthel I have already do this, but no success

Check your docker-compose.yml and make sure there are no duplicate services entries (i.e. mssql, web, attachments, api, identity, nginx, etc). Next, if you have one, check your docker-compose.override.yml for the same thing and remove any duplicate service entries.

enoch85 commented 4 months ago

Just if someone comes here using Alpine:

apk del docker-compose
apk add docker-cli-compose

zatmania commented 4 months ago

Well, update works but as said @MatthieuBarthel , i need to remove docker-compose by hand :

rm -f /usr/local/bin/docker-compose

and then run ./bitwarden.sh updateself and then ./bitwarden.sh update

gpi-ct commented 4 months ago

rm -f /usr/local/bin/docker-compose and then run ./bitwarden.sh updateself and then ./bitwarden.sh update

+1 on that working for me as well. Thank goodness for VM snapshots.. I tried it again with chmod 000, in case I needed it back, but that failed as well. Ended up moving it out of the way and that worked: mv /usr/local/bin/docker-compose /usr/local/bin/docker-compose.disabled The update looked weird as it ran, like formating on the screen. Still worked. I had tried adding "version: '3'" to the top of docker-compose.yml but it gets generated automatically. Just noticed I'm still on Ubuntu 18.04 so I need to get that updated as well.

thedxt commented 3 months ago

most of what @LufoX11 is what I needed to do.

I'm on Debian 11.

I needed to reinstall the docker engine and used the official method listed here then I needed to apt remove docker-compose then apt install docker-compose-plugin

after that ./bitwarden.sh update worked

gpi-ct commented 3 months ago

Thank you, can confirm this worked for me as well. I'm on Ubuntu so one extra step was needed: Modified when writing out to /etc/apt/sources.list.d/docker.list, changed 'debian' to 'ubuntu'

raindropworks commented 2 months ago

Well, update works but as said @MatthieuBarthel , i need to remove docker-compose by hand :

rm -f /usr/local/bin/docker-compose

and then run ./bitwarden.sh updateself and then ./bitwarden.sh update

Another +1 on this from Ubuntu 20 LTS. I really didn't like the idea of manually deleting /usr files, but it's what let the update script work properly.

Frankly, this is a huge bug on the part of BitWarden and ought to have it's priority upgraded. How more users haven't swarmed this thread is beyond me. They're not owned by CloudStrike, are they?

raindropworks commented 2 months ago

The 2024.5.0 image of the setup container now contains a fix for this issue. Please make sure you are running the latest version of Docker and update Bitwarden again to apply the fix.

Thank you for everybody's patience.

How is this fixed again? I just upgraded to 2024.06 and had this exact bug.

Almost like updateself needs an update or something?

droopymiller commented 2 months ago

I'm running Ubuntu 20.04. I ran into this issue and was able to resolve the issue after updating bitwarden by just removing docker-compose.

sudo apt remove docker-compose

crazybadger commented 2 months ago

Thanks to this thread, I've upgraded from 2024.4.2 to 2024.7.2. OS ubuntu 20.04.6 LTS, I needed to:

sudo apt-get update && sudo apt-get upgrade

Followed instructions here - https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository - to reinstall/update Docker, including step 1

mv /usr/local/bin/docker-compose /usr/local/bin/docker-compose.disabled ./bitwarden.sh updateself and then ./bitwarden.sh update

It didn't work without moving the old docker-compose files. Thanks everyone!

ryanov commented 2 months ago

most of what @LufoX11 is what I needed to do.

I'm on Debian 11.

I needed to reinstall the docker engine and used the official method listed here then I needed to apt remove docker-compose then apt install docker-compose-plugin

after that ./bitwarden.sh update worked

Easier fix on Debian 11 was to make sure you have backports enabled:

https://backports.debian.org/Instructions/

...and then update docker-compose from the backports repo, like:

# apt install docker-compose/bullseye-backports

Your fix is probably longer-term, but this worked.

zerobulfa commented 2 months ago

heho, for me following was working:

- apt remove docker-compose
- mv /usr/local/bin/docker-compose /usr/local/bin/docker-compose-backup
- apt install docker-compose-plugin
- run ./bitwarden.sh updateself & ./bitwarden.sh update again

what a crap...

tambenet commented 1 month ago

Thanks to this thread, I've upgraded from 2024.4.2 to 2024.7.2. OS ubuntu 20.04.6 LTS, I needed to:

sudo apt-get update && sudo apt-get upgrade

Followed instructions here - https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository - to reinstall/update Docker, including step 1

mv /usr/local/bin/docker-compose /usr/local/bin/docker-compose.disabled ./bitwarden.sh updateself and then ./bitwarden.sh update

It didn't work without moving the old docker-compose files. Thanks everyone!

Thanks to this thread, thank you very much. I've upgraded from 2024.2.5 to 2024.8.0. OS ubuntu 20.04.6 LTS, following your simple and complete instructions. is anyone considering a ubuntu release upgrade? Bye bye

bitwarden / server