[Bug] Can't connect to database (with reverse proxy?)

[Nyarumi]

RomM version 3.3.0

Describe the bug romm appears to be unable to connect to the database.

To Reproduce Steps to reproduce the behavior:

0. Use Nginx Proxy Manager
1. Fill in docker-compose.yml
2. Run RomM
3. Create user and log in
4. Restart RomM
5. Receive error Unable to login: Internal Server Error in the web UI Check docker logs and find errors Can't connect to server on 'romm-db' (115) and 403: User not found (You'll also find the Something went horribly wrong with our database error there too)

Additional context I use Nginx Proxy Manager on Unraid to handle reverse-proxying various docker containers and I haven't had any issues until now when trying to run RomM. I got RomM to work seemingly correctly on the first boot when I was prompted to create an account. I then discovered that I had binded my ROMs folder incorrectly so I restarted it to fix that and then I ran into the errors described above. I've tried with both the standard mariadb and the linuxserver images for the database and both fail the same way. I also tried giving both RomM and the database container their own private network to connect to each other with but that didn't work either. The database seems to be starting okay so there's some kind of issue preventing RomM from communicating with it. I checked both containers while they were running and they both report being connected to the same network, so I don't really know what's going on.

Any ideas? Thanks!

[gantoine]

can you post your docker-compose, while hiding passwords and keys with *****?

[Nyarumi]

Here you go. This is the version when I tried to give romm and the database a private network to work with.

#  mysql_data:
#  romm_resources:
#  romm_redis_data:

services:
  romm:
    image: rommapp/romm:latest
    container_name: romm
    restart: unless-stopped
    environment:
      - DB_HOST=romm-db
      - DB_NAME=romm # Should match MYSQL_DATABASE in mariadb
      - DB_USER=romm-serv # Should match MYSQL_USER in mariadb
      - DB_PASSWD=********** # Should match MYSQL_PASSWORD in mariadb
      - ROMM_AUTH_SECRET_KEY=********** # Generate a key with `openssl rand -hex 32`
      - IGDB_CLIENT_ID=********** # Generate an ID and SECRET in IGDB
      - IGDB_CLIENT_SECRET=********** # https://api-docs.igdb.com/#account-creation
      - MOBYGAMES_API_KEY=********** # https://www.mobygames.com/info/api/
      - STEAMGRIDDB_API_KEY=********** # https://github.com/rommapp/romm/wiki/Generate-API-Keys#steamgriddb
    volumes:
      - /mnt/cache/****:/romm/resources # Resources fetched from IGDB (covers, screenshots, etc.)
      - /mnt/cache/****:/redis-data # Cached data for background tasks
      - /mnt/user/****:/romm/library/roms # Your game library
      - /mnt/cache/****:/romm/assets # Uploaded saves, states, etc.
      - /mnt/cache/****:/romm/config # Path where config.yml is stored
    #ports:
    #  - 80:8080
    depends_on:
      - romm-db
    networks:
      - default
      - romm-db-net

  romm-db:
    image: linuxserver/mariadb:latest # if you experience issues, try: linuxserver/mariadb:latest
    container_name: romm-db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=********** # Use a unique, secure password
      - MYSQL_DATABASE=romm
      - MYSQL_USER=romm-serv
      - MYSQL_PASSWORD=**********
    volumes:
      - /mnt/cache/****:/var/lib/mysql
    networks:
      - romm-db-net

networks:
  default:
    external: true
    name: himitsu
  romm-db-net:
    external: false

[gantoine]

huh your networks look good, that was my first guess. can you verify MYSQL_PASSWORD matches DB_PASSWD? and also post the container logs on startup?

[Nyarumi]

MYSQL_PASSWORD does in fact match DB_PASSWD. I started up the docker containers after taking a little break and now I don't get the something went horribly wrong with our database error but instead a new error that says Exception in ASGI application

These are the logs from both romm and the database after I did these actions: started up both containers, went to romm's login page, attempted to log in.

romm-logs.txt db-logs.txt

[gantoine]

Ok that's better!

fastapi.exceptions.HTTPException: 403: User not found

go ahead and clear your browser cache and cookies.

[Nyarumi]

Well, the issue came back haha. I have no idea what I did different that time, but I'm back to the original error. I did get the logs for that this time so you can take a look at them. It all seems to be because romm can't connect with the database, but I'm honestly not sure why that is. I've triple checked the compose file and everything seems like it should work. I even changed the passwords to very basic ones to make sure I wasn't somehow messing that up. I'm pretty stumped here.

(logs after deleting all volumes, creating an admin user, restarting romm, then trying to log in again) romm-log-nodb.txt

[gantoine]

You sure romm doens't work? The logs look good now, everything seems to be running correctly.

[Nyarumi]

The logs look good? I thought the error about not being able to connect to the database wasn’t a good thing. Romm works on first run only (when you set up the inital account) and I can get it to scan my library and all that. After any restart of the docker containers it fails. When I try to log in again, I get an internal server error and in the logs it says the user does not exist. I’m not sure if it’s a permissions error, a network error, or something else completely, but for some reason I can’t get romm to work with its database on my machine. Romm itself creates its own files just fine, but the database is never written (because it can’t be connected to?) to the disk.

[gantoine]

The reason I ask is cause the end of the logs DB migrations run, which means your romm container can connect to your mariadb (at least sometimes).

[Nyarumi]

I see… I wonder if there’s a way I can monitor the docker network to see exactly what’s happening (or not happening) through it.

[adamantike]

This could be an issue with the application trying to start before the database is ready to accept connections. Can you test the following changes in your Docker Compose file (and, of course, stop all containers, and run again):

  romm:
    # Here, we change the dependency to wait for the database to be ready
    depends_on:
      romm-db:
        condition: service_healthy
        restart: true

  romm-db:
    # Add a healthcheck to validate the database is ready to receive connections
    healthcheck:
      test: [ "CMD", "healthcheck.sh", "--connect", "--innodb_initialized" ]
      start_period: 30s
      start_interval: 10s
      interval: 10s
      timeout: 5s
      retries: 3

Related context:

• https://docs.docker.com/compose/startup-order/
• https://mariadb.org/mariadb-server-docker-official-images-healthcheck-without-mysqladmin/

[Nyarumi]

I actually looked into that and couldn't get it to work myself. I tried using your suggestions and unfortunately got the same results. I cannot use start_interval because the docker version I'm running is 24 (it needs 25), so I suppose that might be an issue? I've attached the database logs when it starts up, it looks like it's doing something but I'm not sure what. I'm not sure if it fails the healthcheck or if it somehow passes but doesn't report that information correctly. Sorry for all the trouble, this is clearly an issue with my own setup at this point and not one with romm itself. whealthcheck.txt

[NovaCyntax]

I've deployed this container about 5 or 6 times the past year and every single time it has been nothing short of a headache. Following the exact example given in the example compose file, it still refuses to connect to the database no matter what I do. This is a neat project, but the deployment is unnecessarily complicated and leads to issues.

[adamantike]

@Nyarumi, can you test by adding the healthcheck/depends_on block from the previous comment, even if you need to remove the start_interval to make it compatible with your installation?

Please post both your docker-compose.yml, with those latest changes, and Docker Compose logs to keep investigating.

[adamantike]

@NovaCyntax, also feel free to send your docker-compose.yml file (with sensitive information redacted) and Docker Compose logs.

[Nyarumi]

If I change the hostname romm is trying to connect to, I get a different error that seems to imply that the hostname for the database is fine, romm just... can't connect to it for some reason. Wrong hostname: Unknown server host 'whatever-host' Correct hostname: Can't connect to server on 'romm-db'

I can also have them ping each other through the docker network so it's not like they CAN'T connect. They obviously see each other, but something is preventing the actual connection to the database.

@adamantike The containers never start with the healthcheck blocks as the database never reports it is healthy. It SEEMS to be healthy to me though, as its log has no errors and reports that it's ready:

Installing MariaDB/MySQL system tables in '/config/databases' ...
OK

To start mariadbd at boot time you have to... [I removed the generic startup info to save space] 

Database Setup Completed
[custom-init] No custom files found, skipping...
240816 04:21:07 mysqld_safe Logging to '/config/databases/bd0c258b3240.err'.
240816 04:21:07 mysqld_safe Starting mariadbd daemon with databases from /config/databases
Connection to localhost (127.0.0.1) 3306 port [tcp/mysql] succeeded!
[ls.io-init] done.

[Gandalf-the-Blue]

I'm having the same issue - adding the depends on and healthcheck blocks result in the error -Error dependency failed to start: container romm_db is unhealthy