search

bmarsh9 / gapps

Security compliance platform - SOC2, CMMC, ASVS, ISO27001, HIPAA, NIST CSF, NIST 800-53, CSC CIS 18, PCI DSS, SSF tracking. https://gapps.darkbanner.com
Other
438 stars 99 forks source link

DB issue, no persistance #87

Closed avicoder closed 1 year ago

avicoder commented 1 year ago

Hi I cloned the latest version and start with setup with docker-compose.

App is working properly, however upon restarting I am losing all the saved progress.

Logs:

postgres is up-to-date
gapps-worker is up-to-date
Starting gapps ... done
Attaching to postgres, gapps-worker, gapps
postgres        | The files belonging to this database system will be owned by user "postgres".
postgres        | This user must also own the server process.
postgres        |
postgres        | The database cluster will be initialized with locale "en_US.utf8".
postgres        | The default database encoding has accordingly been set to "UTF8".
postgres        | The default text search configuration will be set to "english".
postgres        |
postgres        | Data page checksums are disabled.
postgres        |
postgres        | fixing permissions on existing directory /data/postgres ... ok
postgres        | creating subdirectories ... ok
postgres        | selecting dynamic shared memory implementation ... posix
postgres        | selecting default max_connections ... 100
postgres        | selecting default shared_buffers ... 128MB
postgres        | selecting default time zone ... Etc/UTC
postgres        | creating configuration files ... ok
postgres        | running bootstrap script ... ok
postgres        | performing post-bootstrap initialization ... ok
postgres        | syncing data to disk ... ok
postgres        |
postgres        |
postgres        | Success. You can now start the database server using:
postgres        |
postgres        |     pg_ctl -D /data/postgres -l logfile start
postgres        |
postgres        | initdb: warning: enabling "trust" authentication for local connections
postgres        | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
postgres        | waiting for server to start....2023-09-02 05:08:41.871 UTC [49] LOG:  starting PostgreSQL 15.4 (Debian 15.4-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres        | 2023-09-02 05:08:41.874 UTC [49] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres        | 2023-09-02 05:08:41.884 UTC [52] LOG:  database system was shut down at 2023-09-02 05:08:39 UTC
postgres        | 2023-09-02 05:08:41.895 UTC [49] LOG:  database system is ready to accept connections
postgres        |  done
postgres        | server started
postgres        | CREATE DATABASE
postgres        |
postgres        |
postgres        | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres        |
postgres        | waiting for server to shut down....2023-09-02 05:08:42.131 UTC [49] LOG:  received fast shutdown request
postgres        | 2023-09-02 05:08:42.134 UTC [49] LOG:  aborting any active transactions
postgres        | 2023-09-02 05:08:42.137 UTC [49] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
postgres        | 2023-09-02 05:08:42.139 UTC [50] LOG:  shutting down
postgres        | 2023-09-02 05:08:42.142 UTC [50] LOG:  checkpoint starting: shutdown immediate
postgres        | 2023-09-02 05:08:42.740 UTC [50] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.018 s, sync=0.572 s, total=0.601 s; sync files=301, longest=0.009 s, average=0.002 s; distance=4223 kB, estimate=4223 kB
postgres        | 2023-09-02 05:08:42.749 UTC [49] LOG:  database system is shut down
postgres        |  done
postgres        | server stopped
postgres        |
postgres        | PostgreSQL init process complete; ready for start up.
postgres        |
postgres        | 2023-09-02 05:08:42.869 UTC [1] LOG:  starting PostgreSQL 15.4 (Debian 15.4-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres        | 2023-09-02 05:08:42.869 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres        | 2023-09-02 05:08:42.869 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres        | 2023-09-02 05:08:42.875 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres        | 2023-09-02 05:08:42.883 UTC [65] LOG:  database system was shut down at 2023-09-02 05:08:42 UTC
postgres        | 2023-09-02 05:08:42.890 UTC [1] LOG:  database system is ready to accept connections
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [ERROR] Traceback while querying db model: (psycopg2.OperationalError) could not connect to server: Connection refused
gapps-worker    |       Is the server running on host "postgres" (192.168.112.2) and accepting
gapps-worker    |       TCP/IP connections on port 5432?
gapps-worker    |
gapps-worker    | (Background on this error at: http://sqlalche.me/e/13/e3q8)
gapps-worker    | [ERROR] Error while querying database models
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [WARNING] Unable to query the database models. They need to be created
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [WARNING] Unable to query the database models. They need to be created
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [WARNING] Unable to query the database models. They need to be created
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [WARNING] Unable to query the database models. They need to be created
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [WARNING] Unable to query the database models. They need to be created
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [WARNING] Unable to query the database models. They need to be created
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps-worker    | [INFO] Running as a worker. Trying to start...
gapps-worker    | [INFO] Checking if database models require creation
gapps-worker    | [WARNING] Unable to query the database models. They need to be created
gapps-worker    | [ERROR] Unable to query the database models. Worker is exiting...
gapps           | [INFO] Checking if we can connect to the database server: postgresql://db1:db1@postgres/db1
gapps           | [INFO] Successfully connected to the database server
gapps           | [INFO] Checking if database models require creation
gapps           | [WARNING] Unable to query the database models. They need to be created
gapps           | [INFO] Setting up the database models

docker-compose.yml

version: '3'
services:
  gapps:
    container_name: gapps
    image: bmarsh13/gapps:3.4.8
    depends_on:
      - postgres
    networks:
      - db_nw
      - web_nw
    ports:
      - "5000:5000"
    restart: unless-stopped
    environment:
      - SQLALCHEMY_DATABASE_URI=postgresql://${POSTGRES_USER:-db1}:${POSTGRES_PASSWORD:-db1}@${POSTGRES_HOST:-postgres}/${POSTGRES_DB:-db1}
      - DEFAULT_EMAIL=${DEFAULT_EMAIL:-admin@example.com}
      - DEFAULT_PASSWORD=${DEFAULT_PASSWORD:-admin}
      - RESET_DB=${RESET_DB:-no}
      - DOC_LINK=https://github.com/bmarsh9/gapps
      - VERSION=${VERSION:-3.4.8}
      - APP_NAME=Gapps
      - POSTGRES_USER=${POSTGRES_USER:-db1}
      - POSTGRES_HOST=${POSTGRES_HOST:-postgres}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-db1}
      - POSTGRES_DB=${POSTGRES_DB:-db1}
      - GUNICORN_WORKERS=2
#      - RESET_DB=yes
  gapps-worker:
    container_name: gapps-worker
    image: bmarsh13/gapps:3.4.8
    depends_on:
      - postgres
    networks:
      - db_nw
      - web_nw
    ports:
      - "5000:5000"
    restart: unless-stopped
    environment:
      - SQLALCHEMY_DATABASE_URI=postgresql://${POSTGRES_USER:-db1}:${POSTGRES_PASSWORD:-db1}@${POSTGRES_HOST:-postgres}/${POSTGRES_DB:-db1}
      - VERSION=${VERSION:-3.4.8}
      - APP_NAME=Gapps
      - AS_WORKER=yes
      - POSTGRES_USER=${POSTGRES_USER:-db1}
      - POSTGRES_HOST=${POSTGRES_HOST:-postgres}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-db1}
      - POSTGRES_DB=${POSTGRES_DB:-db1}
      - WORKER_CONCURRENCY=${WORKER_CONCURRENCY:-1}
      - WORKER_LOG_LEVEL=DEBUG
  postgres:
    container_name: postgres
    image: postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-db1}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-db1}
      POSTGRES_DB: ${POSTGRES_DB:-db1}
      PGDATA: /data/postgres
#    volumes:
#       - postgres:/data/postgres
#    ports:
#     - "5432:5432"
    networks:
      - db_nw
    restart: unless-stopped

networks:
  db_nw:
    driver: bridge
  web_nw:
    driver: bridge
volumes:
  dbdata:
bmarsh9 commented 1 year ago

@avicoder hmm when you say restart, what commands are you running?

With the current docker-compose file, the postgres container will not persist data if it is recreated. You would need to define a volume. See here: https://stackoverflow.com/questions/41637505/how-to-persist-data-in-a-dockerized-postgres-database-using-volumes

avicoder commented 1 year ago

found the issue, uncommenting and adding a period helped resolve this issue.

    volumes:
       - ./postgres:/data/postgres
    ports:
      - "5432:5432"