initial Installation from docker is failing on DB init

saadzer commented 1 year ago

Issue Type

Bug Report

Steps to reproduce

Empty Linux VM
docker-compose freshly installed
Run the following command line as per documentation : wget https://raw.githubusercontent.com/infobyte/faraday/master/docker-compose.yaml

` here is the app section of docker file :

app: image: index.docker.io/faradaysec/faraday restart: always container_name: faraday_app volumes:
- "$HOME/.faraday:/home/faraday/.faraday:rw" environment:
  - PGSQL_USER=postgres
  - PGSQL_PASSWD=
  - PGSQL_HOST=192.168.1.34
  - POSTGRES_PORT=5434
  - PGSQL_PORT=5434
  - PGSQL_DBNAME=faday
  - REDIS_SERVER=redis depends_on:
- redis ports:
- "5985:5985"
- "9000:9000" `
changed the connection on faraday app to a db I have on network in host : 192.168.1.34 on port 5434
then ran : sudo docker-compose up

Expected results

expected to create tables on DB

Debugging tracebacks (current results)

faraday_app | Sat Jun 17 20:43:21 UTC 2023 Using existing server.ini
faraday_app | Sat Jun 17 20:43:24 UTC 2023 Running migrations ...
faraday_app | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
faraday_app | INFO  [alembic.runtime.migration] Will assume transactional DDL.
faraday_app | INFO  [alembic.runtime.migration] Running upgrade  -> e61afb450465, Add custom fields
faraday_app | Migration failed! ERROR [faraday.manage] Migration Error: (psycopg2.errors.UndefinedTable) relation "vulnerability" does not exist
faraday_app | 
faraday_app | [SQL: ALTER TABLE vulnerability ADD COLUMN custom_fields JSONB]
faraday_app | (Background on this error at: http://sqlalche.me/e/13/f405)
faraday_app | (psycopg2.errors.UndefinedTable) relation "vulnerability" does not exist
faraday_app | 
faraday_app | [SQL: ALTER TABLE vulnerability ADD COLUMN custom_fields JSONB]
faraday_app | (Background on this error at: http://sqlalche.me/e/13/f405)

OS

Kali linux

ezk06eer commented 1 year ago

Hi @saadzer ! It seems you have something in the database there, try a fresh install, as we dont know which migrations do you have applied in the db. Best option is start from a scratch. Can you try to apply the migrations to the db from the instance that created the using faraday-manage migrate?

the way you are doing the compose is ok, but it will only work with a freshly installed db, or a version that matches the same database schema that youre trying to compose-up.

if you try to reutilize a previous installation we recommend you to apply the migrations one by one until you get it right.

Cheers!

saadzer commented 1 year ago

Hi @saadzer ! It seems you have something in the database there, try a fresh install, as we dont know which migrations do you have applied in the db. Best option is start from a scratch. Can you try to apply the migrations to the db from the instance that created the using faraday-manage migrate?

the way you are doing the compose is ok, but it will only work with a freshly installed db, or a version that matches the same database schema that youre trying to compose-up.

if you try to reutilize a previous installation we recommend you to apply the migrations one by one until you get it right.

Cheers!

hello , thanks for the feedback.

I can assure you that it is an empty database created only for this project. I've tried redoing everything from the beginning multiple times but am always stuck at this error. I've tried putting the database on another docker , I've also created a database on a new VM. I'm not sure which step am doing wrong tbh , is there a docker image for this project with built-in postgresql database ?

bocajspear1 commented 1 year ago

Had this same issue. I think it was because I had existing data in the mounted directory/volume for /home/faraday/.faraday. Once I changed it to a new directory, things started working.

So in the line:

    volumes:
     - "$HOME/.faraday:/home/faraday/.faraday:rw"

my $HOME/.faraday had existing data from other executions. I changed it to:

    volumes:
     - "$HOME/faraday/home:/home/faraday/.faraday:rw"

I also changed the volume for the postgres to a directory instead, so may have helped too.

ezk06eer commented 1 year ago

As @bocajspear1 mentioned, something quite important is that the volume you are mounting should be fresh, you can try using another directory to mount as volume to avoid such error.

Please let us know if you sorted out your issue.

Cheers!

kadonnelly13 commented 1 year ago

@saadzer - I was having the same problem and solved it by removing an old Docker volume that was being used for the database and clearing out my $HOME/.faraday directory.

How I found my database volume:

List our your current Docker volumes

docker volume ls

My volume had the name of the current directory and appended _db to the end of it so it looked like this faraday-testing_db

Then I removed it

docker volume rm `faraday-testing_db`

saadzer commented 1 year ago

I tried restarting fresh multiple times it doesn't resolve my issue.

So I guess I know what is the reason of my problem, however I don't see a way arround it.

My postgres database is not on the default port 5432 but it's available on the port 5434 the PGSQL_PORT=5434 is not taken into consideration. so what happens I think is the following :

the docker image starts for the first time and create the server.ini and the other initial parameters then it try to connect to 5432 but fails
I stop the docker image and add manually the ":5434" to the "[database]" section of the server.ini file
The restart of the container does now pickup the existing server.ini and connects successfully to the database however it fails to run the migrations because it restart from the point where it goes directly to try adding columns in the database but the database have not been initialized yet so it still have no tables.

so the explanation I've found is thI believe if there's a way to setup the Port directly in the initial setup phase it will not have this issue, so I believe it's a bug with the way the docker image starts when postgresql is on a diffrent port than default one.

I tried to setup the PGSQL_HOST=192.168.1.34:5434 , but this fails with the error : forward host lookup failed: Unknown host

gmartinez95 commented 1 year ago

Hi @saadzer, After you modify the server.ini file try to run faraday-manage initdb

saadzer commented 1 year ago

@gmartinez95 I figured from the doc that the "initdb" didn't run correctly. but isn't it supposed to be executed on the docker start ? also I can't run faraday-manage initdb from the docker container because the container shuts down as soon as it hits the first migration error.

I know I can workarround this issue entirely by creating another VM without docker and run trough the python install instead of the docker, but still wanted to highlight this potential bug with docker image to fix it.

Thanks

gmartinez95 commented 1 year ago

The initdb command runs unless the serve.ini file is already created, because it assumes that the database is already set up. I will add this bug to the team backlog and will fix it as soon we can. Thanks for reporting the problem.

Kong-li commented 10 months ago

The initdb command runs unless the serve.ini file is already created, because it assumes that the database is already set up. I will add this bug to the team backlog and will fix it as soon we can. Thanks for reporting the problem.

did this serve.ini bug fixed？

ezk06eer commented 8 months ago

Hi @Kong-li is not a bug, youre not suposed to start a docker install along with another install in your OS, so if you will use a docker you should check that the local user doesnt have any configuration in server.ini, either or backup and change the filename.

Cheers!

infobyte / faraday