Startup failure: dendrite_naffka does not exist

TobiTenno commented 3 years ago

Background information

Dendrite version or git SHA: matrixdotorg/dendrite monolith f7918024c229
Monolith or Polylith?: Monolith
SQLite3 or Postgres?: Postgres
Running in Docker?: ✔️
go version: go1.15.2 linux/amd64

Description

Failing to start dendrite monolith, naffka database isn't created

Steps to reproduce

clone dendrite github
build generate-keys script
generate keys
copy keys to ./config
copy ./build/docker/config/dendrite-config.yaml to ./config/dendrite.yaml
change 'matrix.org' to my hostname
enable use_naffka
docker-compose -f docker-compose.deps.yml up -d postgres
check logs, make sure vacuuming has started
docker-compose -f docker-compose.monolith.yml up
see failure in logs

Expectation: server should start

Logs

time="2020-10-17T01:19:59.471899009Z" level=info msg="Dendrite version 0.1.0" func="NewBaseDendrite\n\t" file=" [github.com/matrix-org/dendrite@/internal/setup/base.go:102]"
time="2020-10-17T01:19:59.477280511Z" level=panic msg="Failed to setup naffka database" func="setupNaffka\n\t" file=" [github.com/matrix-org/dendrite@/internal/setup/base.go:357]" error="pq: database \"dendrite_naffka\" does not exist"
panic: (*logrus.Entry) (0x10bb380,0xc0001f4150)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0000f4230, 0xc0000321e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x339
github.com/sirupsen/logrus.(*Entry).Log(0xc0001f40e0, 0x0, 0xc0001cd7d8, 0x1, 0x1)
    github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xeb
github.com/sirupsen/logrus.(*Entry).Panic(0xc0001f40e0, 0xc0001cd7d8, 0x1, 0x1)
    github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/matrix-org/dendrite/internal/setup.setupNaffka(0xc000192700, 0xc0001b6ed0, 0x10, 0x7f86ac4c0ae8, 0xc0001ce6a0)
    github.com/matrix-org/dendrite@/internal/setup/base.go:357 +0xeb
github.com/matrix-org/dendrite/internal/setup.NewBaseDendrite(0xc000192700, 0x10cd819, 0x8, 0x10ca000, 0x5)
    github.com/matrix-org/dendrite@/internal/setup/base.go:112 +0x39e
main.main()
    github.com/matrix-org/dendrite@/cmd/dendrite-monolith-server/main.go:70 +0x151

neilalexander commented 3 years ago

For Docker, you will want to start from ./build/docker/config/dendrite-config.yaml rather than ./dendrite-config.yaml.

TobiTenno commented 3 years ago

Sorry I wasn't more specific on that, I didn't notice another. I started from the docker one (per the docker docs)

I've corrected my repro steps

neilalexander commented 3 years ago

I'm a bit puzzled - the default Docker config states:

connection_string: postgresql://dendrite:itsasecret@postgres/dendrite_naffka?sslmode=disable

... and the create-db.sh script for PostgreSQL as used by docker-compose.deps.yml states:

for db in ... naffka; do
    createdb -U dendrite -O dendrite dendrite_$db
done

So the template setup for Docker definitely creates dendrite_naffka.

Are you sure you haven't tweaked either of these two things?

TobiTenno commented 3 years ago

yup, i even redid my steps ensuring no extra alterations. do i possibly need to wait longer after startup of the db for it to finish?

I am seeing the following in the database startup:

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/20-create_db.sh
/usr/local/bin/docker-entrypoint.sh: line 159: /docker-entrypoint-initdb.d/20-create_db.sh: Is a directory

PostgreSQL Database directory appears to contain a database; Skipping initialization

After my last build I did delete the volume that was created to make sure there were no dangling artifacts from previous containers, but something seems to be sticking around.

TobiTenno commented 3 years ago

Just fully recloned everything and set up following the repro steps I listed, same result.

I even tried redoing it where I ran the whole deps startup.

kegsay commented 3 years ago

@TobiTenno please re-open this if you can reproduce this on the latest version. Please give repro steps again as I can't repro this from the steps outlined here.

TobiTenno commented 3 years ago

@Kegsay I can't re-open, but here's the repro steps again:

1) go get github.com/matrix-org/dendrite/cmd/generate-keys 1) go build github.com/matrix-org/dendrite/cmd/generate-keys 1) a. create docker b. create docker/config c. cd docker/config d. go run github.com/matrix-org/dendrite/cmd/generate-keys --private-key=matrix_key.pem --tls-cert=server.crt --tls-key=server.key

1) clone dendrite 1) copy dendrite-config.yaml to docker/config/dendrite.yaml 1) copy docker-compose.deps.yml to docker/docker-compose.deps.yml 1) copy docker-compose.monolith.yml to docker/docker-compose.monolith.yml 1) sed 's/use_naffka: false/use_naffka: true/g' dendrite.yaml > dendrite.naffka.yaml 1) mv dendrite.naffka.yaml dendrite.yaml 1) a. cd docker b. run docker-compose -f docker-compose.deps.yml up -d postgres applicable logs:

  postgres_1   | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/20-create_db.sh
  postgres_1   | /usr/local/bin/docker-entrypoint.sh: line 160: /docker-entrypoint-initdb.d/20-create_db.sh: Is a directory

1) run docker-compose -f docker-compose.monolith.yml up -d applicable logs:

  monolith_1  | time="2021-01-06T22:28:53.904980100Z" level=info msg="Dendrite version 0.3.4" func="NewBaseDendrite\n\t" file=" [github.com/matrix-org/dendrite/setup/base.go:102]"
  monolith_1  | time="2021-01-06T22:28:53.907228800Z" level=panic msg="failed to connect to accounts db" func="CreateAccountsDB\n\t" file=" [github.com/matrix-org/dendrite/setup/base.go:243]" error="pq: database \"dendrite_account\" does not exist"

sh scripts that I'm using to test: gist

matrix-org / dendrite