DB persistence not working

[Rudis1261]

• Which Faktory package and version?

  github.com/contribsys/faktory v1.6.2
  github.com/contribsys/faktory_worker_go v1.6.0

#docker image
contribsys/faktory latest 8e53c8a72d2e   8 weeks ago    15.4MB

I have tried using a couple of different configurations, but I can't seem to test the persistence locally in docker, I also looked at the Kubernetes examples.

Using this service definition in docker-compose.yaml:

  faktory:
    image: contribsys/faktory
    privileged: true
    environment:
      FAKTORY_PROVIDER: tcp://localhost:7419
    volumes:
      - ./data/redis.rdb:/var/lib/faktory/db/redis.rdb
    ports:
      - 7419:7419
      - 7420:7420

Using this as a volume mount, when I docker-compose down -v, then docker-compose up I seem to be losing all my data. Is /var/lib/faktory/db/redis.rdb still the correct file in the container?

[Rudis1261]

When I bash into the container:

docker run -it contribsys/faktory sh
/ # ls -alhtr /var/lib/faktory/db/
total 8K     
drwxr-xr-x    3 root     root        4.0K Oct  3 19:20 ..
drwxr-xr-x    2 root     root        4.0K Oct  3 19:20 .

/ # ./faktory &
/ # Faktory 1.6.2
Copyright © 2022 Contributed Systems LLC
Licensed under the GNU Affero Public License 3.0
I 2022-12-01T10:11:33.593Z Initializing redis storage at /root/.faktory/db, socket /root/.faktory/db/redis.sock
I 2022-12-01T10:11:33.606Z Web server now listening at localhost:7420
I 2022-12-01T10:11:33.606Z PID 15 listening at localhost:7419, press Ctrl-C to stop

/ # ls -alhtr /var/lib/faktory/db/
total 8K     
drwxr-xr-x    3 root     root        4.0K Oct  3 19:20 ..
drwxr-xr-x    2 root     root        4.0K Oct  3 19:20 .

[mperham]

It puts the file in /var when running in production mode. Sounds like it’s in development mode.

[Rudis1261]

So a valid service definition would be?

  faktory:
    image: contribsys/faktory
    environment:
      FAKTORY_PROVIDER: tcp://localhost:7419
      Environment: production
    volumes:
      - ./data:/var/lib/faktory
    ports:
      - 7419:7419
      - 7420:7420

[Rudis1261]

I have tried production / development, but nothing happens

[mperham]

FAKTORY_ENV=production

[Rudis1261]

Awesome thanks, lemme try that

[Rudis1261]

These don't indicate that specific ENV: https://github.com/contribsys/faktory/wiki/Installation#docker https://hub.docker.com/r/contribsys/faktory/

[Rudis1261]

Hmm still no joy:

  faktory:
    image: contribsys/faktory
    environment:
      FAKTORY_PROVIDER: tcp://:secret@localhost:7419
      FAKTORY_ENV: production
      FAKTORY_PASSWORD: secret
    volumes:
      - ./data:/var/lib/faktory/db
    ports:
      - 7419:7419
      - 7420:7420

Whether or not I touch the file ./data/redis.rdb, it crashes when I start the container:

distributor-faktory-1      | W 2022-12-01T13:16:35.001Z Redis at /var/lib/faktory/db crashed: exit status 1
certificate_distributor-faktory-1      | panic: exit status 1
certificate_distributor-faktory-1      | 
certificate_distributor-faktory-1      | goroutine 7 [running]:
certificate_distributor-faktory-1      | github.com/contribsys/faktory/storage.bootRedis.func2()
certificate_distributor-faktory-1      |        /Users/mperham/src/github.com/contribsys/faktory/storage/redis.go:160 +0xb4
certificate_distributor-faktory-1      | created by github.com/contribsys/faktory/storage.bootRedis
certificate_distributor-faktory-1      |        /Users/mperham/src/github.com/contribsys/faktory/storage/redis.go:156 +0x83c

[Rudis1261]

Running into all sorts of segmentation faults etc. I am thinking it's got something to do with Mac volume mounts specifically.

This "works" as in the app fires up, but then seg-faults when I try and enqueue a job

version: '3.4'

services:
  faktory:
    image: contribsys/faktory
    environment:
      FAKTORY_PROVIDER: tcp://:secret@localhost:7419
      FAKTORY_ENV: production
      FAKTORY_PASSWORD: secret
    volumes:
      - faktory-data:/var/lib/faktory/db
    ports:
      - 7419:7419
      - 7420:7420

volumes:
  faktory-data:

[mperham]

You shouldn't need FAKTORY_PROVIDER: tcp://:secret@localhost:7419 at all. I'm not sure what that's doing there.

I don't recall the Docker volume config but seems like you aren't mapping the host-side. Try:

volumes:
      - .:/var/lib/faktory/db

You should see a faktory.rdb appear in your current directory.

[mperham]

Looks like people have been editing the Docker wiki page and not explaining things fully.

[mperham]

I've updated this page with more explicit detail and better formatting, thanks for the feedback. https://github.com/contribsys/faktory/wiki/Installation#docker

[Rudis1261]

I think I am also battling the Apple M2 (ARM64) differences between inside and outside of the container. So I will just need to ensure that our kubernetes implementation's persistence will work out, and leave it at that for now 👍 Our development can be ephemeral.

[Rudis1261]

Thanks for the assistance 🥇