Freika / dawarich

Self-hosted alternative to Google Location History (Google Maps Timeline)
https://dawarich.app
GNU Affero General Public License v3.0
2.27k stars 52 forks source link

Dawarich docker - Puma exits - a way to troubleshoot? #365

Closed nitobuendia closed 2 weeks ago

nitobuendia commented 4 weeks ago

Describe the bug

When I ran docker-compose up, dawarich no longer starts.

Fails with css.1 | exited with code 0. Is there a way for me to troubleshoot this?

There's no other logs or errors that I can find. I would be happy to troubleshoot further or provide additional details.

Version

Docker: image: freikin/dawarich:latest which I think it would be 0.15.13.

[freikin/dawarich@sha256:8ef6ac0bd2ebe4b50ceab6e1f6a3fd10cd270e9b92a75a069e5dc3d5f251f790]

To Reproduce

  1. Run docker-compose up -d.
  2. Fails with the below logs. No clear traceback, error or log messages.

(Not on the first run, but subsequent runs).

Expected behavior

Dawarich starts (as it used to).

Other info

Not sure if related - but just before this broke, I added additional sidekiq containers. Then this issue started happening, but it could just have been because I had to stop the container and it never restarted (with or without the new sidekiq). Later, I removed the additional sidekiqs, but it still doesn't start with the same error. I don't see any logs files where I can troubleshoot a bit further beyond what's below. I may have done other changes, but I can't track those downs.

The only action I took was importing a very large Google Takeout, which was running when the container stopped.

Questions

Is there a place where I can find further logs to understand and/or troubleshoot the error? Happy to report back or look for root cause.

Logs

First (failing) start:

Environment: development
<my-postgres-domain> (<my-ip>:<postgres-port>) open
Successfully installed rubygems-update-3.5.7
Installing RubyGems 3.5.7
  Successfully built RubyGem
  Name: bundler
  Version: 2.5.7
  File: bundler-2.5.7.gem
Bundler 2.5.7 installed
RubyGems 3.5.7 installed
Regenerating binstubs
Regenerating plugins

------------------------------------------------------------------------------

RubyGems installed the following executables:
    /usr/local/bin/gem
    /usr/local/bin/bundle
    /usr/local/bin/bundler

RubyGems system software updated
Successfully installed bundler-2.5.9
1 gem installed
Database dawarich already exists, skipping creation...
PostgreSQL is ready. Running database migrations...
[dotenv] Loaded .env.development
D, [2024-11-03T00:21:10.254919 #36] DEBUG -- :   ActiveRecord::SchemaMigration Load (95.7ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
D, [2024-11-03T00:21:10.300785 #36] DEBUG -- :   ActiveRecord::SchemaMigration Load (1.9ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
Running DATA migrations...
[dotenv] Loaded .env.development
D, [2024-11-03T00:21:11.892795 #59] DEBUG -- :    (0.9ms)  SELECT pg_try_advisory_lock(1300285682869233375)
D, [2024-11-03T00:21:11.920080 #59] DEBUG -- :   DataMigrate::DataSchemaMigration Load (25.9ms)  SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC
D, [2024-11-03T00:21:11.939670 #59] DEBUG -- :   ActiveRecord::InternalMetadata Load (14.6ms)  SELECT * FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 ORDER BY "ar_internal_metadata"."key" ASC LIMIT 1  [[nil, "environment"]]
D, [2024-11-03T00:21:11.941142 #59] DEBUG -- :    (0.6ms)  SELECT pg_advisory_unlock(1300285682869233375)
D, [2024-11-03T00:21:11.943232 #59] DEBUG -- :   DataMigrate::DataSchemaMigration Load (0.7ms)  SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC
Running seeds...
[dotenv] Loaded .env.development
D, [2024-11-03T00:21:13.603470 #72] DEBUG -- :   ActiveRecord::SchemaMigration Load (1.8ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
D, [2024-11-03T00:21:14.198345 #72] DEBUG -- :   User Exists? (6.3ms)  SELECT 1 AS one FROM "users" LIMIT $1  [["LIMIT", 1]]
00:21:14 web.1  | started with pid 90
00:21:14 css.1  | started with pid 91
00:21:15 web.1  | => Booting Puma
00:21:15 web.1  | => Rails 7.2.1.2 application starting in development
00:21:15 web.1  | => Run `bin/rails server --help` for more startup options
00:21:15 css.1  | [dotenv] Loaded .env.development
00:21:15 web.1  | [dotenv] Loaded .env.development
00:21:16 web.1  | Puma starting in single mode...
00:21:16 web.1  | * Puma version: 6.4.3 (ruby 3.3.4-p94) ("The Eagle of Durango")
00:21:16 web.1  | *  Min threads: 10
00:21:16 web.1  | *  Max threads: 10
00:21:16 web.1  | *  Environment: development
00:21:16 web.1  | *          PID: 90
00:21:16 web.1  | * Listening on http://0.0.0.0:3000
00:21:16 web.1  | Use Ctrl-C to stop
00:21:17 css.1  | exited with code 0
00:21:17 system | sending SIGTERM to all processes
00:21:17 web.1  | - Gracefully stopping, waiting for requests to finish
00:21:17 web.1  | Exiting
00:21:18 web.1  | terminated by SIGTERM

Environment: development
<my-postgres-domain> (<my-ip>:<postgres-port>) open
Latest version already installed. Done.
Successfully installed bundler-2.5.9
1 gem installed
Database dawarich already exists, skipping creation...
PostgreSQL is ready. Running database migrations...
[dotenv] Loaded .env.development
D, [2024-11-03T00:21:20.992083 #24] DEBUG -- :   ActiveRecord::SchemaMigration Load (1.5ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
D, [2024-11-03T00:21:21.020685 #24] DEBUG -- :   ActiveRecord::SchemaMigration Load (1.6ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
Running DATA migrations...
[dotenv] Loaded .env.development
D, [2024-11-03T00:21:22.400643 #47] DEBUG -- :    (0.7ms)  SELECT pg_try_advisory_lock(1300285682869233375)
D, [2024-11-03T00:21:22.402830 #47] DEBUG -- :   DataMigrate::DataSchemaMigration Load (1.6ms)  SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC
D, [2024-11-03T00:21:22.406360 #47] DEBUG -- :   ActiveRecord::InternalMetadata Load (1.7ms)  SELECT * FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 ORDER BY "ar_internal_metadata"."key" ASC LIMIT 1  [[nil, "environment"]]
D, [2024-11-03T00:21:22.407445 #47] DEBUG -- :    (0.5ms)  SELECT pg_advisory_unlock(1300285682869233375)
D, [2024-11-03T00:21:22.408714 #47] DEBUG -- :   DataMigrate::DataSchemaMigration Load (0.5ms)  SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC
Running seeds...
[dotenv] Loaded .env.development
D, [2024-11-03T00:21:23.879715 #60] DEBUG -- :   ActiveRecord::SchemaMigration Load (1.5ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
D, [2024-11-03T00:21:23.894968 #60] DEBUG -- :   User Exists? (1.0ms)  SELECT 1 AS one FROM "users" LIMIT $1  [["LIMIT", 1]]
00:21:34 web.1  | started with pid 78
00:21:34 css.1  | started with pid 79
00:21:35 web.1  | => Booting Puma
00:21:35 web.1  | => Rails 7.2.1.2 application starting in development
00:21:35 web.1  | => Run `bin/rails server --help` for more startup options
00:21:35 web.1  | [dotenv] Loaded .env.development
00:21:35 css.1  | [dotenv] Loaded .env.development
00:21:35 web.1  | Puma starting in single mode...
00:21:35 web.1  | * Puma version: 6.4.3 (ruby 3.3.4-p94) ("The Eagle of Durango")
00:21:35 web.1  | *  Min threads: 10
00:21:35 web.1  | *  Max threads: 10
00:21:35 web.1  | *  Environment: development
00:21:35 web.1  | *          PID: 78
00:21:35 web.1  | * Listening on http://0.0.0.0:3000
00:21:35 web.1  | Use Ctrl-C to stop
00:21:36 css.1  | exited with code 0
00:21:36 system | sending SIGTERM to all processes
00:21:36 web.1  | - Gracefully stopping, waiting for requests to finish
00:21:36 web.1  | Exiting
00:21:36 web.1  | terminated by SIGTERM
Environment: development

Additional context

docker-compose.yaml

services:
  dawarich_redis:
    image: redis:7.0-alpine
    container_name: dawarich_redis
    command: redis-server
    volumes:
      - ./data/shared:/var/shared/redis

  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    restart: unless-stopped
    command: ['bin/dev']
    entrypoint: dev-entrypoint.sh
    depends_on:
      - dawarich_redis
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"
    environment:
      TIME_ZONE: Asia/Singapore
      RAILS_ENV: development
      DISTANCE_UNIT: "km"
      MIN_MINUTES_SPENT_IN_CITY: 60
    env_file:
      - ./secrets.env
    volumes:
      - ./data/gem_cache:/usr/local/bundle/gems
      - ./data/public:/var/app/public
      - ./data/watched:/var/app/tmp/imports/watched
    labels:
      - "traefik.enable=true"
      # other traefik labels

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    restart: unless-stopped
    command: ['sidekiq']
    entrypoint: dev-entrypoint.sh
    depends_on:
      - dawarich_redis
      - dawarich_app
    env_file:
      - ./secrets.env

secrets.env (anything with <var> has been redacted for obvious reasons:

APPLICATION_PROTOCOL="http"
APPLICATION_HOST="<mydomain.com>:<port>"
APPLICATION_HOSTS="localhost,<mydomain.com>:<port>,<otherdomain.com>:<port>"

REDIS_URL="redis://dawarich_redis:6379/0"

DATABASE_HOST="<dbdomain.com>"
DATABASE_PORT="<port>"
DATABASE_USERNAME="dawarich"
DATABASE_PASSWORD="<password>"
DATABASE_NAME="<database>"

POSTGRES_USER="dawarich"
POSTGRES_PASSWORD="<password>"
REVERSE_GEOCODING_ENABLED="true"

RAILS_MAX_THREADS=10
BACKGROUND_PROCESSING_CONCURRENCY=10
shaman007 commented 3 weeks ago

Having same issue when trying to run it in K8S. More debug info would be really nice! Is there a switch to increase verbosity?

shaman007 commented 2 weeks ago

My setup:

My config so far:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dawarich
  namespace: dawarich
  labels:
    app: dawarich
spec:
  selector:
    matchLabels:
      app: dawarich
  template:
    metadata:
      labels:
        app: dawarich
    spec:
      containers:
        - name: dawarich
          env:
            - name: TIME_ZONE
              value: "Europe/Prague"
            - name: RAILS_ENV
              value: development
            - name: REDIS_URL
              value: redis://:<secret>@redis-master.redis.svc.cluster.local:6379/10
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: redis-password
                  key: redis-password
            - name: DATABASE_HOST
              value: postgres-postgresql.db.svc.cluster.local
            - name: DATABASE_PORT
              value: "5432"
            - name: DATABASE_USERNAME
              value: postgres
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-password
                  key: password
            - name: DATABASE_NAME
              value: dawarich_development
            - name: MIN_MINUTES_SPENT_IN_CITY
              value: "60"
            - name: APPLICATION_HOST
              value: localhost
            - name: APPLICATION_HOSTS
              value: localhost
            - name: APPLICATION_PROTOCOL
              value: http
            - name: DISTANCE_UNIT
              value: km
            - name: PHOTON_API_HOST
              value: photon.komoot.io
            - name: PHOTON_API_USE_HTTPS
              value: "true"
          image: freikin/dawarich:latest
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /usr/local/bundle/gems_app
              name: gem-cache
            - mountPath: /var/app/public
              name: public
            - mountPath: /var/app/tmp/imports/watched
              name: watched
          command:
            - "dev-entrypoint.sh"
          args:
            - "bin/dev"
          resources:
            limits:
              memory: "3Gi"
              cpu: "1500m"
          ports:
          - containerPort: 3000
        - name: dawarich-sidekiq
          env:
            - name: RAILS_ENV
              value: development
            - name: REDIS_URL
              value: redis://:<secret>@redis-master.redis.svc.cluster.local:6379/10
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: redis-password
                  key: redis-password
            - name: DATABASE_HOST
              value: postgres-postgresql.db.svc.cluster.local
            - name: DATABASE_PORT
              value: "5432"
            - name: DATABASE_USERNAME
              value: postgres
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-password
                  key: password
            - name: DATABASE_NAME
              value: dawarich_development
            - name: BACKGROUND_PROCESSING_CONCURRENCY
              value: "10"
            - name: APPLICATION_HOST
              value: localhost
            - name: APPLICATION_HOSTS
              value: localhost
            - name: APPLICATION_PROTOCOL
              value: http
            - name: DISTANCE_UNIT
              value: km
            - name: PHOTON_API_HOST
              value: photon.komoot.io
            - name: PHOTON_API_USE_HTTPS
              value: "true"
          image: freikin/dawarich:latest
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /usr/local/bundle/gems_sidekiq
              name: gem-sidekiq
            - mountPath: /var/app/public
              name: public
            - mountPath: /var/app/tmp/imports/watched
              name: watched
          command:
            - "dev-entrypoint.sh"
          args:
            - "sidekiq"
          resources:
            limits:
              memory: "3Gi"
              cpu: "1500m"
      volumes:
        - name: gem-cache
          persistentVolumeClaim:
            claimName: gem-cache
        - name: gem-sidekiq
          persistentVolumeClaim:
            claimName: gem-sidekiq
        - name: public
          persistentVolumeClaim:
            claimName: public
        - name: watched
          persistentVolumeClaim:
            claimName: watched

My log:

Environment: development
postgres-postgresql.db.svc.cluster.local (10.43.219.108:5432) open
Successfully installed rubygems-update-3.5.7
Installing RubyGems 3.5.7
Successfully built RubyGem
Name: bundler
Version: 2.5.7
File: bundler-2.5.7.gem
Bundler 2.5.7 installed
RubyGems 3.5.7 installed
Regenerating binstubs
Regenerating plugins

------------------------------------------------------------------------------

RubyGems installed the following executables:
/usr/local/bin/gem
/usr/local/bin/bundle
/usr/local/bin/bundler

RubyGems system software updated
Successfully installed bundler-2.5.9
1 gem installed
Database dawarich_development already exists, skipping creation...
PostgreSQL is ready. Running database migrations...
[dotenv] Loaded .env.development
D, [2024-11-08T16:06:44.356351 #36] DEBUG -- : ActiveRecord::SchemaMigration Load (1.0ms) SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
D, [2024-11-08T16:06:44.433081 #36] DEBUG -- : ActiveRecord::SchemaMigration Load (2.7ms) SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
Running DATA migrations...
[dotenv] Loaded .env.development
D, [2024-11-08T16:06:48.304497 #59] DEBUG -- : (0.5ms) SELECT pg_try_advisory_lock(1212213197400985920)
D, [2024-11-08T16:06:48.308679 #59] DEBUG -- : DataMigrate::DataSchemaMigration Load (2.7ms) SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC
D, [2024-11-08T16:06:48.316065 #59] DEBUG -- : ActiveRecord::InternalMetadata Load (2.8ms) SELECT * FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 ORDER BY "ar_internal_metadata"."key" ASC LIMIT 1 [[nil, "environment"]]
D, [2024-11-08T16:06:48.318029 #59] DEBUG -- : (0.6ms) SELECT pg_advisory_unlock(1212213197400985920)
D, [2024-11-08T16:06:48.323404 #59] DEBUG -- : DataMigrate::DataSchemaMigration Load (0.6ms) SELECT "data_migrations"."version" FROM "data_migrations" ORDER BY "data_migrations"."version" ASC
Running seeds...
[dotenv] Loaded .env.development
D, [2024-11-08T16:06:52.321105 #72] DEBUG -- : ActiveRecord::SchemaMigration Load (2.9ms) SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
D, [2024-11-08T16:06:52.400013 #72] DEBUG -- : User Exists? (4.2ms) SELECT 1 AS one FROM "users" LIMIT $1 [["LIMIT", 1]]
16:06:54 web.1 | started with pid 90
16:06:54 css.1 | started with pid 91
16:06:58 css.1 | [dotenv] Loaded .env.development
16:06:59 web.1 | => Booting Puma
16:06:59 web.1 | => Rails 7.2.2 application starting in development
16:06:59 web.1 | => Run `bin/rails server --help` for more startup options
16:06:59 web.1 | [dotenv] Loaded .env.development
16:07:00 web.1 | Puma starting in single mode...
16:07:00 web.1 | * Puma version: 6.4.3 (ruby 3.3.4-p94) ("The Eagle of Durango")
16:07:00 web.1 | * Min threads: 5
16:07:00 web.1 | * Max threads: 5
16:07:00 web.1 | * Environment: development
16:07:00 web.1 | * PID: 90
shaman007 commented 2 weeks ago

WORKAROUND:

Run "bin/rails server -p 3000 -b 0.0.0.0" instead of "bin/dev" in 'args'.

nitobuendia commented 2 weeks ago

It worked for me too. Thank you so much, @shaman007

Freika commented 2 weeks ago

@shaman007 thanks for the fix, the problem was fixed in 0.16.4, so you can either return bin/dev in your config, or continue using your version