Unable to import Owntracks .rec files

jduar commented 3 hours ago

Hi, I've just started using Dawarich. I was trying to import my data from Owntracks but I haven't been successful.

Dawarich version: 0.18.2

Steps to reproduce the behavior:

Go on the Imports section, choose Owntracks, choose the .rec file.
Click 'Create Import'
Nothing happens and there's no apparent error in the logs either.

Logs

dawarich_app logs from the moment of clicking 'Creating Import':

13:53:07 web.1  | I, [2024-11-30T13:53:07.834537 #144]  INFO -- : {"method":"GET","path":"/api/v1/health","format":"*/*","controller":"Api::V1::HealthController","action":"index","status":200,"allocations":277,"duration":0.84,"view":0.12,"db":0.0}
13:53:09 web.1  | D, [2024-11-30T13:53:09.512503 #144] DEBUG -- :   User Load (0.5ms)  SELECT "users".* FROM "users" WHERE "users"."id" = $1 ORDER BY "users"."id" ASC LIMIT $2  [["id", 1], ["LIMIT", 1]]
13:53:09 web.1  | D, [2024-11-30T13:53:09.513435 #144] DEBUG -- :   ↳ app/controllers/application_controller.rb:11:in `unread_notifications'
13:53:09 web.1  | D, [2024-11-30T13:53:09.516645 #144] DEBUG -- :   TRANSACTION (0.2ms)  BEGIN
13:53:09 web.1  | D, [2024-11-30T13:53:09.517979 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:26:in `block in create'
13:53:09 web.1  | D, [2024-11-30T13:53:09.518825 #144] DEBUG -- :   Import Create (2.3ms)  INSERT INTO "imports" ("name", "user_id", "source", "created_at", "updated_at", "raw_points", "doubles", "processed", "raw_data", "points_count") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING "id"  [["name", "2024-02.rec"], ["user_id", 1], ["source", 1], ["created_at", "2024-11-30 13:53:09.515456"], ["updated_at", "2024-11-30 13:53:09.515456"], ["raw_points", 0], ["doubles", 0], ["processed", 0], ["raw_data", nil], ["points_count", 0]]
13:53:09 web.1  | D, [2024-11-30T13:53:09.520164 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:26:in `block in create'
13:53:09 web.1  | D, [2024-11-30T13:53:09.526136 #144] DEBUG -- :   TRANSACTION (5.5ms)  COMMIT
13:53:09 web.1  | D, [2024-11-30T13:53:09.526828 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:26:in `block in create'
13:53:09 web.1  | D, [2024-11-30T13:53:09.550514 #144] DEBUG -- :   Import Load (0.4ms)  SELECT "imports".* FROM "imports" WHERE "imports"."user_id" = $1 AND "imports"."name" = $2  [["user_id", 1], ["name", "2024-02.rec"]]
13:53:09 web.1  | D, [2024-11-30T13:53:09.551160 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:48:in `rescue in create'
13:53:09 web.1  | D, [2024-11-30T13:53:09.552301 #144] DEBUG -- :   TRANSACTION (0.2ms)  BEGIN
13:53:09 web.1  | D, [2024-11-30T13:53:09.553244 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:48:in `rescue in create'
13:53:09 web.1  | D, [2024-11-30T13:53:09.553873 #144] DEBUG -- :   Point Load (0.4ms)  SELECT "points".* FROM "points" WHERE "points"."import_id" = $1  [["import_id", 19]]
13:53:09 web.1  | D, [2024-11-30T13:53:09.554740 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:48:in `rescue in create'
13:53:09 web.1  | D, [2024-11-30T13:53:09.555766 #144] DEBUG -- :   Import Destroy (0.4ms)  DELETE FROM "imports" WHERE "imports"."id" = $1  [["id", 19]]
13:53:09 web.1  | D, [2024-11-30T13:53:09.556499 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:48:in `rescue in create'
13:53:09 web.1  | D, [2024-11-30T13:53:09.597105 #144] DEBUG -- :   TRANSACTION (40.3ms)  COMMIT
13:53:09 web.1  | D, [2024-11-30T13:53:09.597765 #144] DEBUG -- :   ↳ app/controllers/imports_controller.rb:48:in `rescue in create'
13:53:09 web.1  | I, [2024-11-30T13:53:09.598456 #144]  INFO -- : {"method":"POST","path":"/imports","format":"turbo_stream","controller":"ImportsController","action":"create","status":422,"allocations":7264,"duration":88.86,"view":0.0,"db":50.0,"location":"http://192.168.1.75:3000/imports/new"}

docker-compose.yml

(pretty much like the original, I think I've only moved around the sections)

services:
  dawarich_app:
    image: freikin/dawarich:latest
    container_name: dawarich_app
    volumes:
      - gem_cache:/usr/local/bundle/gems_app
      - public:/var/app/public
      - watched:/var/app/tmp/imports/watched
    networks:
      - dawarich
    ports:
      - 3000:3000
      # - 9394:9394 # Prometheus exporter, uncomment if needed
    stdin_open: true
    tty: true
    entrypoint: dev-entrypoint.sh
    command: ['bin/dev']
    restart: on-failure
    environment:
      RAILS_ENV: development
      REDIS_URL: redis://dawarich_redis:6379/0
      DATABASE_HOST: dawarich_db
      DATABASE_USERNAME: postgres
      DATABASE_PASSWORD: password
      DATABASE_NAME: dawarich_development
      MIN_MINUTES_SPENT_IN_CITY: 60
      APPLICATION_HOST: localhost
      APPLICATION_HOSTS: localhost
      TIME_ZONE: Europe/London
      APPLICATION_PROTOCOL: http
      DISTANCE_UNIT: km
      PHOTON_API_HOST: photon.komoot.io
      PHOTON_API_USE_HTTPS: true
      PROMETHEUS_EXPORTER_ENABLED: false
      PROMETHEUS_EXPORTER_HOST: 0.0.0.0
      PROMETHEUS_EXPORTER_PORT: 9394
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"
    healthcheck:
      test: [ "CMD-SHELL", "wget -qO - http://127.0.0.1:3000/api/v1/health | grep -q '\"status\"\\s*:\\s*\"ok\"'" ]
      interval: 10s
      retries: 30
      start_period: 30s
      timeout: 10s
    depends_on:
      dawarich_db:
        condition: service_healthy
        restart: true
      dawarich_redis:
        condition: service_healthy
        restart: true
    deploy:
      resources:
        limits:
          cpus: '0.50'    # Limit CPU usage to 50% of one core
          memory: '2G'    # Limit memory usage to 2GB

  dawarich_redis:
    image: redis:7.0-alpine
    container_name: dawarich_redis
    command: redis-server
    networks:
      - dawarich
    volumes:
      - shared_data:/var/shared/redis
    restart: always
    healthcheck:
      test: [ "CMD", "redis-cli", "--raw", "incr", "ping" ]
      interval: 10s
      retries: 5
      start_period: 30s
      timeout: 10s

  dawarich_db:
    image: postgres:14.2-alpine
    container_name: dawarich_db
    volumes:
      - db_data:/var/lib/postgresql/data
      - shared_data:/var/shared
    networks:
      - dawarich
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres -d dawarich_development" ]
      interval: 10s
      retries: 5
      start_period: 30s
      timeout: 10s

  dawarich_sidekiq:
    image: freikin/dawarich:latest
    container_name: dawarich_sidekiq
    volumes:
      - gem_cache:/usr/local/bundle/gems_sidekiq
      - public:/var/app/public
      - watched:/var/app/tmp/imports/watched
    networks:
      - dawarich
    stdin_open: true
    tty: true
    entrypoint: dev-entrypoint.sh
    command: ['sidekiq']
    restart: on-failure
    environment:
      RAILS_ENV: development
      REDIS_URL: redis://dawarich_redis:6379/0
      DATABASE_HOST: dawarich_db
      DATABASE_USERNAME: postgres
      DATABASE_PASSWORD: password
      DATABASE_NAME: dawarich_development
      APPLICATION_HOST: localhost
      APPLICATION_HOSTS: localhost
      BACKGROUND_PROCESSING_CONCURRENCY: 10
      APPLICATION_PROTOCOL: http
      DISTANCE_UNIT: km
      PHOTON_API_HOST: photon.komoot.io
      PHOTON_API_USE_HTTPS: true
      PROMETHEUS_EXPORTER_ENABLED: false
      PROMETHEUS_EXPORTER_HOST: dawarich_app
      PROMETHEUS_EXPORTER_PORT: 9394
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"
    healthcheck:
      test: [ "CMD-SHELL", "bundle exec sidekiqmon processes | grep $${HOSTNAME}" ]
      interval: 10s
      retries: 30
      start_period: 30s
      timeout: 10s
    depends_on:
      dawarich_db:
        condition: service_healthy
        restart: true
      dawarich_redis:
        condition: service_healthy
        restart: true
      dawarich_app:
        condition: service_healthy
        restart: true
    deploy:
      resources:
        limits:
          cpus: '0.50'    # Limit CPU usage to 50% of one core
          memory: '2G'    # Limit memory usage to 2GB

networks:
  dawarich:

volumes:
  db_data:
  gem_cache:
  shared_data:
  public:
  watched:

I can share .rec file as well if needed, though its structure seems normal to me and Dawarich has been working fine along with the Owntracks mobile app.

jan315 commented 2 hours ago

The problem is, at least in my case, with "lwt" and "event" lines in .rec file. When I removed those lines, file imported correctly.

jduar commented 1 hour ago

Thanks @jan315 ! After removing those lines I've managed to import every monthly .rec file up until june. Every file after that still fails to upload. I'll try and understand what else might be going on.

Though as a starting point it seems that the importer should only pay attention to lines with "_type": "location". I'll update the issue if I find anything else.

Freika / dawarich

Unable to import Owntracks .rec files #464