search

jonaswinkler / paperless-ng

A supercharged version of paperless: scan, index and archive all your physical documents
https://paperless-ng.readthedocs.io/en/latest/
GNU General Public License v3.0
5.37k stars 355 forks source link

[BUG] Database error after updating to 1.1.1 and 1.1.2 #541

Closed Sblop closed 3 years ago

Sblop commented 3 years ago

Describe the bug After moving forward from 1.1.0 (running dev branch via docker) paperless ng wasnt loading any documents. It didnt show any document count on the dashboard and document list was empty. If I went into admin and documents it would crash. After running latest for a few days it worked perfectly (it was 1.1.0) When it updated to 1.1.1 again same problem. Tried dev branch again with 1.1.2 and same error.

I have tried following commands: python3 ./manage.py migrate python3 ./manage.py migrate documents python3 ./manage.py migrate documents (all numbers from documentation)

Paperless doenst show any error message, however logs from database shows log as below:

2021-02-13 23:47:53.139 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:53.142 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:53.142 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:53.146 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:53.146 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:53.150 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:53.150 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:53.153 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:53.153 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:56.953 UTC [376] ERROR:  column documents_document.archive_filename does not exist at character 591

2021-02-13 23:47:56.953 UTC [376] STATEMENT:  SELECT COUNT(*) FROM (SELECT DISTINCT "documents_document"."id" AS Col1, "documents_document"."correspondent_id" AS Col2, "documents_document"."title" AS Col3, "documents_document"."document_type_id" AS Col4, "documents_document"."content" AS Col5, "documents_document"."mime_type" AS Col6, "documents_document"."checksum" AS Col7, "documents_document"."archive_checksum" AS Col8, "documents_document"."created" AS Col9, "documents_document"."modified" AS Col10, "documents_document"."storage_type" AS Col11, "documents_document"."added" AS Col12, "documents_document"."filename" AS Col13, "documents_document"."archive_filename" AS Col14, "documents_document"."archive_serial_number" AS Col15 FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") INNER JOIN "documents_tag" ON ("documents_document_tags"."tag_id" = "documents_tag"."id") WHERE "documents_tag"."is_inbox_tag") subquery

2021-02-13 23:47:57.041 UTC [377] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.041 UTC [377] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") INNER JOIN "documents_tag" ON ("documents_document_tags"."tag_id" = "documents_tag"."id") WHERE "documents_tag"."is_inbox_tag" ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.197 UTC [378] ERROR:  column documents_document.archive_filename does not exist at character 591

2021-02-13 23:47:57.197 UTC [378] STATEMENT:  SELECT COUNT(*) FROM (SELECT DISTINCT "documents_document"."id" AS Col1, "documents_document"."correspondent_id" AS Col2, "documents_document"."title" AS Col3, "documents_document"."document_type_id" AS Col4, "documents_document"."content" AS Col5, "documents_document"."mime_type" AS Col6, "documents_document"."checksum" AS Col7, "documents_document"."archive_checksum" AS Col8, "documents_document"."created" AS Col9, "documents_document"."modified" AS Col10, "documents_document"."storage_type" AS Col11, "documents_document"."added" AS Col12, "documents_document"."filename" AS Col13, "documents_document"."archive_filename" AS Col14, "documents_document"."archive_serial_number" AS Col15 FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 1) subquery

2021-02-13 23:47:57.263 UTC [379] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.263 UTC [379] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 1 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.267 UTC [379] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.267 UTC [379] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 1 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.271 UTC [379] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.271 UTC [379] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 1 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.274 UTC [379] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.274 UTC [379] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 1 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.277 UTC [379] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.277 UTC [379] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 1 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.412 UTC [380] ERROR:  column documents_document.archive_filename does not exist at character 591

2021-02-13 23:47:57.412 UTC [380] STATEMENT:  SELECT COUNT(*) FROM (SELECT DISTINCT "documents_document"."id" AS Col1, "documents_document"."correspondent_id" AS Col2, "documents_document"."title" AS Col3, "documents_document"."document_type_id" AS Col4, "documents_document"."content" AS Col5, "documents_document"."mime_type" AS Col6, "documents_document"."checksum" AS Col7, "documents_document"."archive_checksum" AS Col8, "documents_document"."created" AS Col9, "documents_document"."modified" AS Col10, "documents_document"."storage_type" AS Col11, "documents_document"."added" AS Col12, "documents_document"."filename" AS Col13, "documents_document"."archive_filename" AS Col14, "documents_document"."archive_serial_number" AS Col15 FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 4) subquery

2021-02-13 23:47:57.476 UTC [381] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.476 UTC [381] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 4 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.480 UTC [381] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.480 UTC [381] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 4 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.484 UTC [381] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.484 UTC [381] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 4 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.489 UTC [381] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.489 UTC [381] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 4 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.493 UTC [381] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.493 UTC [381] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 4 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.618 UTC [380] ERROR:  column documents_document.archive_filename does not exist at character 591

2021-02-13 23:47:57.618 UTC [380] STATEMENT:  SELECT COUNT(*) FROM (SELECT DISTINCT "documents_document"."id" AS Col1, "documents_document"."correspondent_id" AS Col2, "documents_document"."title" AS Col3, "documents_document"."document_type_id" AS Col4, "documents_document"."content" AS Col5, "documents_document"."mime_type" AS Col6, "documents_document"."checksum" AS Col7, "documents_document"."archive_checksum" AS Col8, "documents_document"."created" AS Col9, "documents_document"."modified" AS Col10, "documents_document"."storage_type" AS Col11, "documents_document"."added" AS Col12, "documents_document"."filename" AS Col13, "documents_document"."archive_filename" AS Col14, "documents_document"."archive_serial_number" AS Col15 FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3) subquery

2021-02-13 23:47:57.666 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.666 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.670 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.670 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.673 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.673 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.677 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.677 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

2021-02-13 23:47:57.680 UTC [372] ERROR:  column documents_document.archive_filename does not exist at character 461

2021-02-13 23:47:57.680 UTC [372] STATEMENT:  SELECT DISTINCT "documents_document"."id", "documents_document"."correspondent_id", "documents_document"."title", "documents_document"."document_type_id", "documents_document"."content", "documents_document"."mime_type", "documents_document"."checksum", "documents_document"."archive_checksum", "documents_document"."created", "documents_document"."modified", "documents_document"."storage_type", "documents_document"."added", "documents_document"."filename", "documents_document"."archive_filename", "documents_document"."archive_serial_number" FROM "documents_document" INNER JOIN "documents_document_tags" ON ("documents_document"."id" = "documents_document_tags"."document_id") WHERE "documents_document_tags"."tag_id" = 3 ORDER BY "documents_document"."created" DESC LIMIT 21

Below my docker compose file:

version: '3.7' services:


#### PAPER MERGE

  paperless-broker:
    image: redis:6.0
    container_name: paperless-broker   
    restart: always
    networks:
      docker_frontend:

  paperless-db:
    image: postgres:13
    container_name: paperless-db   
    restart: always
    volumes:
      - /share/Document_Management/3.DATA/db:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: paperless
      POSTGRES_USER: paperless
      POSTGRES_PASSWORD: paperless
    networks:
      docker_frontend:

  paperless-webserver:
#    image: jonaswinkler/paperless-ng:1.1.0
    image: jonaswinkler/paperless-ng:dev
    container_name: paperless-webserver   
    restart: always
    depends_on:
      - paperless-broker
      - paperless-db
      - paperless-gotenberg
      - paperless-tika
    ports:
      - 8050:8000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - /share/Document_Management/3.DATA/data:/usr/src/paperless/data
      - /share/Document_Management/1.DOC_STOREAGE:/usr/src/paperless/media
      - /share/Document_Management/2.BACKUP:/usr/src/paperless/export
      - /share/Document_Management/0.IMPORT/:/usr/src/paperless/consume
      - /share/Document_Management/3.DATA/data/tmp:/tmp
#    env_file: docker-compose.env
    environment:
      PAPERLESS_REDIS: "redis://paperless-broker:6379"
#      PAPERLESS_TIKA_ENABLED: "0"
      PAPERLESS_TIKA_ENABLED: "1"
      PAPERLESS_TIKA_GOTENBERG_ENDPOINT: "http://paperless-gotenberg:3000"
      PAPERLESS_TIKA_ENDPOINT: "http://paperless-tika:9998"
      PAPERLESS_DBHOST: "paperless-db"
#      PAPERLESS_FILENAME_FORMAT: "{correspondent}/{document_type}/{created_year}-{created_month}/{title},{tag_list}"
      PAPERLESS_FILENAME_FORMAT: "{correspondent}/{document_type}/{created_year}-{created_month}-{created_day} - {title}, {tag_list}"
      PAPERLESS_SECRET_KEY: "XXXXX"
#      PAPERLESS_AUTO_LOGIN_USERNAME: "root"
#      PAPERLESS_OCR_LANGUAGE: "dan+eng"
      PAPERLESS_OCR_LANGUAGE: "eng"
      PAPERLESS_TIME_ZONE: "Europe/Copenhagen"
    networks:
      docker_frontend:

  paperless-gotenberg:
    image: thecodingmachine/gotenberg
    container_name: paperless-gotenberg   
    restart: always
    environment:
        DISABLE_GOOGLE_CHROME: 1
        DEFAULT_WAIT_TIMEOUT: 30
    networks:
      docker_frontend:

  paperless-tika:
    image: apache/tika
    container_name: paperless-tika   
    restart: always
    networks:
      docker_frontend:

networks:
  docker_qnet:
    external: true 
  docker_frontend:
    external: true 
  docker_backend:
    external: true
Sblop commented 3 years ago

I run various commands from a .sh file. I get the following errors:

Paperless commands document renamer Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedColumn: column documents_document.archive_filename does not exist LINE 1: ...cument"."added", "documents_document"."filename", "documents... ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "./manage.py", line 11, in execute_from_command_line(sys.argv) File "/usr/local/lib/python3.7/site-packages/django/core/management/init.py", line 401, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.7/site-packages/django/core/management/init.py", line 395, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv self.execute(*args, *cmd_options) File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute output = self.handle(args, **options) File "/usr/src/paperless/src/documents/management/commands/document_renamer.py", line 20, in handle for document in tqdm.tqdm(Document.objects.all()): File "/usr/local/lib/python3.7/site-packages/tqdm/std.py", line 978, in init total = len(iterable) File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 269, in len self._fetch_all() File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1308, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 53, in iter results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 90, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: column documents_document.archive_filename does not exist LINE 1: ...cument"."added", "documents_document"."filename", "documents... ^

Exception ignored in: <function tqdm.del at 0x7f245d600ef0> Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/tqdm/std.py", line 1134, in del File "/usr/local/lib/python3.7/site-packages/tqdm/std.py", line 1250, in close AttributeError: 'tqdm' object has no attribute 'disable' Train AI [2021-02-14 11:03:33,437] [WARNING] [paperless.tasks] Classifier error: column documents_document.archive_filename does not exist LINE 1: ...cument"."added", "documents_document"."filename", "documents...

Sblop commented 3 years ago

Commands are these:

echo "Paperless commands" echo "document renamer" docker exec paperless-webserver python3 ./manage.py document_renamer echo "Train AI" docker exec paperless-webserver python3 ./manage.py document_create_classifier echo "index reindex" docker exec paperless-webserver python3 ./manage.py document_index reindex echo "index optimize" docker exec paperless-webserver python3 ./manage.py document_index optimize echo "fetch mail" docker exec paperless-webserver python3 ./manage.py mail_fetcher echo "data export" docker exec paperless-webserver python3 ./manage.py document_exporter ../export echo "restart paperless" docker restart paperless-webserver -t 10

Sblop commented 3 years ago

I believe it might be related to bug report #534, but I am not able to figure this out.

jonaswinkler commented 3 years ago

Hello!

docker-compose exec webserver python3 manage.py showmigrations

Either search for a line that says [X] 1012_fix_archive_files and verify that it's there, or post any error.

docker-compose exec webserver python3 manage.py inspectdb

Search for something that looks like this:

class DocumentsDocument(models.Model):
    title = models.CharField(max_length=128)
    content = models.TextField()
    created = models.DateTimeField()
    modified = models.DateTimeField()
    correspondent = models.ForeignKey(DocumentsCorrespondent, models.DO_NOTHING, blank=True, null=True)
    checksum = models.CharField(unique=True, max_length=32)
    added = models.DateTimeField()
    storage_type = models.CharField(max_length=11)
    archive_serial_number = models.IntegerField(unique=True, blank=True, null=True)
    document_type = models.ForeignKey('DocumentsDocumenttype', models.DO_NOTHING, blank=True, null=True)
    mime_type = models.CharField(max_length=256)
    archive_checksum = models.CharField(max_length=32, blank=True, null=True)
    archive_filename = models.CharField(unique=True, max_length=1024, blank=True, null=True)
    filename = models.CharField(unique=True, max_length=1024, blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'documents_document'

and verify that the line archive_filename = ... is there or post any errors that might occur.

Sblop commented 3 years ago

Hi Jonas, I was running dev and all was going fine, however at some update it stopped working. From now on I will only run latest. Sorry for the inconvenience.

To answer your questions, in 1.1.0 1012 fix archive is shown., However archive_filename is not.

I have tried to manually force the migrations running ./manage.py migrate documents 1012 (No error), but running 1011, I get the following:

root@44afaf78d088:/usr/src/paperless/src# python3 ./manage.py migrate documents 1011 Operations to perform: Target specific migration: 1011_auto_20210101_2340, from documents Running migrations: Rendering model states... DONE Unapplying documents.1012_fix_archive_files...Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UndefinedColumn: column documents_document.archive_filename does not exist LINE 1: ...e_type", "documents_document"."archive_checksum", "documents... ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 245, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 121, in migrate
    state = self._migrate_all_backwards(plan, full_plan, fake=fake)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 196, in _migrate_all_backwards
    self.unapply_migration(states[migration], migration, fake=fake)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 251, in unapply_migration
    state = migration.unapply(state, schema_editor)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/migration.py", line 175, in unapply
    operation.database_backwards(self.app_label, schema_editor, from_state, to_state)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/operations/special.py", line 196, in database_backwards
    self.reverse_code(from_state.apps, schema_editor)
  File "/usr/src/paperless/src/documents/migrations/1012_fix_archive_files.py", line 286, in move_new_to_old_locations
    for doc in Document.objects.filter(archive_checksum__isnull=False):
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 287, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column documents_document.archive_filename does not exist
LINE 1: ...e_type", "documents_document"."archive_checksum", "documents...
                                                         ^

Inspect DB gives me this:


root@44afaf78d088:/usr/src/paperless/src# python3 ./manage.py inspectdb
# This is an auto-generated Django model module.
# You'll have to do the following manually to clean this up:
#   * Rearrange models' order
#   * Make sure each model has one field with primary_key=True
#   * Make sure each ForeignKey and OneToOneField has `on_delete` set to the desired behavior
#   * Remove `managed = False` lines if you wish to allow Django to create, modify, and delete the table
# Feel free to rename the models, but don't rename db_table values or field names.
from django.db import models

class AuthGroup(models.Model):
    name = models.CharField(unique=True, max_length=150)

    class Meta:
        managed = False
        db_table = 'auth_group'

class AuthGroupPermissions(models.Model):
    group = models.ForeignKey(AuthGroup, models.DO_NOTHING)
    permission = models.ForeignKey('AuthPermission', models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'auth_group_permissions'
        unique_together = (('group', 'permission'),)

class AuthPermission(models.Model):
    name = models.CharField(max_length=255)
    content_type = models.ForeignKey('DjangoContentType', models.DO_NOTHING)
    codename = models.CharField(max_length=100)

    class Meta:
        managed = False
        db_table = 'auth_permission'
        unique_together = (('content_type', 'codename'),)

class AuthUser(models.Model):
    password = models.CharField(max_length=128)
    last_login = models.DateTimeField(blank=True, null=True)
    is_superuser = models.BooleanField()
    username = models.CharField(unique=True, max_length=150)
    first_name = models.CharField(max_length=150)
    last_name = models.CharField(max_length=150)
    email = models.CharField(max_length=254)
    is_staff = models.BooleanField()
    is_active = models.BooleanField()
    date_joined = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'auth_user'

class AuthUserGroups(models.Model):
    user = models.ForeignKey(AuthUser, models.DO_NOTHING)
    group = models.ForeignKey(AuthGroup, models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'auth_user_groups'
        unique_together = (('user', 'group'),)

class AuthUserUserPermissions(models.Model):
    user = models.ForeignKey(AuthUser, models.DO_NOTHING)
    permission = models.ForeignKey(AuthPermission, models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'auth_user_user_permissions'
        unique_together = (('user', 'permission'),)

class AuthtokenToken(models.Model):
    key = models.CharField(primary_key=True, max_length=40)
    created = models.DateTimeField()
    user = models.OneToOneField(AuthUser, models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'authtoken_token'

class DjangoAdminLog(models.Model):
    action_time = models.DateTimeField()
    object_id = models.TextField(blank=True, null=True)
    object_repr = models.CharField(max_length=200)
    action_flag = models.SmallIntegerField()
    change_message = models.TextField()
    content_type = models.ForeignKey('DjangoContentType', models.DO_NOTHING, blank=True, null=True)
    user = models.ForeignKey(AuthUser, models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'django_admin_log'

class DjangoContentType(models.Model):
    app_label = models.CharField(max_length=100)
    model = models.CharField(max_length=100)

    class Meta:
        managed = False
        db_table = 'django_content_type'
        unique_together = (('app_label', 'model'),)

class DjangoMigrations(models.Model):
    app = models.CharField(max_length=255)
    name = models.CharField(max_length=255)
    applied = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'django_migrations'

class DjangoQOrmq(models.Model):
    key = models.CharField(max_length=100)
    payload = models.TextField()
    lock = models.DateTimeField(blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'django_q_ormq'

class DjangoQSchedule(models.Model):
    func = models.CharField(max_length=256)
    hook = models.CharField(max_length=256, blank=True, null=True)
    args = models.TextField(blank=True, null=True)
    kwargs = models.TextField(blank=True, null=True)
    schedule_type = models.CharField(max_length=1)
    repeats = models.IntegerField()
    next_run = models.DateTimeField(blank=True, null=True)
    task = models.CharField(max_length=100, blank=True, null=True)
    name = models.CharField(max_length=100, blank=True, null=True)
    minutes = models.SmallIntegerField(blank=True, null=True)
    cron = models.CharField(max_length=100, blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'django_q_schedule'

class DjangoQTask(models.Model):
    name = models.CharField(max_length=100)
    func = models.CharField(max_length=256)
    hook = models.CharField(max_length=256, blank=True, null=True)
    args = models.TextField(blank=True, null=True)
    kwargs = models.TextField(blank=True, null=True)
    result = models.TextField(blank=True, null=True)
    started = models.DateTimeField()
    stopped = models.DateTimeField()
    success = models.BooleanField()
    id = models.CharField(primary_key=True, max_length=32)
    group = models.CharField(max_length=100, blank=True, null=True)
    attempt_count = models.IntegerField()

    class Meta:
        managed = False
        db_table = 'django_q_task'

class DjangoSession(models.Model):
    session_key = models.CharField(primary_key=True, max_length=40)
    session_data = models.TextField()
    expire_date = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'django_session'

class DocumentsCorrespondent(models.Model):
    name = models.CharField(unique=True, max_length=128)
    match = models.CharField(max_length=256)
    matching_algorithm = models.IntegerField()
    is_insensitive = models.BooleanField()

    class Meta:
        managed = False
        db_table = 'documents_correspondent'

class DocumentsDocument(models.Model):
    title = models.CharField(max_length=128)
    content = models.TextField()
    created = models.DateTimeField()
    modified = models.DateTimeField()
    correspondent = models.ForeignKey(DocumentsCorrespondent, models.DO_NOTHING, blank=True, null=True)
    checksum = models.CharField(unique=True, max_length=32)
    added = models.DateTimeField()
    storage_type = models.CharField(max_length=11)
    filename = models.CharField(max_length=1024, blank=True, null=True)
    archive_serial_number = models.IntegerField(unique=True, blank=True, null=True)
    document_type = models.ForeignKey('DocumentsDocumenttype', models.DO_NOTHING, blank=True, null=True)
    mime_type = models.CharField(max_length=256)
    archive_checksum = models.CharField(max_length=32, blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'documents_document'

class DocumentsDocumentTags(models.Model):
    document = models.ForeignKey(DocumentsDocument, models.DO_NOTHING)
    tag = models.ForeignKey('DocumentsTag', models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'documents_document_tags'
        unique_together = (('document', 'tag'),)

class DocumentsDocumenttype(models.Model):
    name = models.CharField(unique=True, max_length=128)
    match = models.CharField(max_length=256)
    matching_algorithm = models.IntegerField()
    is_insensitive = models.BooleanField()

    class Meta:
        managed = False
        db_table = 'documents_documenttype'

class DocumentsLog(models.Model):
    group = models.UUIDField(blank=True, null=True)
    message = models.TextField()
    level = models.IntegerField()
    created = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'documents_log'

class DocumentsSavedview(models.Model):
    name = models.CharField(max_length=128)
    show_on_dashboard = models.BooleanField()
    show_in_sidebar = models.BooleanField()
    sort_field = models.CharField(max_length=128)
    sort_reverse = models.BooleanField()
    user = models.ForeignKey(AuthUser, models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'documents_savedview'

class DocumentsSavedviewfilterrule(models.Model):
    rule_type = models.IntegerField()
    value = models.CharField(max_length=128, blank=True, null=True)
    saved_view = models.ForeignKey(DocumentsSavedview, models.DO_NOTHING)

    class Meta:
        managed = False
        db_table = 'documents_savedviewfilterrule'

class DocumentsTag(models.Model):
    name = models.CharField(unique=True, max_length=128)
    colour = models.IntegerField()
    match = models.CharField(max_length=256)
    matching_algorithm = models.IntegerField()
    is_insensitive = models.BooleanField()
    is_inbox_tag = models.BooleanField()

    class Meta:
        managed = False
        db_table = 'documents_tag'

class PaperlessMailMailaccount(models.Model):
    name = models.CharField(unique=True, max_length=256)
    imap_server = models.CharField(max_length=256)
    imap_port = models.IntegerField(blank=True, null=True)
    imap_security = models.IntegerField()
    username = models.CharField(max_length=256)
    password = models.CharField(max_length=256)

    class Meta:
        managed = False
        db_table = 'paperless_mail_mailaccount'

class PaperlessMailMailrule(models.Model):
    name = models.CharField(unique=True, max_length=256)
    folder = models.CharField(max_length=256)
    filter_from = models.CharField(max_length=256, blank=True, null=True)
    filter_subject = models.CharField(max_length=256, blank=True, null=True)
    filter_body = models.CharField(max_length=256, blank=True, null=True)
    maximum_age = models.IntegerField()
    action = models.IntegerField()
    action_parameter = models.CharField(max_length=256, blank=True, null=True)
    assign_title_from = models.IntegerField()
    assign_correspondent_from = models.IntegerField()
    account = models.ForeignKey(PaperlessMailMailaccount, models.DO_NOTHING)
    assign_correspondent = models.ForeignKey(DocumentsCorrespondent, models.DO_NOTHING, blank=True, null=True)
    assign_document_type = models.ForeignKey(DocumentsDocumenttype, models.DO_NOTHING, blank=True, null=True)
    assign_tag = models.ForeignKey(DocumentsTag, models.DO_NOTHING, blank=True, null=True)
    order = models.IntegerField()
    attachment_type = models.IntegerField()
    filter_attachment_filename = models.CharField(max_length=256, blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'paperless_mail_mailrule'
root@44afaf78d088:/usr/src/paperless/src#
jonaswinkler commented 3 years ago

To answer your questions, in 1.1.0 1012 fix archive is shown., However archive_filename is not.

Just to explain what's going on: Halfway through addressing #511 I decided to take a different approach and alter that migration. This was absolutely necessary to do things properly. The version you executed on your DB did not introduce that new database field, however, the new version does. This is usually fine on the dev branch, since the dev branch isn't exactly supposed to be used by anyone (I don't advertise that anywhere, it's more of a very useful tool for me to verify new releases easily). Now, when unapplying that new migration it also wants to remove that database field, and it's not even there to begin with.

The easy way out is to restore a backup of your database and media folder before encountering the issue.

Or (absolutely no guarantees):

Sblop commented 3 years ago

Hi Jonas,

Thanks for the above. Unfortunately it didnt work. I have reverted to my backup from february 1st and added the latest documents again. Thanks again for your assistance and have a nice weekend.