alexbelgium / hassio-addons

My homeassistant addons
MIT License
1.36k stars 194 forks source link

🐛 [paperless-ngx] document exporter not working #1317

Open KapUttyy opened 3 months ago

KapUttyy commented 3 months ago

Description

As your Paperless-ngx add-on is deprecated, I would like to export all my stuff from Paperless NGX and import it into a new instance as the Addon by BenoitAnastay is not giving me the option to use my installation with the MariaDB as DBMS as your add-on does.

Paperless is offering a document exporter for this (https://docs.paperless-ngx.com/administration/#exporter), but it is not working with your add-on.

Unfortunately it is not working as it seems to be trying to use the sqlite DB instead of the connected MariaDB for the export and so not finding any data.

Reproduction steps

1. Go to 'Home Assistant CLI'
2. Enter the AddOns Container: "docker exec --tty -i add-on_UID_paperless_ng /bin/bash"
3. run "document_exporter ../export"
4. See error

Addon Logs

root@UID-paperless-ng:/usr/src/paperless/src# document_exporter ../export/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such table: documents_correspondent

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/paperless/src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)                                                                                                                                                                   
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                   
  File "/usr/src/paperless/src/documents/management/commands/document_exporter.py", line 203, in handle                                                                                                      
    self.dump(options["no_progress_bar"])                                                                                                                                                                    
  File "/usr/src/paperless/src/documents/management/commands/document_exporter.py", line 232, in dump                                                                                                        
    serializers.serialize("json", Correspondent.objects.all()),                                                                                                                                              
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                               
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/__init__.py", line 134, in serialize                                                                                                 
    s.serialize(queryset, **options)                                                                                                                                                                         
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/base.py", line 132, in serialize                                                                                                     
    for count, obj in enumerate(queryset, start=1):                                                                                                                                                          
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                           
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 398, in __iter__                                                                                                            
    self._fetch_all()                                                                                                                                                                                        
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all                                                                                                         
    self._result_cache = list(self._iterable_class(self))                                                                                                                                                    
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                       
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: no such table: documents_correspondent

Here are my Paperless set variables shown when running "set" inside the container:

PAPERLESS_ADMIN_PASSWORD=#######
PAPERLESS_ADMIN_USER=########
PAPERLESS_CONSUMPTION_DIR=/config/addons_config/paperless_ng/consume
PAPERLESS_DATA_DIR=/config/addons_config/paperless_ng/data
PAPERLESS_DBENGINE=mariadb
PAPERLESS_DBHOST=core-mariadb
PAPERLESS_DBNAME=paperless
PAPERLESS_DBPASS=######################################
PAPERLESS_DBPORT=3306
PAPERLESS_DBUSER=service
PAPERLESS_EXPORT_DIR=/config/export
PAPERLESS_MEDIA_ROOT=/mnt/Paperless-DMS/media
PAPERLESS_OCR_LANGUAGE=eng
PAPERLESS_OCR_MODE=skip
PAPERLESS_TIME_ZONE=Europe/Berlin
PAPERLESS_URL=https://paperless.###########.###########.##

Architecture

amd64

OS

HAos

github-actions[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

alexbelgium commented 3 months ago

Hi, not sure how to be able to help you... Btw the addon will continue to be updated and will continue working, the deprecation notice is more because I truly believe the repo I have linked has a better one but this one will continue updating as before ;-)

Have you tried to run the exporter using portainer for example to have a better handle on the cli arguments ?

KapUttyy commented 3 months ago

If you say your addon will continue to be updated, you already helped me. ;-)

I have tried running the exporter using portainer, but I get the exact same error messages:

root@db21ed7f-paperless-ng:/usr/src/paperless/src# document_exporter ../export/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such table: documents_correspondent

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/paperless/src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/management/commands/document_exporter.py", line 203, in handle
    self.dump(options["no_progress_bar"])
  File "/usr/src/paperless/src/documents/management/commands/document_exporter.py", line 232, in dump
    serializers.serialize("json", Correspondent.objects.all()),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/__init__.py", line 134, in serialize
    s.serialize(queryset, **options)
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/base.py", line 132, in serialize
    for count, obj in enumerate(queryset, start=1):
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 398, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return super().execute(query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: no such table: documents_correspondent
root@db21ed7f-paperless-ng:/usr/src/paperless/src# 

I have already tried to export everything filebased and dumped the database with phpmyadmin. Then I installed a new docker host and I was able to import in a native Paperless NGX docker-instance on that, but some documents were corrupted due to coding issues in the filenames. If you say, you are updating the addon, I will try that again later with some work on those automatic filenames, which caused the issues. But I think I won't have time for that experiments in the next few month ;-)

alexbelgium commented 3 months ago

No hurry indeed the addon should count nue working and be updated for some time as it is still on the automatic system!