If backend is stoped in middle of LongRunningJob it will get stuck in the db forever

[guysoft]

Steps to reproduce:

1. start ownphotos
2. run a directory scan with a large directory (few hundred images)
3. stop containers/process before its finished.
4. start service

Expected: All stuck db jobs should be cleaned .

Actual: Job hangs finished_at field has NULL. The issue can be seen using Adminer, looks like this:

[guysoft]

@gregbert42

You append to your docker-compose.yml:

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080
    container_name: ownphotos-adminer
    links:
      - "ownphotos-db:db"

Then docker-compose up -d

Then access it via port 8080, login using your posgres as user, db as host and password are your set password. From there you can find the table and click the "select" word next to its name.

[danielsjf]

For other people trying this, the code above should be inserted in the services part, not after the volume part (which is at the end of the default file).

The full login is:

System: PostgreSQL
Server: [DB_HOST from your docker-compose]
Username: [DB_USER from your docker-compose]
Password: [DB_PASS from your docker-compose]
Database: [LEAVE EMPTY]

[guysoft]

Still getting this, the current error is a little different:

Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.IntegrityError: duplicate key value violates unique constraint "api_longrunningjob_job_id_key"
DETAIL:  Key (job_id)=(09ff6379-4fdf-4451-9de3-d976625c0646) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/miniconda/lib/python3.6/site-packages/rq/worker.py", line 793, in perform_job
    rv = job.perform()
  File "/miniconda/lib/python3.6/site-packages/rq/job.py", line 599, in perform
    self._result = self._execute()
  File "/miniconda/lib/python3.6/site-packages/rq/job.py", line 605, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/code/api/directory_watcher.py", line 178, in scan_photos
    job_type=LongRunningJob.JOB_SCAN_PHOTOS)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/query.py", line 413, in create
    obj.save(force_insert=True, using=self.db)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save
    force_update=force_update, update_fields=update_fields)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert
    using=using, raw=raw)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/miniconda/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
    cursor.execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/miniconda/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/miniconda/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "api_longrunningjob_job_id_key"
DETAIL:  Key (job_id)=(09ff6379-4fdf-4451-9de3-d976625c0646) already exists.

07:56:05 Invoking exception handler <bound method Worker.move_to_failed_queue of <rq.worker.Worker object at 0x7fa61f660b70>>
07:56:05 Moving job to 'failed' queue
07:56:05 Sent heartbeat to prevent worker timeout. Next one should arrive within 420 seconds.
07:56:05 Sent heartbeat to prevent worker timeout. Next one should arrive within 420 seconds.
07:56:05 *** Listening on default...
07:56:05 Sent heartbeat to prevent worker timeout. Next one should arrive within 420 seconds.

**
**