BUG: Sometimes, the creation of zipped archives never leaves the execution phase

[kimakan]

The archive job (creation of the zipped files) never finishes if one of the requested files is missing in the archive. The task fails with the error message (see below) and doesn't change the job phase to PHASE_ERROR remaining in the PHASE_EXECUTING forever.

ERROR daiquiri on_failure: Traceback (most recent call last):
  File "/home/dq/.local/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/dq/.local/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/dq/source/daiquiri/query/tasks.py", line 327, in create_download_archive_task
    z.write(file_path)
  File "/usr/lib/python3.9/zipfile.py", line 1727, in write
    zinfo = ZipInfo.from_file(filename, arcname,
  File "/usr/lib/python3.9/zipfile.py", line 501, in from_file
    st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: 'DR4/scans/HAM-LA/LA00944_y.fits'

The responsible code is https://github.com/django-daiquiri/daiquiri/blob/9132cb90128bd3d93dae99fadbaf3391a9e3baf9/daiquiri/query/tasks.py#L319-L331

The simplest solution would be

        # create a zipfile with all files
        try:
            with zipfile.ZipFile(archive_job.file_path, 'w') as z:
                os.chdir(settings.FILES_BASE_PATH)
                for file_path in archive_job.files:
                    z.write(file_path)
        except Exception as e:
            archive_job.phase = download_job.PHASE_ERROR
            archive_job.error_summary = str(e)
            archive_job.save()
            logger.info('archive_job %s failed (%s)' % (archive_job.id, archive_job.error_summary))
            raise e