KyleMaas
commented
10 months ago Aha, caught one in the act. From the Celery log:
[2023-12-02 15:06:34,881: ERROR/ForkPoolWorker-4] Task encode_media[91a5fb7e-4692-4c6f-b872-cfbd757d8985] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
File "/home/mediacms.io/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/mediacms.io/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "(path redacted)/files/tasks.py", line 365, in encode_media
encoding.media_file.save(content=myfile, name=output_name)
File "/home/mediacms.io/lib/python3.10/site-packages/django/db/models/fields/files.py", line 99, in save
self.instance.save()
File "(path redacted)/files/models.py", line 1171, in save
super(Encoding, self).save(*args, **kwargs)
File "/home/mediacms.io/lib/python3.10/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/home/mediacms.io/lib/python3.10/site-packages/django/db/models/base.py", line 892, in save_base
post_save.send(
File "/home/mediacms.io/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 176, in send
return [
File "/home/mediacms.io/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
(receiver, receiver(signal=self, sender=sender, **named))
File "(path redacted)/files/models.py", line 1569, in encoding_file_save
encoding.save()
File "(path redacted)/files/models.py", line 1171, in save
super(Encoding, self).save(*args, **kwargs)
File "/home/mediacms.io/lib/python3.10/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/home/mediacms.io/lib/python3.10/site-packages/django/db/models/base.py", line 892, in save_base
post_save.send(
File "/home/mediacms.io/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 176, in send
return [
File "/home/mediacms.io/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
(receiver, receiver(signal=self, sender=sender, **named))
File "(path redacted)/files/models.py", line 1629, in encoding_file_save
instance.media.post_encode_actions(encoding=instance, action="add")
File "(path redacted)/files/models.py", line 643, in post_encode_actions
tasks.create_hls(self.friendly_token)
File "/home/mediacms.io/lib/python3.10/site-packages/celery/local.py", line 182, in __call__
return self._get_current_object()(*a, **kw)
File "/home/mediacms.io/lib/python3.10/site-packages/celery/app/trace.py", line 761, in __protected_call__
return orig(self, *args, **kwargs)
File "/home/mediacms.io/lib/python3.10/site-packages/celery/app/task.py", line 411, in __call__
return self.run(*args, **kwargs)
File "(path redacted)/files/tasks.py", line 445, in create_hls
shutil.rmtree(output_dir)
File "/usr/lib/python3.10/shutil.py", line 715, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib/python3.10/shutil.py", line 713, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '(path redacted)/media_files/hls/5033c4cda075419cb11bd712b21a88c66Di5uhLjQ'
Describe the issue Often times when transcoding to an MP4 profile one of the chunks of an encoding will come back as an encoding fail but the video seems to play fine if I pull the actual encoded file. The logs don't show any errors and look the same as successful chunks,
Retriesis0,StatusisFail, andTotal run timeis also0. But then if I click onEncoding fileit plays just fine and everything looks okay. I also never see this with VP9 encoding.To Reproduce I'm not sure how to reproduce this. However, I did not see this problem until I moved to a server setup with much higher capacity but also higher disk latency. I've raised the timeouts for Celery to allow each task to run for considerably longer, far longer than successful chunks take to encode, but the issue persists.
Expected behavior I would expect that the chunk, which from everything I'm seeing did in fact succeed in encoding, would be marked as
Successinstead. What I find most odd is thatTotal run timeis0- seems like even for a failure, that should be filled in with some kind of a value.Screenshots Can't share screenshots of this setup unfortunately.
Environment (please complete the following information):
Additional context Add any other context about the problem here.