search

mediacms-io / mediacms

MediaCMS is a modern, fully featured open source video and media CMS, written in Python/Django and React, featuring a REST API.
https://mediacms.io
GNU Affero General Public License v3.0
2.67k stars 496 forks source link

Transcoding Fails on Large Files #689

Closed jsimons256 closed 1 year ago

jsimons256 commented 1 year ago

Environment (please complete the following information):

Describe the issue Encodings fail on large files at high resolutions. I am running MediaCMS on a VM through TrueNas Scale with media_files and pg_dump mounted as NFS shares. I initially assigned 24 threads and 8 GB or RAM to the VM. It worked well with 1080p mp4s under 1 GB with all encodings succeeding. For 1080p encodings over 1 GB, the 720p and under encodings work while the 1080p encoding fails. I changed from allocating 8 to 12 GB or RAM and the 1080p encoding worked for ~1 GB files but failed for 2 GB files.

I also noticed that the chunks and hls folders are empty. Additionally, when I delete the mp4, the chunks in the originals folder labeled 0x....mp4.mkv do not get deleted. I initially thought this was a permission error with the NFS shares, but I don't understand why allocating more RAM can fix the error. I'm also pretty sure that my setup is enough to handle these encodings. Any help would appreciated.

Logs

celery_long.log

[2023-01-06 04:06:01,801: INFO/ForkPoolWorker-7] encode_media[7c5829a6-b598-4312-bb97-f6677e41a653]: Saved 4.57 [2023-01-06 04:06:03,628: WARNING/MainProcess] Soft time limit (7200s) exceeded for encode_media[5f0daa50-f971-4c6c-b271-014ca4e6a5e1] [2023-01-06 04:06:03,847: ERROR/ForkPoolWorker-4] Task encode_media[5f0daa50-f971-4c6c-b271-014ca4e6a5e1] raised unexpected: ValueError("The 'media_file' attribute has no file associated with it.") Traceback (most recent call last): File "/home/mediacms.io/mediacms/files/tasks.py", line 308, in encode_media output = next(encoding_command) File "/home/mediacms.io/mediacms/files/backends.py", line 50, in encode out = process.stderr.read(10) File "/home/mediacms.io/lib/python3.8/site-packages/billiard/pool.py", line 229, in soft_timeout_sighandler raise SoftTimeLimitExceeded() billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/mediacms.io/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task R = retval = fun(*args, kwargs) File "/home/mediacms.io/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__ return self.run(*args, *kwargs) File "/home/mediacms.io/mediacms/files/tasks.py", line 335, in encode_media encoding.save(update_fields=["status", "logs"]) File "/home/mediacms.io/mediacms/files/models.py", line 1115, in save super(Encoding, self).save(args, kwargs) File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save self.save_base(using=using, force_insert=force_insert, File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 801, in save_base post_save.send( File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 177, in send return [ File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 178, in (receiver, receiver(signal=self, sender=sender, **named)) File "/home/mediacms.io/mediacms/files/models.py", line 1554, in encoding_file_save chunks_paths = [f.media_file.path for f in chunks] File "/home/mediacms.io/mediacms/files/models.py", line 1554, in chunks_paths = [f.media_file.path for f in chunks] File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/fields/files.py", line 57, in path self._require_file() File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/fields/files.py", line 39, in _require_file raise ValueError("The '%s' attribute has no file associated with it." % self.field.name) ValueError: The 'media_file' attribute has no file associated with it.

livvius commented 1 year ago

I can confirm that issue

Micrufun commented 1 year ago

Temporary solution

A temporary workaround is disabling the Encode profiles for any profile other than 240 and 360:

https://github.com/mediacms-io/mediacms/blob/main/docs/admins_docs.md#12-video-transcoding

Add / remove resolutions and profiles by modifying the database table of Encode profiles through https://your_installation/admin/files/encodeprofile/

For example, the Active state of any profile can be toggled to enable or disable it.

Not the best solution, just a temporary workaround.