search

mediacms-io / mediacms

MediaCMS is a modern, fully featured open source video and media CMS, written in Python/Django and React, featuring a REST API.
https://mediacms.io
GNU Affero General Public License v3.0
2.86k stars 533 forks source link

Cannot upload files #1058

Closed Code-Quake closed 1 month ago

Code-Quake commented 2 months ago

Describe the issue Whenever I try to upload a file it gets to a certain extent, starts retrying and fails. The size of the file doesn't matter. I tried it with a 9MB file and it gets to 66% retries twice and fails every time.

To Reproduce Steps to reproduce the issue:

  1. Go to http://localhost:8680/upload
  2. Upload a file
  3. See error

Expected behavior Video uploads

Screenshots image

Environment (please complete the following information):

mgogoulos commented 2 months ago

What is the OS of the base system? Is it Linux, Max or Windows?

What do the logs suggest? I imagine this must be a permissions issue. Do a tail -f logs/* while you upload a file. Also check what the web container shows, with docker-compose logs -f web. If you see an error, place it here so I can debug

dudubroering commented 1 month ago

I have the same issue, but I'm not using "Docker Install" method.

The strange thing is that it worked normally and no changes or manual updates were made. Only after the problem, I tried to update to the latest version, but it still didn't work.

This is the log:

==> logs/debug.log <==
Internal Server Error: /fu/upload/
Traceback (most recent call last):
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 49, in dispatch
    return super(FineUploaderView, self).dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/edit.py", line 153, in post
    return self.form_valid(form)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 62, in form_valid
    self.upload.save()
  File "/home/mediacms.io/mediacms/uploader/fineuploader.py", line 90, in save
    chunk = self._save_chunk()
  File "/home/mediacms.io/mediacms/uploader/fineuploader.py", line 86, in _save_chunk
    return self.storage.save(self.chunk_file, self.file)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/storage/base.py", line 38, in save
    name = self._save(name, content)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/storage/filesystem.py", line 110, in _save
    for chunk in content.chunks():
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/base.py", line 60, in chunks
    data = self.read(chunk_size)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/utils.py", line 42, in <lambda>
    read = property(lambda self: self.file.read)
AttributeError: 'NoneType' object has no attribute 'read'
Internal Server Error: /fu/upload/
Traceback (most recent call last):
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 49, in dispatch
    return super(FineUploaderView, self).dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/edit.py", line 153, in post
    return self.form_valid(form)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 62, in form_valid
    self.upload.save()
  File "/home/mediacms.io/mediacms/uploader/fineuploader.py", line 90, in save
    chunk = self._save_chunk()
  File "/home/mediacms.io/mediacms/uploader/fineuploader.py", line 86, in _save_chunk
    return self.storage.save(self.chunk_file, self.file)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/storage/base.py", line 38, in save
    name = self._save(name, content)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/storage/filesystem.py", line 110, in _save
    for chunk in content.chunks():
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/base.py", line 60, in chunks
    data = self.read(chunk_size)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/utils.py", line 42, in <lambda>
    read = property(lambda self: self.file.read)
AttributeError: 'NoneType' object has no attribute 'read'
Internal Server Error: /fu/upload/
Traceback (most recent call last):
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 49, in dispatch
    return super(FineUploaderView, self).dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/views/generic/edit.py", line 153, in post
    return self.form_valid(form)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 62, in form_valid
    self.upload.save()
  File "/home/mediacms.io/mediacms/uploader/fineuploader.py", line 90, in save
    chunk = self._save_chunk()
  File "/home/mediacms.io/mediacms/uploader/fineuploader.py", line 86, in _save_chunk
    return self.storage.save(self.chunk_file, self.file)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/storage/base.py", line 38, in save
    name = self._save(name, content)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/storage/filesystem.py", line 110, in _save
    for chunk in content.chunks():
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/base.py", line 60, in chunks
    data = self.read(chunk_size)
  File "/home/mediacms.io/lib/python3.9/site-packages/django/core/files/utils.py", line 42, in <lambda>
    read = property(lambda self: self.file.read)
AttributeError: 'NoneType' object has no attribute 'read'
mgogoulos commented 1 month ago

@dudubroering I'm not sure I understand what you write. Do you have the same issue while working on the latest version?

Also can you add the formating in a way that can be easily read? It's not easy to spot the issue now!

Code-Quake commented 1 month ago

I got past this by disabling the video transcoding

dudubroering commented 1 month ago

@dudubroering I'm not sure I understand what you write. Do you have the same issue while working on the latest version?

Also can you add the formating in a way that can be easily read? It's not easy to spot the issue now!

Yes. Files up to 2mb are working normally. But the limit in local_settings.py is 4GB and in nginx it is 10GB. Is there any other place that needs to adjust the file size limit?

mgogoulos commented 1 month ago

Hi @dudubroering , did you see my last comment? What OS are you using as the host for MediaCMS?

This seems as a permissions issue, not as an nginx one, unless there is some other setting added by you. By default, any file up to 5GB should be uploaded without issues. I haven't tested the system how it behaves in Mac and Windows, I mostly run it in Linux systems. Maybe in Mac/Windows some other setting needs to be made in the Dockerfile in order for the upload dir (media_files/chunks) to be writable

In my case, where I'm running MediaCMS with Docker, from an Ubuntu system, this is what I see 

user@user:~/mediacms$ ls -l
...
drwxrwxr-x  8 www-data user  4096 Σεπ  19 17:10 media_files
...

user@user:~/mediacms$ ls -l media_files/
total 24
drwxr-xr-x  2 www-data user 4096 Οκτ   8 16:43 chunks
drwxr-xr-x  8 www-data user 4096 Σεπ  19 17:10 encoded
...

all files are having www-data.user as the owner (user is my username btw). Can you do ls -l on these folders and type what is the output too?

dudubroering commented 1 month ago

I'm using Debian GNU/Linux 11 (bullseye).

I remember it stopped working without any changes being made. So it was working normally a few months ago.

Apparently permissions are similar to yours:

root@video:/home/mediacms.io/mediacms/media_files# ls -lah
total 48K
drwxr-xr-x  9 www-data www-data 4.0K Sep 24 18:06 .
drwxr-xr-x 21 www-data www-data 4.0K Oct  7 19:56 ..
drwxr-xr-x 56 www-data www-data 4.0K Oct  7 20:25 chunks
drwxr-xr-x  8 www-data www-data 4.0K Mar 24  2023 encoded
drwxr-xr-x 61 www-data www-data 4.0K Oct  7 19:28 hls
drwx------  2 www-data www-data  16K Apr  8  2023 lost+found
drwxr-xr-x  4 www-data www-data 4.0K Feb  7  2023 original
drwxr-xr-x 24 www-data www-data 4.0K Oct  7 20:25 uploads
drwxr-xr-x  3 www-data www-data 4.0K Sep 19 14:08 userlogos
root@video:/home/mediacms.io/mediacms# ls -alh
total 12M
drwxr-xr-x 21 www-data www-data 4.0K Oct  7 19:56 .
drwxr-xr-x  5 www-data www-data 4.0K Oct  7 20:16 ..
-rw-r--r--  1 www-data www-data   45 Sep 19 14:08 .coveragerc
-rw-r--r--  1 www-data www-data   26 Sep 19 14:08 .dockerignore
drwxr-xr-x  8 www-data www-data 4.0K Sep 19 14:08 .git
drwxr-xr-x  4 www-data www-data 4.0K Sep 19 14:08 .github
-rw-r--r--  1 www-data www-data  347 Sep 19 14:08 .gitignore
-rw-r--r--  1 www-data www-data  387 Sep 19 14:08 .pre-commit-config.yaml
-rw-r--r--  1 www-data www-data  124 Sep 19 14:08 AUTHORS.txt
drwxr-xr-x  7 www-data www-data 4.0K Feb  7  2023 Bento4-SDK-1-6-0-637.x86_64-unknown-linux
-rw-r--r--  1 www-data www-data  11M Jun 13  2020 Bento4-SDK-1-6-0-637.x86_64-unknown-linux.zip
-rw-r--r--  1 www-data www-data 1.4K Sep 19 14:08 CODE_OF_CONDUCT.md
-rw-r--r--  1 www-data www-data 2.3K Sep 19 14:08 Dockerfile
-rw-r--r--  1 www-data www-data 2.3K Sep 19 14:08 Dockerfile-dev
-rw-r--r--  1 www-data www-data 1.7K Sep 19 14:08 HISTORY.md
-rw-r--r--  1 www-data www-data  34K Sep 19 14:08 LICENSE.txt
-rw-r--r--  1 www-data www-data 6.7K Sep 19 14:08 README.md
drwxr-xr-x  4 www-data www-data 4.0K Sep 19 14:08 actions
-rw-r--r--  1 www-data www-data  12K Oct  7 20:21 celerybeat-schedule.db
drwxr-xr-x  2 www-data www-data 4.0K Sep 19 14:08 cli-tool
drwxr-xr-x  3 www-data www-data 4.0K Oct  7 19:44 cms
-rw-r--r--  1 www-data www-data  109 Sep 19 14:08 conftest.py
drwxr-xr-x  4 www-data www-data 4.0K Feb  7  2023 deploy
-rw-r--r--  1 www-data www-data 2.0K Sep 19 14:08 docker-compose-dev.yaml
-rw-r--r--  1 www-data www-data 2.3K Sep 19 14:08 docker-compose-http-proxy.yaml
-rw-r--r--  1 www-data www-data 2.4K Sep 19 14:08 docker-compose-https-proxy.yaml
-rw-r--r--  1 www-data www-data 3.0K Sep 19 14:08 docker-compose-letsencrypt.yaml
-rw-r--r--  1 www-data www-data 2.6K Sep 19 14:08 docker-compose-named-volumes.yaml
-rw-r--r--  1 www-data www-data 2.1K Sep 19 14:08 docker-compose.yaml
drwxr-xr-x  3 www-data www-data 4.0K Sep 19 14:08 docs
drwxr-xr-x  5 www-data www-data 4.0K Sep 24 17:56 files
drwxr-xr-x  2 www-data www-data 4.0K Sep 19 14:08 fixtures
drwxr-xr-x  6 www-data www-data 4.0K Sep 19 14:08 frontend
-rw-r--r--  1 www-data www-data  11K Sep 19 14:08 install-rhel.sh
-rw-r--r--  1 www-data www-data 6.7K Sep 19 14:08 install.sh
drwxr-xr-x  2 www-data www-data 4.0K Sep 19 12:02 logs
-rwxr-xr-x  1 www-data www-data  537 Sep 19 14:08 manage.py
drwxr-xr-x  9 www-data www-data 4.0K Sep 24 18:06 media_files
drwxr-xr-x  2 www-data www-data 4.0K Oct  7 20:22 pids
-rw-r--r--  1 www-data www-data  310 Sep 19 14:08 pyproject.toml
-rw-r--r--  1 www-data www-data  110 Sep 19 14:08 pytest.ini
-rw-r--r--  1 www-data www-data  151 Sep 19 14:08 requirements-dev.txt
-rw-r--r--  1 www-data www-data  429 Sep 19 14:08 requirements.txt
-rw-r--r--  1 www-data www-data   99 Sep 19 14:08 setup.cfg
drwxr-xr-x 13 www-data www-data 4.0K Feb  7  2023 static
drwxr-xr-x  7 www-data www-data 4.0K Sep 19 14:08 templates
drwxr-xr-x  8 www-data www-data 4.0K Sep 19 14:08 tests
drwxr-xr-x  3 www-data www-data 4.0K Oct  7 19:33 uploader
drwxr-xr-x  4 www-data www-data 4.0K Oct  7 19:32 users
-rw-r--r--  1 www-data www-data   18 Sep 19 14:08 version.py
mgogoulos commented 1 month ago

Strange. Is there a chance that the filesystem is 100% filled in? I haven't seen this error before and would be curious to troubleshoot. I can send you my public key so I can temporarily ssh to the server if you want.

dudubroering commented 1 month ago

No, the media_files partition have 70% of usage today.

~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          1.9G     0  1.9G   0% /dev
tmpfs                         392M  588K  392M   1% /run
/dev/vda1                      50G  7.3G   40G  16% /
tmpfs                         2.0G   64K  2.0G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/mapper/media_files-lv01  196G  129G   58G  70% /home/mediacms.io/mediacms/media_files
/dev/vda15                    124M   11M  114M   9% /boot/efi
tmpfs                         392M     0  392M   0% /run/user/1000