search

mediacms-io / mediacms

MediaCMS is a modern, fully featured open source video and media CMS, written in Python/Django and React, featuring a REST API.
https://mediacms.io
GNU Affero General Public License v3.0
2.53k stars 459 forks source link

ffprobe outputs an invalid integer in some cases #417

Open tschig opened 2 years ago

tschig commented 2 years ago

Describe the issue I am trying to upload video.mkv File with ~200mb. When I Upload it via "https://mediacms.exmaple.com/upload" the fineupload "?done" request fails with Status Code 400 and Response:

Request URL: https://mediacms.exmaple.com/fu/upload/?done
Request Method: POST
Status Code: 400
{"success": false, "error": "Error with File Uploading"}

On the server, the file video.mkv is present and does have the same md5 as the file on my local harddrive.

Server:

root@d0dd55e3af29:/home/mediacms.io/mediacms/media_files/uploads/1f2ca7a0-167b-42a8-9a43-84a54e85b406# pwd
/home/mediacms.io/mediacms/media_files/uploads/1f2ca7a0-167b-42a8-9a43-84a54e85b406
root@d0dd55e3af29:/home/mediacms.io/mediacms/media_files/uploads/1f2ca7a0-167b-42a8-9a43-84a54e85b406# ls -al
total 204236
drwxr-xr-x 2 www-data www-data       4096 Feb 19 10:49 .
drwxr-xr-x 9 www-data tempgroup      4096 Feb 19 10:40 ..
-rw-r--r-- 1 www-data www-data  209126313 Feb 19 10:40 video.mkv
-rw-r--r-- 1 www-data www-data          0 Feb 19 10:49 video_8pZlTQB.mkv
-rw-r--r-- 1 www-data www-data          0 Feb 19 10:41 video_aqUqPbb.mkv
-rw-r--r-- 1 www-data www-data          0 Feb 19 10:41 video_oLhah4M.mkv
root@d0dd55e3af29:/home/mediacms.io/mediacms/media_files/uploads/1f2ca7a0-167b-42a8-9a43-84a54e85b406# md5sum video.mkv
0a1dcb25b162786b87826b95be394584  video.mkv

Local:

C:\Users\justi>md5sum video.mkv
0a1dcb25b162786b87826b95be394584 *video.mkv

So upload was correct. When trying the same with a smaller video file, everything works. When trying the same file with a Single Server installation in a VM everything works

Log from Docker Container "web":


[pid: 61|app: 0|req: 125/442] 172.25.0.8 () {76 vars in 1671 bytes} [Sat Feb 19 10:40:58 2022] POST /fu/upload/ => generated 17 bytes in 15 msecs (HTTP/1.1 200) 6 headers in 175 bytes (1 switches on core 0)
172.25.0.8 - - [19/Feb/2022:10:40:58 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
2022/02/19 10:40:58 [warn] 58#58: *358 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000419, client: 172.25.0.8, server: , request: "POST /fu/upload/ HTTP/1.1", host: "mediacms.example.com", referrer: "https://mediacms.example.com/upload"
[pid: 63|app: 0|req: 318/443] 172.25.0.8 () {76 vars in 1671 bytes} [Sat Feb 19 10:40:58 2022] POST /fu/upload/ => generated 17 bytes in 15 msecs (HTTP/1.1 200) 6 headers in 175 bytes (2 switches on core 1)
172.25.0.8 - - [19/Feb/2022:10:40:58 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
[pid: 63|app: 0|req: 319/444] 172.25.0.8 () {76 vars in 1671 bytes} [Sat Feb 19 10:40:58 2022] POST /fu/upload/ => generated 17 bytes in 15 msecs (HTTP/1.1 200) 6 headers in 175 bytes (1 switches on core 0)
172.25.0.8 - - [19/Feb/2022:10:40:58 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
2022/02/19 10:40:58 [warn] 58#58: *357 a client request body is buffered to a temporary file /var/lib/nginx/body/0000000420, client: 172.25.0.8, server: , request: "POST /fu/upload/ HTTP/1.1", host: "mediacms.example.com", referrer: "https://mediacms.example.com/upload"
[pid: 61|app: 0|req: 126/445] 172.25.0.8 () {76 vars in 1671 bytes} [Sat Feb 19 10:40:58 2022] POST /fu/upload/ => generated 17 bytes in 15 msecs (HTTP/1.1 200) 6 headers in 175 bytes (1 switches on core 1)
172.25.0.8 - - [19/Feb/2022:10:40:58 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
[pid: 63|app: 0|req: 320/446] 172.25.0.8 () {76 vars in 1671 bytes} [Sat Feb 19 10:40:58 2022] POST /fu/upload/ => generated 17 bytes in 13 msecs (HTTP/1.1 200) 6 headers in 175 bytes (1 switches on core 1)
172.25.0.8 - - [19/Feb/2022:10:40:58 +0000] "POST /fu/upload/ HTTP/1.1" 200 17 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
[pid: 63|app: 0|req: 321/447] 172.25.0.8 () {72 vars in 1533 bytes} [Sat Feb 19 10:40:59 2022] POST /fu/upload/?done => generated 21654 bytes in 2245 msecs (HTTP/1.1 500) 6 headers in 206 bytes (1 switches on core 0)
172.25.0.8 - - [19/Feb/2022:10:41:01 +0000] "POST /fu/upload/?done HTTP/1.1" 500 21654 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
[pid: 63|app: 0|req: 322/448] 172.25.0.8 () {72 vars in 1533 bytes} [Sat Feb 19 10:41:06 2022] POST /fu/upload/?done => generated 56 bytes in 9 msecs (HTTP/1.1 400) 6 headers in 184 bytes (1 switches on core 1)
172.25.0.8 - - [19/Feb/2022:10:41:06 +0000] "POST /fu/upload/?done HTTP/1.1" 400 56 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
[pid: 63|app: 0|req: 323/449] 172.25.0.8 () {72 vars in 1533 bytes} [Sat Feb 19 10:41:11 2022] POST /fu/upload/?done => generated 56 bytes in 8 msecs (HTTP/1.1 400) 6 headers in 184 bytes (1 switches on core 0)
172.25.0.8 - - [19/Feb/2022:10:41:11 +0000] "POST /fu/upload/?done HTTP/1.1" 400 56 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
[pid: 63|app: 0|req: 324/450] 172.25.0.8 () {72 vars in 1533 bytes} [Sat Feb 19 10:49:01 2022] POST /fu/upload/?done => generated 56 bytes in 10 msecs (HTTP/1.1 400) 6 headers in 184 bytes (1 switches on core 1)
172.25.0.8 - - [19/Feb/2022:10:49:01 +0000] "POST /fu/upload/?done HTTP/1.1" 400 56 "https://mediacms.example.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
[pid: 63|app: 0|req: 325/451] 172.25.0.8 () {66 vars in 1415 bytes} [Sat Feb 19 10:49:56 2022] GET /fu/upload/?done => generated 0 bytes in 7 msecs (HTTP/1.1 405) 7 headers in 211 bytes (1 switches on core 0)
172.25.0.8 - - [19/Feb/2022:10:49:56 +0000] "GET /fu/upload/?done HTTP/1.1" 405 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

To Reproduce Steps to reproduce the issue:

  1. Setup MediaCMS via Docker-Compose installation
  2. Upload File
  3. See error on GUI

Expected behavior Upload works.

Screenshots image

Environment (please complete the following information):

docker-compose.yml

version: "3"

services:
  mediacms_migrations:
    image: mediacms/mediacms:latest
    volumes:
      - ./mediacms/:/home/mediacms.io/mediacms/
    environment:
      ENABLE_UWSGI: 'no'
      ENABLE_NGINX: 'no'
      ENABLE_CELERY_SHORT: 'no'
      ENABLE_CELERY_LONG: 'no'
      ENABLE_CELERY_BEAT: 'no'
      ADMIN_USER: 'admin'
      ADMIN_EMAIL: 'admin@localhost'
      #ADMIN_PASSWORD: 'uncomment_and_set_password_here'
    command: "./deploy/docker/prestart.sh"
    restart: on-failure
    depends_on:
      mediacms_redis:
        condition: service_healthy
      mediacms_db:
        condition: service_healthy
    networks:
      - proxy
  mediacms_web:
    image: mediacms/mediacms:latest
    restart: always
    deploy:
      replicas: 1
    ports:
      - 80
    volumes:
      - ./mediacms/:/home/mediacms.io/mediacms/
    environment:
      ENABLE_CELERY_BEAT: 'no'
      ENABLE_CELERY_SHORT: 'no'
      ENABLE_CELERY_LONG: 'no'
      ENABLE_MIGRATIONS: 'no'
    depends_on:
      - mediacms_migrations
    networks:
      - proxy
    labels:     
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.mediacms.rule=Host(`mediacms.example.com`)"
      - "traefik.http.routers.mediacms.entrypoints=https"
      - "traefik.http.routers.mediacms.service=mediacms"
      - "traefik.http.routers.mediacms.tls=true"
      - "traefik.http.services.mediacms.loadbalancer.server.port=80"
  celery_beat:
    image: mediacms/mediacms:latest
    restart: always
    volumes:
      - ./mediacms/:/home/mediacms.io/mediacms/
    environment:
      ENABLE_UWSGI: 'no'
      ENABLE_NGINX: 'no'
      ENABLE_CELERY_SHORT: 'no'
      ENABLE_CELERY_LONG: 'no'
      ENABLE_MIGRATIONS: 'no'
    depends_on:
      - mediacms_redis
    networks:
      - proxy
  celery_worker:
    image: mediacms/mediacms:latest
    restart: always
    deploy:
      replicas: 1
    volumes:
      - ./mediacms/:/home/mediacms.io/mediacms/
    environment:
      ENABLE_UWSGI: 'no'
      ENABLE_NGINX: 'no'
      ENABLE_CELERY_BEAT: 'no'
      ENABLE_MIGRATIONS: 'no'
    depends_on:
      - mediacms_migrations
    networks:
      - proxy
  mediacms_db:
    image: postgres:13
    volumes:
      - /srv/mediacms/postgres_data:/var/lib/postgresql/data/
    restart: always
    environment:
      POSTGRES_USER: mediacms
      POSTGRES_PASSWORD: mediacms
      POSTGRES_DB: mediacms
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mediacms"]
      interval: 10s
      timeout: 5s
      retries: 5
    #ports:
    #  - "5433:5432"
    networks:
      - proxy
  mediacms_redis:
    image: "redis:alpine"
    restart: always
    healthcheck:
      test: ["CMD", "redis-cli","ping"]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - proxy
networks:
  proxy:
    external: true

local_settings.py

FRONTEND_HOST = 'https://mediacms.example.com'
PORTAL_NAME = 'MediaCMS'
SECRET_KEY = 'xxx'
POSTGRES_HOST = 'mediacms_db'
REDIS_LOCATION = "redis://mediacms_redis:6379/1"

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "mediacms",
        "HOST": POSTGRES_HOST,
        "PORT": "5432",
        "USER": "mediacms",
        "PASSWORD": "mediacms",
    }
}

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": REDIS_LOCATION,
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        },
    }
}

# CELERY STUFF
BROKER_URL = REDIS_LOCATION
CELERY_RESULT_BACKEND = BROKER_URL

MP4HLS_COMMAND = "/home/mediacms.io/bento4/bin/mp4hls"

DEBUG = True

# Flowside Config

REGISTER_ALLOWED = False
PORTAL_WORKFLOW = "private"
UPLOAD_MAX_SIZE = 800 * 1024 * 1000 * 5
tschig commented 2 years ago

Activated Debug Mode and found this in a debug.log:

root@0dc592633793:/home/mediacms.io/mediacms/logs# cat debug.log
Internal Server Error: /fu/upload/
Traceback (most recent call last):
  File "/home/mediacms.io/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 49, in dispatch
    return super(FineUploaderView, self).dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 68, in form_valid
    new = Media.objects.create(media_file=myfile, user=self.request.user)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/mediacms.io/mediacms/files/models.py", line 368, in save
    super(Media, self).save(*args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 801, in save_base
    post_save.send(
  File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 177, in send
    return [
  File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/home/mediacms.io/mediacms/files/models.py", line 1349, in media_save
    instance.media_init()
  File "/home/mediacms.io/mediacms/files/models.py", line 433, in media_init
    self.set_media_type()
  File "/home/mediacms.io/mediacms/files/models.py", line 458, in set_media_type
    ret = helpers.media_file_info(self.media_file.path)
  File "/home/mediacms.io/mediacms/files/helpers.py", line 446, in media_file_info
    stream_size = sum([int(line) for line in stdout.split("\n") if line != ""])
  File "/home/mediacms.io/mediacms/files/helpers.py", line 446, in <listcomp>
    stream_size = sum([int(line) for line in stdout.split("\n") if line != ""])
ValueError: invalid literal for int() with base 10: '3|'
Internal Server Error: /fu/upload/
Traceback (most recent call last):
  File "/home/mediacms.io/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 49, in dispatch
    return super(FineUploaderView, self).dispatch(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/home/mediacms.io/mediacms/uploader/views.py", line 68, in form_valid
    new = Media.objects.create(media_file=myfile, user=self.request.user)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/mediacms.io/mediacms/files/models.py", line 368, in save
    super(Media, self).save(*args, **kwargs)
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 801, in save_base
    post_save.send(
  File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 177, in send
    return [
  File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/home/mediacms.io/mediacms/files/models.py", line 1349, in media_save
    instance.media_init()
  File "/home/mediacms.io/mediacms/files/models.py", line 433, in media_init
    self.set_media_type()
  File "/home/mediacms.io/mediacms/files/models.py", line 458, in set_media_type
    ret = helpers.media_file_info(self.media_file.path)
  File "/home/mediacms.io/mediacms/files/helpers.py", line 446, in media_file_info
    stream_size = sum([int(line) for line in stdout.split("\n") if line != ""])
  File "/home/mediacms.io/mediacms/files/helpers.py", line 446, in <listcomp>
    stream_size = sum([int(line) for line in stdout.split("\n") if line != ""])
ValueError: invalid literal for int() with base 10: '3|'
tschig commented 2 years ago

Adjusted "helpers.py" in "files" folder and added the replacement on line 446.

            stdout = run_command(cmd).get("out")
            stdout = stdout.replace("|","")
            stream_size = sum([int(line) for line in stdout.split("\n") if line != ""])

Upload does now work. Seems to be a problem with the video missing the bit_rate.

Videofile was created by using yt-dlp for fetching test videos:

yt-dlp.exe -f bv+ba/b --merge-output-format mkv YOUTUBE_URL
mgogoulos commented 2 years ago

hi, so this is irrelevant with the way you run (single server vs docker), right?

Do you want to open a PR with your solution, and I can comment there? thanks

KyleMaas commented 5 months ago

Nice find!