search

WWBN / AVideo

Create Your Own Broadcast Network With AVideo Platform Open-Source. OAVP OVP
https://avideo.tube/AVideo_OpenSource
Other
1.92k stars 972 forks source link

With AWS_S3 Plugin, videos still stored on local storage instead of S3 #9484

Closed XDmToter closed 1 month ago

XDmToter commented 1 month ago

I have purchased the AWS_S3 Plugin and configured it. The "Upload Test" button works, and the "test.txt" file that is created in the S3 Bucket correctly loads from the CDN. However, when someone uploads a new video, it is still saved in the "videos" directory on Local storage instead of being uploaded to S3. I don't see anywhere in the system where I can set S3 to be the Default storage location for all new videos. What am I missing?

DanielnetoDotCom commented 1 month ago

can you please go to your server

cd /var/www/html/AVideo/plugin/AWS_S3

then 

php transfer.php

and let me know the results?

DanielnetoDotCom commented 1 month ago

Anyway, I just sent a new AWS_S3 version 4.7, please check if this version will upload the files automatically

To update it download it again from the marketplace site

XDmToter commented 1 month ago

I manually ran "transfer.php". That Copied the video files to the S3 Bucket, but they are still stored on Local storage as well. Also, playing the video on the site uses Local storage instead of the CDN Link. How do I configure this to Move files to S3 instead of Copy? Also, I noticed that the Thumbs were Not copied to S3. Is there a way to move Thumbs to S3 and load from the CDN link as well?

XDmToter commented 1 month ago 
$ /usr/bin/php82 transfer.php 
AVideoLog::DEBUG: S3 Transfer Searching [1/0]  /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Transfer found [1/0]  /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_240.mp4 6.50MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Uploading /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_240.mp4 6817471 6.50MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start video_241004201402_vc122_240.mp4 6.50MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: client OK video_241004201402_vc122_240.mp4 SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start putObject video_241004201402_vc122_240.mp4 6.50MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_240.mp4  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_240.mp4 Transfer: 17.31MBps SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_240.mp4 Takes: 0.01 minutes to complete the transfer SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_240.mp4 File size: 6.50MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: ** S3 Transfer SUCCESS  Uploading Seconds=0.43 15.05/MBps /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_240.mp4 video_241004201402_vc122_240.mp4 DONE  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Transfer found [1/0]  /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_360.mp4 9.44MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Uploading /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_360.mp4 9897931 9.44MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start video_241004201402_vc122_360.mp4 9.44MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: client OK video_241004201402_vc122_360.mp4 SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start putObject video_241004201402_vc122_360.mp4 9.44MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_360.mp4  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_360.mp4 Transfer: 14.40MBps SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_360.mp4 Takes: 0.01 minutes to complete the transfer SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_360.mp4 File size: 9.44MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: ** S3 Transfer SUCCESS  Uploading Seconds=0.66 14.38/MBps /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_360.mp4 video_241004201402_vc122_360.mp4 DONE  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Transfer found [1/0]  /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_480.mp4 12.58MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Uploading /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_480.mp4 13189168 12.58MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start video_241004201402_vc122_480.mp4 12.58MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: client OK video_241004201402_vc122_480.mp4 SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start putObject video_241004201402_vc122_480.mp4 12.58MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_480.mp4  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_480.mp4 Transfer: 45.63MBps SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_480.mp4 Takes: 0.00 minutes to complete the transfer SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_480.mp4 File size: 12.58MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: ** S3 Transfer SUCCESS  Uploading Seconds=0.28 45.46/MBps /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_480.mp4 video_241004201402_vc122_480.mp4 DONE  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Transfer found [1/0]  /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_540.mp4 15.85MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Uploading /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_540.mp4 16618820 15.85MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start video_241004201402_vc122_540.mp4 15.85MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: client OK video_241004201402_vc122_540.mp4 SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start putObject video_241004201402_vc122_540.mp4 15.85MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_540.mp4  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_540.mp4 Transfer: 45.94MBps SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_540.mp4 Takes: 0.01 minutes to complete the transfer SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_540.mp4 File size: 15.85MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: ** S3 Transfer SUCCESS  Uploading Seconds=0.35 45.80/MBps /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_540.mp4 video_241004201402_vc122_540.mp4 DONE  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Transfer found [1/0]  /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_720.mp4 30.03MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: S3 Uploading /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_720.mp4 31491166 30.03MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start video_241004201402_vc122_720.mp4 30.03MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: client OK video_241004201402_vc122_720.mp4 SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: start putObject video_241004201402_vc122_720.mp4 30.03MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_720.mp4  SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_720.mp4 Transfer: 36.39MBps SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_720.mp4 Takes: 0.01 minutes to complete the transfer SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: copy_to_s3: done putObject video_241004201402_vc122_720.mp4 File size: 30.03MB SCRIPT_NAME: transfer.php
AVideoLog::DEBUG: ** S3 Transfer SUCCESS  Uploading Seconds=0.83 36.34/MBps /home/httpd/html/demo.example.com/public_html/videos/video_241004201402_vc122/video_241004201402_vc122_720.mp4 video_241004201402_vc122_720.mp4 DONE  SCRIPT_NAME: transfer.php
# rclone ls endpoint:bucket
      194 test.txt
  6817471 video_241004201402_vc122_240.mp4
  9897931 video_241004201402_vc122_360.mp4
 13189168 video_241004201402_vc122_480.mp4
 16618820 video_241004201402_vc122_540.mp4
 31491166 video_241004201402_vc122_720.mp4
[root@server video_241004201402_vc122]# ls -al
total 79788
drwxr-xr-x+ 2 apache        apache     4096 Oct  4 16:22 .
drwxrwxr-x+ 6 apache        apache     4096 Oct 10 13:47 ..
-rw-rw-rw-  1 apache        apache    12209 Oct  4 16:14 video_241004201402_vc122_128.jpg
-rw-rw-rw-  1 apache        apache    24486 Oct  4 16:14 video_241004201402_vc122_192.jpg
-rw-r--r--  1 apache        apache  6817471 Oct  4 16:14 video_241004201402_vc122_240.mp4
-rw-rw-rw-  1 apache        apache    38745 Oct  4 16:14 video_241004201402_vc122_256.jpg
-rw-r--r--  1 apache        apache  9897931 Oct  4 16:14 video_241004201402_vc122_360.mp4
-rw-rw-rw-  1 apache        apache    75928 Oct  4 16:14 video_241004201402_vc122_384.jpg
-rw-r--r--  1 apache        apache 13189168 Oct  4 16:14 video_241004201402_vc122_480.mp4
-rw-rw-rw-  1 apache        apache   121930 Oct  4 16:14 video_241004201402_vc122_512.jpg
-rw-r--r--  1 apache        apache 16618820 Oct  4 16:14 video_241004201402_vc122_540.mp4
-rw-r--r--  1 apache        apache 31491166 Oct  4 16:14 video_241004201402_vc122_720.mp4
-rw-rw-rw-  1 apache        apache     7839 Oct  4 16:14 video_241004201402_vc122_96.jpg
-rw-rw-r--  1 apache        apache   989452 Oct  4 16:14 video_241004201402_vc122.gif
-rw-rw-r--  1 apache        apache    62488 Oct  4 16:14 video_241004201402_vc122.jpg
-rw-rw-rw-  1 apache        apache   178389 Oct  4 16:14 video_241004201402_vc122_thumbsV2.jpg
-rw-rw-r--  1 apache        apache    42498 Oct  4 16:22 video_241004201402_vc122_thumbsV2_jpg.webp
-rw-rw-r--  1 apache        apache  2095341 Oct  4 16:14 video_241004201402_vc122.webp
DanielnetoDotCom commented 1 month ago

Images will not be copied, only videos

I am surprised that the videos were not removed locally, they should transform into a 10bytes file (Dummy file)

I am sending another update (version 4.8) to try to detect what is missing , if you can please update, try the transfer.php again and check if it replace the original video with a 10 bytes file

XDmToter commented 1 month ago

We downloaded V5.0 of the AWS_S3 plugin. This appears to be working correctly. After running transfer.php the videos on Local Storage were replaced with a 10Kb file as you said, and the site is loading them from the CDN now. I was also able to upload a new video through the Encoder, and that was automatically transferred to S3 as well.

Thank You!

DanielnetoDotCom commented 1 month ago

I am glad to hear it, thanks for the feed back