search

WWBN / AVideo

Create Your Own Broadcast Network With AVideo Platform Open-Source. OAVP OVP
https://avideo.tube/AVideo_OpenSource
Other
1.9k stars 972 forks source link

Encoder videos not saving to storage #8152

Open para2022 opened 1 year ago

para2022 commented 1 year ago

Hi i have been beating my head on this for the past day. Here is what happens.

  1. My encoder is on the same server as the storage. Streamer is on its own server.
  2. If i upload a file on a test encoder on the streamer site the video plays fine and is moved into storage straight away (no problems)
  3. If i upload to the remote encoder the file encodes with no errors. It moves the file and if i quickly refresh the 'my videos' page i see the file went to storage but when i refresh it again its not in storage it is in local storage. At that point the videos does not play at all and there are no errors in the log. The videos are not in storage and the local video folder shows the images and 0byte video files.
  4. This happens over and over again on any upload.
  5. Direct uploads go straight to storage.
  6. local encoded files on the same streamer site go straight to storage.
  7. I built another encoder on another server that is not on the streamer site or storage site to see if it was a problem on the storage server. Same problem.

Also ive noticed the script isn't moving files from local to storage anymore. It runs without error but doesn't move anything.

I have this error only in my log during the move process.:

YPTStorage getFileInfo ERROR empty siteURL video_230213120102_v3b1a, , SCRIPT_NAME: /view/modeYoutube.php

So i don't beleive this is a problem my end. Any pointers?

Thanks

para2022 commented 1 year ago

I only have this in my storage apache log.

[04-Jul-2023 13:01:09 Europe/london] Storage xsendfile https://ourtube.co.uk/plugin/YPTStorage/canWatchVideo.json.php?token=0&filename=video_230119122840_vfbd3 => {"authorization":true,"msg":"SecureVideosDirectory NOT enabled"} [04-Jul-2023 13:01:33 Europe/london] Storage xsendfile https://ourtube.co.uk/plugin/YPTStorage/canWatchVideo.json.php?token=0&filename=video_230422110032_v7815_360 => {"authorization":true,"msg":"SecureVideosDirectory NOT enabled"} [04-Jul-2023 13:01:38 Europe/london] Storage xsendfile https://ourtube.co.uk/plugin/YPTStorage/canWatchVideo.json.php?token=0&filename=video_230422110032_v7815_240 => {"authorization":true,"msg":"SecureVideosDirectory NOT enabled"} [04-Jul-2023 13:01:46 Europe/london] Storage xsendfile https://ourtube.co.uk/plugin/YPTStorage/canWatchVideo.json.php?token=0&filename=DNMwfn88yepMiklIC3Ug_240 => {"authorization":true,"msg":"SecureVideosDirectory NOT enabled"} [04-Jul-2023 13:02:03 Europe/london] Storage xsendfile https://ourtube.co.uk/plugin/YPTStorage/canWatchVideo.json.php?token=0&filename=video_220713115331_ve178_360 => {"authorization":true,"msg":"SecureVideosDirectory NOT enabled"} [04-Jul-2023 13:02:09 Europe/london] Storage xsendfile https://ourtube.co.uk/plugin/YPTStorage/canWatchVideo.json.php?token=0&filename=video_230422110032_v7815_720 => {"authorization":true,"msg":"SecureVideosDirectory NOT enabled"}

para2022 commented 1 year ago

Ive just got up and tested it again and it worked the remote encoder moved it straight into storage. And the following isnt there on reading the transfer data from avdieo.log

YPTStorage getFileInfo ERROR empty siteURL video_230213120102_v3b1a, , SCRIPT_NAME: /view/modeYoutube.php

So its something to do with that error causing this im guessing. Any clues as to whats going on?

Again there are no errors in avideo.log (it is very clean) same with php and apache logs

para2022 commented 1 year ago

The shell script is working properly again again now so Ignore that.

DanielnetoDotCom commented 1 year ago

My encoder is on the same server as the storage. Streamer is on its own server.

that is the best configuration

If i upload to the remote encoder the file encodes with no errors. It moves the file and if i quickly refresh the 'my videos' page i see the file went to storage but when i refresh it again its not in storage it is in local storage. At that point the videos does not play at all and there are no errors in the log. The videos are not in storage and the local video folder shows the images and 0byte video files.

this is what we have to investigate

I built another encoder on another server that is not on the streamer site or storage site to see if it was a problem on the storage server. Same problem.

The encoder transfers the file to the streamer, so the encoder job is complete, we need only to investigate the streamer and the storage.

[04-Jul-2023 13:01:09 Europe/london] Storage xsendfile https://ourtube.co.uk/plugin/YPTStorage/canWatchVideo.json.php?token=0&filename=video_230119122840_vfbd3 => {"authorization":true,"msg":"SecureVideosDirectory NOT enabled"}

as long as authorization = true, this should not be a problem.

Ive just got up and tested it again and it worked the remote encoder moved it straight into storage. And the following isnt there on reading the transfer data from avdieo.log

I cannot see a connection that works with a remote encoder but does not work with yours. the only reason I can see is that your encoder is not transferring to the streamer properly, so it will not be an issue with the storage.

YPTStorage getFileInfo ERROR empty siteURL video_230213120102_v3b1a, , SCRIPT_NAME: /view/modeYoutube.php

If the encoder error above happens, this storage error makes sense.

DanielnetoDotCom commented 1 year ago

Let's do one thing at a time.

First, make sure your encoder is transferring the file fine to your streamer.

  1. disable the YPTStorage plugin
  2. Encode a small file so it will encode fast
  3. After finishing save the log so we can compare (first.log)
  4. check if the file is fine in the streamer.

If the above fails

  1. Share the first.log

If the above works try this

  1. enable the YPTStorage plugin
  2. Encode the same small file as you did above
  3. After finishing save the log so we can compare it with the previews log (second.log)
  4. it may take longer than the above, because it will transfer to the YPTStorage
  5. Get the YPTStorage Apache error log (storage.log)
  6. Share the logs (first.log, second.log, storage.log)
para2022 commented 1 year ago

Thank you. Here is the log. YPTStorage plugin was turned off on the streamer. Log was cleared. The video plays on the streamer and goes to /videos folder as expected.

https://ourtube.co.uk/logs/first_test_plugin_off.txt

DanielnetoDotCom commented 1 year ago

image

para2022 commented 1 year ago

Im sharing the logs separate and naming them. I cant post logs that long on here as it wont let me. Anyway that was the log for the storage turned off i will send more.

DanielnetoDotCom commented 1 year ago

please share them separately, one log per file, also let me know what was the video filename and video id that was processed.

para2022 commented 1 year ago

Second test is plugin turned back on. Result is that the video is transferred but doesn't play.

https://ourtube.co.uk/logs/second_test_YTPStorage_plugin_on.txt

para2022 commented 1 year ago

Video name:

Alive (ASOT 1037)

ID:

47680

para2022 commented 1 year ago

Last file. Server log

https://ourtube.co.uk/logs/server.log

para2022 commented 1 year ago

Screenshot 2023-07-04 at 15 37 37

para2022 commented 1 year ago

Their is one video i also try and it is intermittent. It goes through 90% of the time and plays and goes in storage. Some will never ever work and others are intermittent and will sometimes work and go in storage but then i try it again and it fails. Its really weird, but the bugs i have usually always are. Never straight forward lol. Hope this all helps.

DanielnetoDotCom commented 1 year ago

can you please make this command in your storage server?

ls -lah /home/storage1/public_html/storage1/videos/video_230704152311_vd470_480.mp4

para2022 commented 1 year ago

ls /home/storage1/public_html/storage1/videos/video_230704152311_vd470_480.mp4 /home/storage1/public_html/storage1/videos/video_230704152311_vd470_480.mp4

DanielnetoDotCom commented 1 year ago

image

para2022 commented 1 year ago

ls -lah /home/storage1/public_html/storage1/videos/video_230704152311_vd470_480.mp4 -rw-r--r-- 1 storage1 storage1 10 Jul 4 15:23 /home/storage1/public_html/storage1/videos/video_230704152311_vd470_480.mp4

para2022 commented 1 year ago

At the moment the plugin is set to default local storage and all uploads play fine when uploaded or encoded. I can manually move them with the web plugin and they move successfully and also play fine afterwards. That is my workaround for now.

DanielnetoDotCom commented 1 year ago

can you please update your storage server with a git pull and try again?

para2022 commented 1 year ago

I have done that but its still the same. The video wont play.

DanielnetoDotCom commented 1 year ago

Can you please send me the logs again?

para2022 commented 1 year ago

Yes but users keep uploading to the encoders and switching the storage to remote I dont want to take the risk of breaking their uploads so i will report back when i can do this.

para2022 commented 1 year ago

https://ourtube.co.uk/logs/another_test_1.log

DanielnetoDotCom commented 1 year ago

can you please also share the apache storage log?

para2022 commented 1 year ago

Nothing other than a page of this which is always in my logs flooding them

[04-Jul-2023 22:19:57 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:20:37 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:24:35 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:26:03 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:26:36 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:28:21 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:28:29 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:28:43 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:29:11 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:29:36 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:29:52 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:30:26 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:30:38 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:31:21 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:33:31 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:34:15 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:34:28 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:35:30 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:35:35 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php [04-Jul-2023 22:35:52 Europe/London] /home/ourtube/public_html/ourtube/view/../plugin/VideoOffline/offlineVideo.php

DanielnetoDotCom commented 1 year ago

without log I cannot help you

I need your storage logs in the time 04-Jul-2023 21:46, 21:47, 21:48

DanielnetoDotCom commented 1 year ago

also, where did you get the VideoOffline plugin from?

para2022 commented 1 year ago

sorry my bad. Those errors are always in my streamer log for as long as a can remember. separate issue.

https://ourtube.co.uk/logs/server2.log

thats the storage server log

DanielnetoDotCom commented 1 year ago

the log you sent is in the wrong time

image

para2022 commented 1 year ago

What do you mean the wrong time?

I have no errors in my apache log to show. It is clean. The log i sent is from my php.log where there is relevant info for the storage site. As far as i can tell my clock is correct on both servers.

DanielnetoDotCom commented 1 year ago

the logs you sent happen 1 hour after then what I need

image

I need to compare logs, so all the logs must be from the same time

para2022 commented 1 year ago

Both servers have the same time:

date

Wed Jul 5 01:50:25 PM BST 2023

date

Wed 5 Jul 13:50:35 BST 2023

DanielnetoDotCom commented 1 year ago

but the logs you shared do not have the same time,

again, YOU MUST SHARE THE LOGS AT THE SAME TIME AND THE ONES YOU SHARED HAVE 1 HOUR DIFFERENCE

para2022 commented 1 year ago

Well i have both timezones set the same and in the avideo advanced plugin i have europe/london set in both and in php.ini so im not sure what else to do if they are the same.

timedatectl Local time: Wed 2023-07-05 14:00:07 BST Universal time: Wed 2023-07-05 13:00:07 UTC RTC time: Wed 2023-07-05 13:00:07 Time zone: Europe/London (BST, +0100) System clock synchronized: yes NTP service: active RTC in local TZ: no

timedatectl Local time: Wed 2023-07-05 13:59:51 BST Universal time: Wed 2023-07-05 12:59:51 UTC RTC time: Wed 2023-07-05 12:59:51 Time zone: Europe/London (BST, +0100) System clock synchronized: yes NTP service: n/a RTC in local TZ: no

DanielnetoDotCom commented 1 year ago

I am NOT saying your servers are in a different time

I am saying that the logs you shared with me are from 1-hour difference, you need to share the logs at the same time.

if you still do not understand what I mean, check the logs, you have shared, by yourself. Check the streamer and storage logs

para2022 commented 1 year ago

I can assure you that was the storage server log sent after the upload was done. I did send the wrong one to start of the streamer then i sent the storage server php.log as the apache_error.log shows nothing.

para2022 commented 1 year ago

I will start a new php.log and do it again if there is a time error then i have no idea what is causing a time difference.

DanielnetoDotCom commented 1 year ago

on, send me a new log and make sure the time is the same in all of them, also send me the log when you encode a video

para2022 commented 1 year ago

https://ourtube.co.uk/logs/newavideo.log (avideo log)

https://ourtube.co.uk/logs/new_server.log (php.log)

para2022 commented 1 year ago

Its doesnt play and its called Test 1 and its ID is 47770 and its source files are:

https://ourtube.co.uk/videos/video_230705143817_v44d6/video_230705143817_v44d6_240.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJ2aWRlb18yMzA3MDUxNDM4MTdfdjQ0ZDYiLCJ0aW1lIjoxNjg4NTY0NzI5fQ== 480 https://ourtube.co.uk/videos/video_230705143817_v44d6/video_230705143817_v44d6_480.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJ2aWRlb18yMzA3MDUxNDM4MTdfdjQ0ZDYiLCJ0aW1lIjoxNjg4NTY0NzI5fQ== 720 https://ourtube.co.uk/videos/video_230705143817_v44d6/video_230705143817_v44d6_720.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJ2aWRlb18yMzA3MDUxNDM4MTdfdjQ0ZDYiLCJ0aW1lIjoxNjg4NTY0NzI5fQ==

DanielnetoDotCom commented 1 year ago

ok, and what about the file in your storage server, does it exists?

ls -lah /home/storage1/public_html/storage1/videos/video_230705143817_v44d6_480.mp4

para2022 commented 1 year ago

ls -lah /home/storage1/public_html/storage1/videos/video_230705143817_v44d6_480.mp4 -rw-r--r-- 1 storage1 storage1 23M Jul 5 14:40 /home/storage1/public_html/storage1/videos/video_230705143817_v44d6_480.mp4

DanielnetoDotCom commented 1 year ago

great, your storage file is fine.

so it is transferring perfectly. now we need to find out why when you try to play it is not loading the storage file. can you clear your cache and try again?

if fails share a video link where it supposes to play

para2022 commented 1 year ago

Ive cleared cache it makes no difference.

https://ourtube.co.uk/v/47770?channelName=OurTube

para2022 commented 1 year ago

Again if i set the storage to local they encode and play fine. I can then also move them to storage using the plugin and they play still but will not go into storage automatically if i set the plugin to storage. They will auto go into storage with direct upload though so its only affecting remote encoder as the problem doesn't exist if streamer/encoder on same server.

DanielnetoDotCom commented 1 year ago

FYI the video in your storage is fine:

https://storage1.ourtube.co.uk/videos/video_230705143817_v44d6/video_230705143817_v44d6_480.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJ2aWRlb18yMzA3MDUxNDM4MTdfdjQ0ZDZfNDgwIiwidGltZSI6MTY4ODU2NzkwNH0=

para2022 commented 1 year ago

https://ourtube.co.uk/v/20567?channelName=MusicMonster

Source Files

MP4

240 https://storage1.ourtube.co.uk/videos/Gq4yVyfwZkYj7DKpRP9R/Gq4yVyfwZkYj7DKpRP9R_240.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJHcTR5Vnlmd1prWWo3REtwUlA5UiIsInRpbWUiOjE2ODg1NjgxMzh9 360 https://storage1.ourtube.co.uk/videos/Gq4yVyfwZkYj7DKpRP9R/Gq4yVyfwZkYj7DKpRP9R_360.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJHcTR5Vnlmd1prWWo3REtwUlA5UiIsInRpbWUiOjE2ODg1NjgxMzh9 480 https://storage1.ourtube.co.uk/videos/Gq4yVyfwZkYj7DKpRP9R/Gq4yVyfwZkYj7DKpRP9R_480.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJHcTR5Vnlmd1prWWo3REtwUlA5UiIsInRpbWUiOjE2ODg1NjgxMzh9

MP3

https://ourtube.co.uk/videos/Gq4yVyfwZkYj7DKpRP9R/Gq4yVyfwZkYj7DKpRP9R.mp3?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJHcTR5Vnlmd1prWWo3REtwUlA5UiIsInRpbWUiOjE2ODg1NjgxMzh9

para2022 commented 1 year ago

The above was bulk moved using the script a few days ago.

para2022 commented 1 year ago

Notice the source files on the broken one are not in storage?

240 https://ourtube.co.uk/videos/video_230705143817_v44d6/video_230705143817_v44d6_240.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJ2aWRlb18yMzA3MDUxNDM4MTdfdjQ0ZDYiLCJ0aW1lIjoxNjg4NTY4NDMzfQ== 480 https://ourtube.co.uk/videos/video_230705143817_v44d6/video_230705143817_v44d6_480.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJ2aWRlb18yMzA3MDUxNDM4MTdfdjQ0ZDYiLCJ0aW1lIjoxNjg4NTY4NDMzfQ== 720 https://ourtube.co.uk/videos/video_230705143817_v44d6/video_230705143817_v44d6_720.mp4?token=eyJzYWx0IjoiaHR0cHM6XC9cL291cnR1YmUuY28udWtcLyIsInRva2VuVGltZU91dCI6IjIxNjAwIiwiZmlsZW5hbWUiOiJ2aWRlb18yMzA3MDUxNDM4MTdfdjQ0ZDYiLCJ0aW1lIjoxNjg4NTY4NDMzfQ==