Broken missing files from encoder

[para2022]

Hi,

My streamer site keeps getting broken missing files from the encoders. It happens with both encoders.

When i check the files are there with the correct permissions and owner. I have downloaded the files and they play fine.

I have looked in the database and the entries are how they should be but set to (b) broken. I set it to (a) and it still fails. So the question is what the hell is causing this? Why will they not set back to active if everything is there and how it should be?

No errors in the log but i can see it receives the file and everything looks good but it sets the file as broken for no apparent reason. Some are successful but very few as most are failing now. Cant get my head around it.

This doesn't happen with direct upload.

Used another server and it happens on that one too.

Thanks

[para2022]

It is not producing all the files. See below, the same file, one is successful with lots of files and the one saying broken is missing some files that the encoder did not produce. is this causing it?

All the broken vids are missing thumbs ive just gone through them.

[para2022]

This happens locally as well as ive installed an encoder on the streamer site for testing, still happens. The encoder servers fail locally also by using the encoder on the same site. So this is not a remote issue is seems the encoder code is broken. This cannot be a problem my end on all 3 servers. One was freshly reinstalled today to see if it was a server issue. Brand new install of everything and still broken files on the new server.

[para2022]

I see these lines in the log on the failed ones EE:EE:EE. These are uploads to the encoder not imports.

[14-May-2023 01:34:07 UTC] AVideoLog::DEBUG: Video::save (7-1 DTS HD MA (dtshd) surround pink noise test files for each speaker calibration 4k) Saved id = 32 EE:EE:EE [14-May-2023 01:34:07 UTC] AVideoLog::DEBUG: setDuration after EE:EE:EE SCRIPT_NAME: /myMySQLDatabase/objects/aVideoEncoder.json.php [14-May-2023 01:34:07 UTC] AVideoLog::DEBUG: setDuration before EE:EE:EE SCRIPT_NAME: /myMySQLDatabase/objects/aVideoEncoder.json.php

[DanielnetoDotCom]

the EE:EE:EE is because of the missing files, you MUST check the logs, during the encoder process and transfer process.

both logs, encoder and streamer

[para2022]

I have checked already. No errors in encoder log . It does its thing and sends the file without error. The above is what i find in the streamer log.

ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

[para2022]

Here is my encoder log. As you can see it ends successfully with no errors.

[18-May-2023 20:44:00 UTC] sendToStreamer https://encoder2.ourtube.co.uk/ourtube/aVideoEncoder.json in 1.360 seconds {"error":false,"target":"aVideoEncoder.json","postFields":{"duration":"0:09:17","title":"It\'s down to us","categories_id":0,"format":"mp4","resolution":"240","videoDownloadedLink":"","description":"","downloadURL":"https:\/\/encoder2.ourtube.co.uk\/2\/videos\/avideoTmpFile_69_streamers_id_15_240.mp4","encoderURL":"https:\/\/encoder2.ourtube.co.uk\/2\/","streamers_id":"15","user":"ourtube","pass":"_user_hash_dWxnRGxQeFNRWG9rRnRtQituZERrcWdCclIvVmxCK0p1VlNadjBPZVV2UXlyKzV2cTM5aXNVUFFsM2ZzQXUrSXdvQlRod24yb0pDaEFveWo0WlhHYlE9PQ==","return_vars":"{\"videos_id\":45696,\"video_id_hash\":\"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=\",\"releaseDate\":\"2023-05-18 19:39:00\"}","releaseDate":"2023-05-18 19:39:00","videos_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","timezone":"UTC"},"response_raw":"{\"error\":false,\"video_id\":45696,\"video_id_hash\":\"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=\",\"releaseDate\":\"2023-05-18 19:39:00\"}","response":{"error":false,"video_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","releaseDate":"2023-05-18 19:39:00"}} [18-May-2023 20:44:00 UTC] Encoder:sendFileChunk no need, we could download [18-May-2023 20:44:13 UTC] Duration found: 0:09:17 [18-May-2023 20:44:13 UTC] sendToStreamer to aVideoEncoder.json [18-May-2023 20:44:14 UTC] Encoder::save id=(69) title=(It\'s down to us) [18-May-2023 20:44:14 UTC] Encoder::save id=(69) title=(It\'s down to us) [18-May-2023 20:44:14 UTC] sendToStreamer spectrumimage not set [18-May-2023 20:44:14 UTC] sendToStreamer rawVideo not set [18-May-2023 20:44:14 UTC] sendToStreamer image not set [18-May-2023 20:44:14 UTC] sendToStreamer gifimage not set [18-May-2023 20:44:14 UTC] sendToStreamer webpimage not set [18-May-2023 20:44:14 UTC] sendToStreamer video not set [18-May-2023 20:44:14 UTC] sendToStreamer https://encoder2.ourtube.co.uk/ourtube/aVideoEncoder.json in 1.472 seconds {"error":false,"target":"aVideoEncoder.json","postFields":{"duration":"0:09:17","title":"It\'s down to us","categories_id":0,"format":"mp4","resolution":"480","videoDownloadedLink":"","description":"","downloadURL":"https:\/\/encoder2.ourtube.co.uk\/2\/videos\/avideoTmpFile_69_streamers_id_15_480.mp4","encoderURL":"https:\/\/encoder2.ourtube.co.uk\/2\/","streamers_id":"15","user":"admin","pass":"_user_hash_dWxnRGxQeFNRWG9rRnRtQituZERrcWdCclIvVmxCK0p1VlNadjBPZVV2UXlyKzV2cTM5aXNVUFFsM2ZzQXUrSXdvQlRod24yb0pDaEFveWo0WlhHYlE9PQ==","return_vars":"{\"videos_id\":45696,\"video_id_hash\":\"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=\",\"releaseDate\":\"2023-05-18 19:39:00\"}","releaseDate":"2023-05-18 19:39:00","videos_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","timezone":"UTC"},"response_raw":"{\"error\":false,\"video_id\":45696,\"video_id_hash\":\"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=\",\"releaseDate\":\"2023-05-18 19:39:00\"}","response":{"error":false,"video_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","releaseDate":"2023-05-18 19:39:00"}} [18-May-2023 20:44:14 UTC] Encoder:sendFileChunk no need, we could download [18-May-2023 20:44:36 UTC] Duration found: 0:09:17 [18-May-2023 20:44:36 UTC] sendToStreamer to aVideoEncoder.json [18-May-2023 20:44:38 UTC] Encoder::save id=(69) title=(It\'s down to us) [18-May-2023 20:44:38 UTC] Encoder::save id=(69) title=(It\'s down to us) [18-May-2023 20:44:38 UTC] sendToStreamer spectrumimage not set [18-May-2023 20:44:38 UTC] sendToStreamer rawVideo not set [18-May-2023 20:44:38 UTC] sendToStreamer image not set [18-May-2023 20:44:38 UTC] sendToStreamer gifimage not set [18-May-2023 20:44:38 UTC] sendToStreamer webpimage not set [18-May-2023 20:44:38 UTC] sendToStreamer video not set [18-May-2023 20:44:38 UTC] sendToStreamer https://encoder2.ourtube.co.uk/ourtube/aVideoEncoder.json in 2.141 seconds {"error":false,"target":"aVideoEncoder.json","postFields":{"duration":"0:09:17","title":"It\'s down to us","categories_id":0,"format":"mp4","resolution":"720","videoDownloadedLink":"","description":"","downloadURL":"https:\/\/encoder2.ourtube.co.uk\/2\/videos\/avideoTmpFile_69_streamers_id_15_720.mp4","encoderURL":"https:\/\/encoder2.ourtube.co.uk\/2\/","streamers_id":"15","user":"admin","pass":"_user_hash_dWxnRGxQeFNRWG9rRnRtQituZERrcWdCclIvVmxCK0p1VlNadjBPZVV2UXlyKzV2cTM5aXNVUFFsM2ZzQXUrSXdvQlRod24yb0pDaEFveWo0WlhHYlE9PQ==","return_vars":"{\"videos_id\":45696,\"video_id_hash\":\"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=\",\"releaseDate\":\"2023-05-18 19:39:00\"}","releaseDate":"2023-05-18 19:39:00","videos_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","timezone":"UTC"},"response_raw":"{\"error\":false,\"video_id\":45696,\"video_id_hash\":\"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=\",\"releaseDate\":\"2023-05-18 19:39:00\"}","response":{"error":false,"video_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","releaseDate":"2023-05-18 19:39:00"}} [18-May-2023 20:44:38 UTC] Encoder:sendFileChunk no need, we could download [18-May-2023 20:44:38 UTC] Encoder::save id=(69) title=(It\'s down to us) [18-May-2023 20:44:38 UTC] getReturn_vars {"videos_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","releaseDate":"2023-05-18 19:39:00"} [18-May-2023 20:44:38 UTC] AVideo-Encoder sending confirmation to objects/aVideoEncoderNotifyIsDone.json.php [18-May-2023 20:44:38 UTC] sendToStreamer to objects/aVideoEncoderNotifyIsDone.json.php [18-May-2023 20:44:39 UTC] Encoder::save id=(69) title=(It\'s down to us) [18-May-2023 20:44:39 UTC] sendToStreamer spectrumimage not set [18-May-2023 20:44:39 UTC] sendToStreamer rawVideo not set [18-May-2023 20:44:39 UTC] sendToStreamer image not set [18-May-2023 20:44:39 UTC] sendToStreamer gifimage not set [18-May-2023 20:44:39 UTC] sendToStreamer webpimage not set [18-May-2023 20:44:39 UTC] sendToStreamer video not set [18-May-2023 20:44:39 UTC] sendToStreamer https://encoder2.ourtube.co.uk/ourtube/objects/aVideoEncoderNotifyIsDone.json.php in 0.749 seconds {"error":false,"target":"objects\/aVideoEncoderNotifyIsDone.json.php","postFields":{"fail":0,"streamers_id":"15","user":"admin","pass":"_user_hash_dWxnRGxQeFNRWG9rRnRtQituZERrcWdCclIvVmxCK0p1VlNadjBPZVV2UXlyKzV2cTM5aXNVUFFsM2ZzQXUrSXdvQlRod24yb0pDaEFveWo0WlhHYlE9PQ==","return_vars":"{\"videos_id\":45696,\"video_id_hash\":\"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=\",\"releaseDate\":\"2023-05-18 19:39:00\"}","releaseDate":"2023-05-18 19:39:00","videos_id":45696,"video_id_hash":"aXZETnRtRjlCTld5RTBhZ3I4YW92ZGFpaTArWDRnT2FHMWZXQWtxQUMyVT0=","timezone":"UTC"},"response_raw":"{\"error\":false,\"video_id\":45696}","response":{"error":false,"video_id":45696}} [18-May-2023 20:44:39 UTC] Encoder::save id=(69) title=(It\'s down to us)

[para2022]

The above was an upload, not import and streamer and encoder is on same site.

[mohmafia]

the encoder is horrible at times it starts 2 main processes on same file and results in corrupted video's and goes in a loop Especially with files over 1GB it has serious problems ... plus that it has to encode the whole file once again for the error report it takes ages to get a video on the streamer which is absolutely not needed .. i stopped using Avideo completely went to jellyfin as that does what i need it to do and faster without any errors as i am the only one to upload on my video site jellyfin works perfect .... but lately Avideo the streamer and the encoders are terrible this script/project has gone down the drain if you ask me . my major problem i have with it is that it needs root permissions and is totally not acceptable !!!

[para2022]

I have just installed a fresh install using an OVH server 32GB ram and xeon.

Followed this to the letter and the other install guide for Avideo and encoder https://github.com/WWBN/AVideo/wiki/How-to-install-LAMP,-FFMPEG-and-Git-on-a-fresh-Ubuntu-22.x-for-AVideo-Platform-version-11.x-or-newer

Im using mod_php 8.1 and apache.

Broken files on the local encoder again so im sorry to say this but it is clearly broken. So how can 3 new server builds all result with the same identical problem? This worked fine before a few weeks ago.

[DanielnetoDotCom]

@para2022

• Does this happen with any file or only some files?
• check errors in your streamer logs
• Try to use the public encoder https://encoder1.wwbn.net/
• Share the file with problems so I ca test

[para2022]

These files work fine that is why i use them cause i know them to have worked on the site. They are files that were uploaded before the problems started. They are on the site working now so it is not the files that are a problem. I will post logs

[DanielnetoDotCom]

I mean to be able to help you I need to reproduce the issue. I cannot see it in my encoders , that is why I want you to test in one of my encoders

[para2022]

I do not have a login. Also just because things work on your server doesn't mean this is not an issue for others, which is not an ideal situation. All i know is this problem only reared its ugly head recently as i never had this issue before. Id have thought building a whole new system would eradicate the problem but it does not no matter what i do and the issue is random. Let me give you an example. I will upload the same file 5 files and it will be random how many work. The image attached all those uploads worked then within half hour they are now broken, so why??

[para2022]

Here is the streamer log for the neil oliver that failed:

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: created changed in table=videos id=3 created=2023-05-19 00:42:05 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: setDuration after 0:08:51 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: setDuration before 0:08:51 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: ReceiveImage: duration Line 179 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: ReceiveImage: duration NOT empty 0:08:51 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: ReceiveImage: download https://www.mysite.com/1/videos/3_tmpFile.mp4.webp 1147416 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: ReceiveImage: download https://www.mysite.com/1/videos/3_tmpFile.mp4.gif 413452 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: ReceiveImage: download https://www.mysite.com/1/videos/3_tmpFile.mp4.jpg to path/to/my/myMySQLDatabase/site/videos/video_230519004205_v492a/video_230519004205_v492a.jpg 99.25KB SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: ReceiveImage: videoFilename = [video_230519004205_v492a] destination_local = path/to/my/myMySQLDatabase/site/videos/video_230519004205_v492a/video_230519004205_v492a Encoder receiving post {"image":{"name":"3tmpFile.mp4.jpg","full_path":"\/var\/www\/html\/myMySQLDatabase\/1\/videos\/3_tmpFile.mp4.jpg","type":"application\/octet-stream","tmp_name":"\/tmp\/php9ntMiX","error":0,"size":101637},"gifimage":{"name":"3tmpFile.mp4.gif","full_path":"\/var\/www\/html\/myMySQLDatabase\/1\/videos\/3_tmpFile.mp4.gif","type":"application\/octet-stream","tmp_name":"\/tmp\/php3eoSVR","error":0,"size":413452},"webpimage":{"name":"3tmpFile.mp4.webp","full_path":"\/var\/www\/html\/myMySQLDatabase\/1\/videos\/3_tmpFile.mp4.webp","type":"application\/octet-stream","tmp_name":"\/tmp\/phpVNrKzC","error":0,"size":1147416}} SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

**** **** ***** * * ***** * ** ** **** **** *** - pass

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: useVideoHashOrLogin: 1, 3 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: GET: [] SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

**** **** ***** * ***** ** **** **** * - pass

**** **** * **** *** ** **** **** *** - pass

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: User:login finish users_id=1 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 00:45:13 UTC] AVideoLog::DEBUG: useVideoHashOrLogin: 1, 3 SCRIPT_NAME: /objects/aVideoEncoderReceiveImage.json.php

[19-May-2023 01:45:09 Europe/London] AVideoLog::DEBUG: 1 Image not found for videos/video_230519004234_vc48a/video_230519004234_vc48a_thumbsV2_jpg.webp converting path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/video_230519004234_vc48a.jpg,path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/video_230519004234_vc48a_thumbsV2_jpg.webp SCRIPT_NAME: /view/img/image404.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: clearCache end in 0.00067496299743652 seconds SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: clearCache starts SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video::setStatus(a) AVideoPlugin::onNewVideo SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video::setStatus(a) [{"file":"\/var\/www\/html\/myMySQLDatabase\/plugin\/Scheduler\/Scheduler.php","line":415,"function":"setStatus","class":"Video","object":{"views_count_25":null,"views_count_50":null,"views_count_75":null,"views_count_100":null,"modified":"2023-05-19 00:45:09"},"type":"->","args":["a"]},{"file":"\/var\/www\/html\/myMySQLDatabase\/plugin\/Scheduler\/Scheduler.php","line":458,"function":"releaseVideosNow","class":"Scheduler","type":"::","args":[4]},{"file":"\/var\/www\/html\/myMySQLDatabase\/plugin\/Scheduler\/Scheduler.php","line":434,"function":"addNewVideoToRelease","class":"Scheduler","type":"::","args":[4,"2023-05-18 23:38:00",null]},{"file":"\/var\/www\/html\/myMySQLDatabase\/plugin\/AVideoPlugin.php","line":795,"function":"afterNewVideo","class":"Scheduler","object":{},"type":"->","args":[4]},{"file":"\/var\/www\/html\/myMySQLDatabase\/objects\/aVideoEncoderNotifyIsDone.json.php","line":59,"function":"afterNewVideo","class":"AVideoPlugin","type":"::","args":[4]}] SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

**** **** * **** ** ** **** **** *** - pass

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: updateFilesize: No need to update videos_id=4 filename=video_230519004234_vc48a filesize=54029694 51.53MB SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: getDirSize: found 54029694 from - 54029694 path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/ SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: getDirSize: start path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/ SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video is done notified 4: Neil Oliver's Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs- SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: updateFilesize: videos_id=4 filename=video_230519004234_vc48a filesize=54029694 51.53MB SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video::save (Neil Oliver's Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-) Saved id = 4 0:10:29 [{"file":"\/var\/www\/html\/myMySQLDatabase\/objects\/video.php","line":1991,"function":"save","class":"Video","type":"->"},{"file":"\/var\/www\/html\/myMySQLDatabase\/objects\/aVideoEncoderNotifyIsDone.json.php","line":50,"function":"updateFilesize","class":"Video","type":"::"}] SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: created changed in table=videos id=4 created=2023-05-19 00:42:34 SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: getDirSize: found 54029694 from - 54029694 path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/ SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: getDirSize: start path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/ SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video::save (Neil Oliver's Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-) Saved id = 4 0:10:29 [{"file":"\/var\/www\/html\/myMySQLDatabase\/objects\/aVideoEncoderNotifyIsDone.json.php","line":44,"function":"save","class":"Video","type":"->"}] SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: created changed in table=videos id=4 created=2023-05-19 00:42:34 SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video::setStatus(h) [h] SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video::setStatus(h) [{"file":"\/var\/www\/html\/myMySQLDatabase\/objects\/video.php","line":791,"function":"setStatus","class":"Video","object":{"views_count_25":null,"views_count_50":null,"views_count_75":null,"views_count_100":null,"modified":"2023-05-19 00:45:09"},"type":"->","args":["h"]},{"file":"\/var\/www\/html\/myMySQLDatabase\/objects\/aVideoEncoderNotifyIsDone.json.php","line":42,"function":"setAutoStatus","class":"Video","object":{"views_count_25":null,"views_count_50":null,"views_count_75":null,"views_count_100":null,"modified":"2023-05-19 00:45:09"},"type":"->","args":["a"]}] SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

**** **** * **** ** ** **** **** *** - pass

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: useVideoHashOrLogin: 1, 4 SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: User:login finish users_id=1 SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: useVideoHashOrLogin: 1, 4 SCRIPT_NAME: /objects/aVideoEncoderNotifyIsDone.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: aVideoEncoder.json: Success path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/video_230519004234_vc48a_480.mp4 SCRIPT_NAME: /objects/aVideoEncoder.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: aVideoEncoder.json: Files Received for video 4: Neil Oliver's Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs- SCRIPT_NAME: /objects/aVideoEncoder.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Do not need update duration: duration is valid 0:10:29, file not exists SCRIPT_NAME: /objects/aVideoEncoder.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: Video::save (Neil Oliver's Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-) Saved id = 4 0:10:29 [{"file":"\/var\/www\/html\/myMySQLDatabase\/objects\/aVideoEncoder.json.php","line":188,"function":"save","class":"Video","type":"->"}] SCRIPT_NAME: /objects/aVideoEncoder.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: created changed in table=videos id=4 created=2023-05-19 00:42:34 SCRIPT_NAME: /objects/aVideoEncoder.json.php

[19-May-2023 00:45:09 UTC] AVideoLog::DEBUG: decideMoveUploadedToVideos: destinationFile path/to/my/myMySQLDatabase/site/videos/video_230519004234_vc48a/video_230519004234_vc48a_480.mp4 filesize=34719824 (33.11MB) SCRIPT_NAME: /objects/aVideoEncoder.json.php

[para2022]

Encoder log:

[19-May-2023 00:42:34 UTC] decideFormatOrder: {"file":"Neil Oliver's Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-.mp4","audioOnly":"false","spectrum":"false","webm":"false","override_status":"","update_video_id":0,"inputHLS":"false","inputLow":"false","inputSD":"false","inputHD":"false","inputAutoHLS":"false","inputAutoMP4":"false","inputAu> [19-May-2023 00:42:34 UTC] AVideo-Encoder Format::getFromOrder(7) [19-May-2023 00:42:34 UTC] Upload.php line: 113 "mp4" [19-May-2023 00:42:34 UTC] Encoder::sendFile videos_id=0, format=mp4 [19-May-2023 00:42:34 UTC] Encoder::sendFile sending file to aVideoEncoder.json from [19-May-2023 00:42:34 UTC] sendToStreamer to aVideoEncoder.json [19-May-2023 00:42:34 UTC] Encoder::save id=() title=(Neil Oliver\'s Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-) [19-May-2023 00:42:34 UTC] Encoder::save id=(4) title=(Neil Oliver\'s Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-) [19-May-2023 00:42:34 UTC] sendToStreamer spectrumimage not set [19-May-2023 00:42:34 UTC] sendToStreamer rawVideo not set [19-May-2023 00:42:34 UTC] sendToStreamer image not set [19-May-2023 00:42:34 UTC] sendToStreamer gifimage not set [19-May-2023 00:42:34 UTC] sendToStreamer webpimage not set [19-May-2023 00:42:34 UTC] sendToStreamer video not set [19-May-2023 00:42:34 UTC] sendToStreamer https://encoder1.ourtube.co.uk/aVideoEncoder.json in 0.142 seconds {"error":false,"target":"aVideoEncoder.json","postFields":{"duration":"EE:EE:EE","title":"Neil Oliver's Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-","first_request":1,"categories_id":0,"format":"mp4","resolution":"","videoDownloaded> [19-May-2023 00:42:34 UTC] Encoder::save id=(4) title=(Neil Oliver\'s Saturday Monologue - 13th May 2023- Duration - 10 mins 30 secs-)

[para2022]

the encoder is horrible at times it starts 2 main processes on same file and results in corrupted video's and goes in a loop Especially with files over 1GB it has serious problems ... plus that it has to encode the whole file once again for the error report it takes ages to get a video on the streamer which is absolutely not needed .. i stopped using Avideo completely went to jellyfin as that does what i need it to do and faster without any errors as i am the only one to upload on my video site jellyfin works perfect .... but lately Avideo the streamer and the encoders are terrible this script/project has gone down the drain if you ask me . my major problem i have with it is that it needs root permissions and is totally not acceptable !!!

I dont think jellyfin is better. This system just needs to get on top of its bugs. I seem to get over one hurdle then their is another problem and it seems endless, causing me a great deal of unnecessary stress and time that i do not have.

[para2022]

Heres the exact same file uploaded again and now not broken. Its infuriating. The neil oliver one.

[para2022]

And here is another server. Note the exact same file uploaded 3 times and 2 failed and one worked. This is enough proof its a bug in the script.

[mohmafia]

look with htop and see if you see 2 white (main|) ffmpeg processes on the same file ..... as that is what it was doing for me and results in corrupted video and missing files... and it wont show in the error logs

[para2022]

look with htop and see if you see 2 white (main|) ffmpeg processes on the same file ..... as that is what it was doing for me and results in corrupted video and missing files... and it wont show in the error logs

Thanks how did you fix it?

[para2022]

look with htop and see if you see 2 white (main|) ffmpeg processes on the same file ..... as that is what it was doing for me and results in corrupted video and missing files... and it wont show in the error logs

There are about 50 processes spawned, all the same

[DanielnetoDotCom]

Hi,

I could not see any issue in your log. and I can see this is randomly what makes it harder to resolve.

but our best shot is to find something in your logs, encoder, or streamer at the time the error file is transferred

the ffmpeg process can be something else, not encoding. it may be an image extraction for example. for the sprits thumbs of live images.

[para2022]

There is no error in the logs which is why its doing my head in! This is a big problem that needs resolving because no matter what linux flavours i use wether it debian or ubuntu (all versions) The problem exists on 3 servers and i use 2 separate hosting companies so its not that either. Never happened before until weeks ago. I had the looping issue in the past and vids being stuck in the encoder. I get none of those now just broken files endlessly which means ive had no encoder on my site for 2 weeks now. Those files that are converted work perfect when i download them, there is nothing wrong with them when i play them.

[para2022]

Also if you look at the images i sent there is a failed one and a successful one in the same image. They are identical in file size so my guess is they are being converted correctly and the system doesnt recognise that

[DanielnetoDotCom]

the only way for me to fix is if I can reproduce it, otherwise, I cannot help you.

I asked you to try to reproduce in my encoders so we can try to see if the problem is in your streamer or in the encoder as long as I am having no problem on my side.

[para2022]

The ffmpeg processes in htop are all the same they are not image creation they are video conversion of the same file it seems strange that there are 30-50 of these:

31_tmpFile.mp4

[para2022]

Cannot login in your encoders. My Login does not work from the streamer site using google! And you are aware this problem is intermittent so im supposed to just endlessly upload until it occurs? And thats, if!

[para2022]

???

[para2022]

is this correct?

[DanielnetoDotCom]

???

where this come from?

[DanielnetoDotCom]

is this correct?

Seems to be a correct encoding process.

[para2022]

???

where this come from?

Your demo site. I cannot login to your encoder site

[DanielnetoDotCom]

Cannot login in your encoders. My Login does not work from the streamer site using google! And you are aware this problem is intermittent so im supposed to just endlessly upload until it occurs? And thats, if!

you can add my encoder in your site configuration and click in the encoder button in the top

[DanielnetoDotCom]

???

where this come from?

Your demo site. I cannot login to your encoder site

this is the DiskUploadQuota, I just checked and it is not enabled, can you please try again?

[para2022]

I am now going to see what happens with good old faithful Centos. If it does not work im giving up and using direct upload which is a shocking state of affairs. im just getting so pissed off and mentally worn out with it all, as their always seems to be some big issue for me to fix that wasnt there before like the thumbnails ect ect. I know you do your best but these issues have totally killed off my site as who wants to use a site that has issue after issue all the time. Its not practical for any site.

[DanielnetoDotCom]

Just confirming you have an encoder installed in Ubuntu 22 and it is also giving you errors?

is your streamer installed in Ubuntu?

[para2022]

yes one encoder is on 22 and one on 20.04 and both do the exact same wether it is encoded locally or remotely makes no difference.

[DanielnetoDotCom]

That is weird, I hope you can help me reproduce it.

the weirdest part is that it is random and there is nothing in the log.

[para2022]

Brand new installs each time. I tried both 22 and 20 versions of ubuntu to see if it was a version bug.

Php 8.1 Apache Mysql

32 GB ram and xeon 8 core

2GB upload limit and memory limit 4096 to allow for this

Health status is all green.

No errors in any logs.

If i go in database and set it from broken to active it works, sometimes it stays fixed other times it sets back to broken again which is what mainly happens.

If i play those files they play fine. Also when they are temporary fixed in the database they play.

Ive even tried static binaries of ffmpeg.

Nothing works

[para2022]

Il tell you what else is weird. Historically i always got looping and vids stuck in the encoder. That never ever happens now and i have not seen it in such a long time which is great, but now this bug i have instead.

[DanielnetoDotCom]

Hold on ...

If you change it in the database it works?

If yes the thing is in your streamer and it may happen if you try to access the video before finish the transfer, there is no need to change in the database, if you try to access the video again it may auto fix the broken missing file.

I mean this is not a real issue it will auto fix

[para2022]

Sometimes it will stay working but not often as it will in time just switch back to broken files again. The odd one will stay working again. I am testing locally at the moment as i am not testing remotely until i get to the bottom of whats going on, so this is not a files transfer issue as the encoder/streamer are on same server. The main streamer site is in direct upload only. The 2 encoder servers are being used to solve this without causing any disruption to my main site. The websocket doesnt need to be on for the encoder sites yea?

[para2022]

Your encoder login does not work. You can see there im logged in, but no encoder page.

[DanielnetoDotCom]

if it stays working we should not worry about it.

when it says broken missing files, check if your encoder does not still transfer your files

once is finished try to access it again, if it fixes, that means it is all good, if still broken we can focus on that file and then try to find errors in the logs.

[para2022]

All the files are there when transfered. It never goes from broken to active automatically. Once it goes broken it stays broken even though all its files are there. Like i say sometimes if i set it to active in the database sometimes it will work again, however it will still display as broken files in the video panel and cycling through all the statuses on it will set it to active again. It may stay that way or it will turn broken again. Users will delete broken files before i get a chance to look at them which is why im testing on other servers.

[DanielnetoDotCom]

that is important information, are all the video files present and it is saying the video is broken? is that correct?

if this is correct, forget your encoders, and let's focus on the streamer server only. are you using a separate drive to store your videos?

[para2022]

Yes all files are always there they have never actually been missing throughout this issue. I download then from there directory and play them fine.

Both encoders are raid 0 and ssd. Vids are stored on the same drive.

[DanielnetoDotCom]

so as long as your videos are transferred to the streamer, forget your encoders, they are good,

if you try to play a "broken" video with an admin user if should detect the video the first time you try, then if you try again it should be ok.

if it is not detecting we may want to investigate that.

so from now on

1. forget your encoders, they are good
2. do not change the broken video status in the database anymore, let the script fix
3. get a broken video, double-check if the video files are there, and try to access it (twice).
4. if auto fix, it is all good, if fail check the logs, and let's investigate that.

[DanielnetoDotCom]

I just tested something here, for now, clear your cache before access the video, the video may be back

[para2022]

Im using debian buster for now. Il let you know how i get on.