Why does editing a videos image cause "Broken Missing Files"

[walt93]

Describe the bug I uploaded a video and it was fine. I edited the video's thumbnail, uploading a new one. It changed state to "Broken Missing Files"

To Reproduce Steps to reproduce the behavior:

1. Encode a video
2. Verify the video appears
3. Click on edit video
4. Upload a new thumbnail
5. Go to /mvideos and observe the video is in "Broken missing files" state.

Expected behavior The video to not be in "Broken missing files" state. The video to maintain the URL I downloaded it from, because we know that videos break all the time, so that I might recover it again (until Broken missing files is ultimately solved for good - we need something like this)

Error Logs https://pastebin.com/z4jt3Qcy

Screenshots

[DanielnetoDotCom]

Weird, can you reproduce it on the demo site?

[walt93]

Daniel, if I upload ten videos and go to edit the thumbs this will happen 1-2 times. Time is valuable and I do not have it to reproduce on the demo site. Did you scan the log? I captured it as the event occurred.

UPDATE: the video file is most definitely missing, it is unclear if it ever successfully encoded, or, if it "pretended to" and we didn't detect the "Broken Missing Files" condition until I did a pageload of conspyre.tv/v/18427?

This is after fixing permissions on the videos directory, which, we have no idea how the permissions got ganked in the first place. 🤨

mysql> select id,title,filename from videos where title like "Stanley Meyer%";
+-------+--------------------------------------------------------------+------------------------------------+
| id    | title                                                        | filename                           |
+-------+--------------------------------------------------------------+------------------------------------+
|   411 | Stanley Meyer on How the Water Fuel Injection System Works   | _YPTuniqid_604b11ac18e6b9.63756340 |
| 11742 | Stanley Meyers Water Powered Car                             | video_220924180445_v55c5           |
| 11743 | Stanley Meyer 1995 It runs on water                          | video_220924180556_vb2a2           |
| 18427 | Stanley Meyer: The Mysterious Inventor of Water-Powered Cars | v_240429231036_v7db3               |
+-------+--------------------------------------------------------------+------------------------------------+
4 rows in set (0.03 sec)

mysql> ^DBye
ubuntu@pyron:~/scripts$ ls -ls /var/www/html/conspyre.tv/videos/v_240429231036_v7db3
total 1784
1172 -rw-r--r-- 1 www-data www-data 1197354 Apr 30 07:27 v_240429231036_v7db3.jpg
   4 -rw-r--r-- 1 www-data www-data      10 Apr 30 04:16 v_240429231036_v7db3.notfound
  16 -rw-rw-rw- 1 www-data www-data   15164 Apr 30 08:04 v_240429231036_v7db3_128.jpg
  32 -rw-rw-rw- 1 www-data www-data   30858 Apr 30 08:04 v_240429231036_v7db3_192.jpg
  52 -rw-rw-rw- 1 www-data www-data   50447 Apr 30 08:04 v_240429231036_v7db3_256.jpg
 100 -rw-rw-rw- 1 www-data www-data   98755 Apr 30 08:04 v_240429231036_v7db3_384.jpg
 160 -rw-rw-rw- 1 www-data www-data  161905 Apr 30 08:04 v_240429231036_v7db3_512.jpg
  12 -rw-rw-rw- 1 www-data www-data    9491 Apr 30 08:04 v_240429231036_v7db3_96.jpg
 236 -rw-rw-rw- 1 www-data www-data  239818 Apr 30 08:04 v_240429231036_v7db3_thumbsV2.jpg

If I continue to look for more "Broken Missing Files" I can see that not all are created equally:

This one doesn't appear to be in the same state:

mysql> select id,title,filename from videos where title like "United We%";
+-------+---------------------------------------------------+----------------------+
| id    | title                                             | filename             |
+-------+---------------------------------------------------+----------------------+
| 18409 | United We Stream: Navigating Tough Paths Together | v_240428124253_v9bb1 |
+-------+---------------------------------------------------+----------------------+
1 row in set (0.01 sec)

mysql> ^DBye
ubuntu@pyron:~/scripts$ ls -ls /var/www/html/conspyre.tv/videos/v_240428124253_v9bb1
total 4
4 -rw-r--r-- 1 www-data www-data 10 Apr 28 19:04 v_240428124253_v9bb1.notfound

My workflow is I have very little time in the morning to scan social media for new content. I upload 3-5 pieces and come back at the end of the week to publish on the Roku. There are always several broken videos. Always.

Why is this so flaky, Daniel? It's always been this way, just haven't been vocal about it. :-/

[DanielnetoDotCom]

you may want to check your logs to try to find some clue

in this dir /var/www/html/conspyre.tv/videos/v_240429231036_v7db3there is no video, so it is really a broken missing file video

there was any video before changing the image?

[GreenDragonTavern]

How many concurrent in encoder configuration.php? If multiple, are there enough server resources to handle concurrent count? Largest source and format? Are php.ini settings configured to handle largest source and processing?

[gbrn]

@DanielnetoDotCom @walt93 I also have old videos, which are in "Broken Missing Files", even though the files are present on the server.

[walt93]

There's only two of us using the encoder & I have PHP set to 10GB upload size and the ram is IIRC 512GB (or whatever was recommended at install time). I had an encoder farm setup, but, moved the server & the instructions to install it don't work at all so I'm down to a single encoder.

It seems that a video does not go into "Broken Missing Files" until someone loads it via /video or /v/ routes, so I suspect that many times when I hit this it's genuinely missing the video.

But why? Why is it missing the video? It seems the encoder is having tons of issues, it's not reliable for me.

[walt93]

I'm coming to the conclusion that its a lot more than 1-2 files in 10. It's looking more like 80% of the uploads I did this week are broken.

My time schedule is too limited for this. What can I do to help you debug this encoder so it's not producing so many "broken" files. FWIW I'm not calling these files "broken". I claim it's the encoder which is "broken" Daniel…

The streamer log is super noisy. It sprays tons of noise.

The encoder log is super clean. There are only two people using it.

Can we put some logging in the encoder which is useful? It seems that when the encoder is done, we should know if the encode is "broken" or not. If we put some logging in here, we will actually be able to find it again.

When I upload files on Monday morning, and on Tuesday morning, and on Wednesday morning... and eventually find time on Saturday or maybe Sunday, the streamer logs are so far in the past as to be gone.

Let's use some of our cleverness to solve the problem, the computer is [still] stupid (for the moment…)

[GreenDragonTavern]

Not saying this is causing issue but in your php.ini for apache2 settings you have: upload_max_filesize = 10240M Yes?

What are the settings for: max_execution_time = ? max_input_time = ? memory_limit = ?(should be slightly larger than post_max_size, something like 10252M) post_max_size = ? (should be slightly larger than upload_max_filesize, something like 10246M)

[walt93]

max_execution_time = 7200 memory_limit = 512M max_input_time = 60 post_max_size = 10G

I just did a deeper dive. Filtering /mvideos on Broken Missing Files doesn't do anything (finds nothing). You have to open every videos page before it even knows that it has a "broken missing files" condition - that seems to be the only place its detected. So I have a LOT more of these than I was aware of.

I'm using this /plugin/MobileManager/upload.php which Daniel said to use about a year ago.

It used to work.

Now it seems like it does nothing but fall on its face.

I literally cannot use the regular encode/upload flow as it takes a retarded number of steps which equates to about 3-4 minutes per video including doing the thumbnail - not going to happen.

My script which calls this accomplishes everything with about 5 seconds of user time required but it seems like it has been broken for some time now.

[DanielnetoDotCom]

https://github.com/WWBN/AVideo/issues/9030#issuecomment-2088731114

[walt93]

Daniel - we went through this already and there is nothing in the encoder logs.

I've just come to the realization that it is failing more like 80% of the time. It will fail fo a short 5 minute video, and work for a 90 minute video - so I don't think its a resource constraint.

What can we do to diagnose / debug this when there is nothing in the encoder log for all of these broken/missing files?

[DanielnetoDotCom]

I am not asking for the logs, I am asking for some clues, without clues I cannot help

but any way, I hardly can help if there is nothing in the logs

[GreenDragonTavern]

In order of size, memory_limit should be larger than post_max_size and post_max_size should be larger than upload_max_filesize (there's inherent overhead in addition to uploaded video). You also may want to change max_input_time to match max_exection_time. Your setting of 60 seconds may timeout prior to parsing an uploaded video.