WWBN / AVideo-Storage

Storage Server for AVideo Platform Open-Source
26 stars 37 forks source link

Update Disk Usage ; #12

Closed akhilleusuggo closed 4 years ago

akhilleusuggo commented 4 years ago

When I update disk usage , does not report the right usage . For some reason does it multiple times . Those are the logs of after 1 click on disk update usage .

[30/Jun/2020:08:52:54 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_HD.mp4 HTTP/1.1" 200 3646 "-" "-"
[30/Jun/2020:08:52:59 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_Low.mp4 HTTP/1.1" 200 3645 "-" "-"
[30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_SD.mp4 HTTP/1.1" 200 3645 "-" "-"
[30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_HD.mp4 HTTP/1.1" 200 3646 "-" "-"
[30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_Low.mp4 HTTP/1.1" 200 3645 "-" "-"
[30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_SD.mp4 HTTP/1.1" 200 3645 "-" "-"
[30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_HD.mp4 HTTP/1.1" 200 3646 "-" "-"
[30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_Low.mp4 HTTP/1.1" 200 3645 "-" "-"
[30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_SD.mp4 HTTP/1.1" 200 3645 "-" "-"
 [30/Jun/2020:08:53:01 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_HD.mp4 HTTP/1.1" 200 3646 "-" "-"

File size shown on MyVideos section : image

Real file size : image

DanielnetoDotCom commented 4 years ago

this is really strange. does it happen every time you click on it? or just some times. I was not able to reproduce it here.

akhilleusuggo commented 4 years ago

everytime . I mean this gonna be an issue for many people who's planning to get the DiskQuota plugin .

DanielnetoDotCom commented 4 years ago

Agree this is wrong, I just cannot see how is it happening.

is it a video hosted in the storage?

the same happens with local videos?

akhilleusuggo commented 4 years ago

@DanielnetoDotCom happens much more on storage site then the local site . I mean right now , all my storage site is wrong . Everything indicates x2 x3 more size then it is .

I'm not sure if this is caused by the dump files on the streamer site .

[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179/index.m3u8 in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179.mp4 in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179.webm in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179_Low.webm in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179_SD.webm in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179_HD.webm in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179.pdf in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179.zip in /var/www/html/YouPHPTube/objects/video.php on line 2575
[03-Jul-2020 15:12:54 Europe/Helsinki] PHP Warning:  filesize(): stat failed for /var/www/html/YouPHPTube/videos/_ID.81107179.mp3 in /var/www/html/YouPHPTube/objects/video.php on line 2575

Then it's start some curl for header . But again , does that x2 x3 times . Witch explains why does report wrong filesize . Because the dump files does redirect to the storage site . So it's counting the dump files as files .

DanielnetoDotCom commented 4 years ago

It will count the dummy files, but it supposes to be only 10 bytes each. also count the images, and zip files if they were not deleted. can you please check if it deletes the zip files?

I mean it will count anything with the video "filename" form database

akhilleusuggo commented 4 years ago

image

No zip files , nothing except dump files.

DanielnetoDotCom commented 4 years ago

can you please make something like this in your streamer and storage

ls -lah /var/www/html/YouPHPTube/videos/_ID.81107179*

akhilleusuggo commented 4 years ago

STORAGE SITE : image

STREAMER SITE: image

DanielnetoDotCom commented 4 years ago

Ok, all looks perfect, so if you click on the "update disk storage" it will give you the wrong size?

image

akhilleusuggo commented 4 years ago

Yes Sir

DanielnetoDotCom commented 4 years ago

ef852e370b841.68237251_HD.mp4 HTTP/1.1" 200 3646 "-" "-" [30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_Low.mp4 HTTP/1.1" 200 3645 "-" "-" [30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_SD.mp4 HTTP/1.1" 200 3645 "-" "-" [30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_HD.mp4 HTTP/1.1" 200 3646 "-" "-" [30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_Low.mp4 HTTP/1.1" 200 3645 "-" "-" [30/Jun/2020:08:53:00 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_SD.mp4 HTTP/1.1" 200 3645 "-" "-" [30/Jun/2020:08:53:01 +0000] "HEAD /videos/_YPTuniqid_5ef852e370b841.68237251_HD.mp4 HTTP/1.1" 200 3646 "

Is this log from storage or streamer?

Could you detect if is the streamer that is requesting multiple times or is the storage that is processing multiple times?

DanielnetoDotCom commented 4 years ago

I will fix, I could reproduce it here

akhilleusuggo commented 4 years ago

Hello Daniel , on the streamer site , too many requests are made that looks like inspecting the header

[03-Jul-2020 15:38:29 Europe/Helsinki] AVideoLog::DEBUG: getUsageFromURL: start (https://storage.example.com/videos/_ID.68237251_HD.mp4)
[03-Jul-2020 15:38:29 Europe/Helsinki] AVideoLog::DEBUG: getUsageFromURL: curl_init 
[03-Jul-2020 15:38:29 Europe/Helsinki] AVideoLog::DEBUG: getUsageFromURL: response header HTTP/1.1 200 OK
Date: Fri, 03 Jul 2020 12:38:29 GMT
Server: Apache
Set-Cookie: 46545649a030c1c8227=srpekdqtov1654654654jt0vj6; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: max-age=1, private, must-revalidate
Pragma: no-cache
Set-Cookie: 79f9a78441fe287987930c1c8227=srpekdqtov16543219b7jt0vj6; path=/
Access-Control-Allow-Origin: *
Last-Modified: Mon, 29 Jun 2020 13:49:14 GMT
ETag: "11b31ec4e-fb8a80"
Content-Length: 4751223886
Content-Type: video/mp4

I don't wanna past here all the log , but you got the idea , just too many of this requests of header .

DanielnetoDotCom commented 4 years ago

Please try this update on the Streamer site

akhilleusuggo commented 4 years ago

Works for me , but php updateDiskUsage not working . Nor image

akhilleusuggo commented 4 years ago

When running php updateDiskUsage on /install directory

image

akhilleusuggo commented 4 years ago

@DanielnetoDotCom Daniel , I have to delete everyday handreds of this files . Can you send a fix ? Maybe they should be on /tmp directory and not on the/object directory or /install