search

FTW-Entertainment-LLC / Fay-Ray

Fay & Ray are the autonomous bots for AnimeFTW. The bots will download, encode and upload episodes to AnimeFTW as they get released to the nyaa feed.
0 stars 0 forks source link

Estimate filesize #6

Closed hanimayahi closed 7 years ago

hanimayahi commented 8 years ago

Add in extra support to Bot-san. I want it to be able to do the following:

Depending on video and audio length from the source, calculate an estimate of the final encode source (since we already know the bitrates we use it shouldn't be too hard to come up with a close estimate). Then check if the encoded file is around that expected size. If it isn't close at all to what is expected, try to re encode that one, check again and if the size is still off, send out a warning for this file so a tech can check it out.

Nikey646 commented 8 years ago

With CancerCoder 4+ this will be counter productive, since it uses the source video's bitrate if the calculated bitrate is > the source bitrate, to prevent bloated bitrates. Same with resolution and ratio.

hanimayahi commented 8 years ago

I could add in the same check to Bot-san. Mostly I want this to be just an extra check in case something wrong happens. I've noticed that sometimes while I've been testing, when I shut down bot-san (and CC with it) while it's encoding something, some files with very low file sizes will show up in the encoded folder. I have no idea why this happens, Bot-san doesn't copy or mux any files from the temp folder, and the temp folder is not the same as the output folder.

Nikey646 commented 8 years ago

You really shouldn't be randomly killing CancerCoder, there is no code at all to handle forced shutdowns, and it may still execute a huge portion of code (Such as trying to finish the current execution of the Foreach loop, and copying files across despite not being ready, which is likely what is happening), and there is no safe way to handle this in a way that will execute for every method of closing a console window.

hanimayahi commented 8 years ago

I'm only killing CC during encode tests, but the reason why I want to have this check is what if Bot-san would crash? CC would go with it and it's Node after all and I am sure that during release, I'll have missed a few places I'd need to be error handled. I want this check in case something like this happens

Nikey646 commented 8 years ago

You do not need to worry about that. The Detached option automatically handles that. It will also save time, because it will allow the current encode to finish, rather than having to start from scratch.

hanimayahi commented 8 years ago

I'm using the Detached option, and I've tried it. Mono still shuts down with Bot-san. Plus Bot-san won't recognize that CC is encoding this specific episode after it's restarted because it will have lost the process.

Nikey646 commented 8 years ago

Well, you are not running mono but sh, so there is a slight difference there.

Also, testing on windows with the code below produces a valid child process with node exiting immediately, and removing child.unref() allows the cmd window to be closed while leaving CancerCoder executing, so presuming it was using mono directly, and at worst case unreferenced the window, then all would be good in the world. The only downside is that it for some reason randomly spawns console windows when CancerCoder executes a new portion of the encode, but that won't matter on the encoding server.

As for detecting encodes, CancerCoder always puts the completed file into the output folder, using the prefix provided and episode number. This means you could easily detect the completed encodes based off of files from the output folder, since you are storing the prefix in the savefile. From there deleting the files (after confirming that they were successfully uploaded to FTP) should be fine (Source files would be kept on the seedbox), solving the issue of requiring the process to determine what the output is, no?

Hardcode heaven example i previously mentioned:

var spawn = require('child_process').spawn

// CancerCoder 4.0.a3 (Unreleased, a2 doesn't work w/ node), requires the Mode param and no longer takes the QualityBuff param
var child = spawn('C:\\Projects\\AnimeFTW\\CancerCoder\\bin\\Debug\\CancerCoder.exe', ['mode:encode', 'SourceFolder:C:\\e\\debug', 'OutputFolder:C:\\e\\AEE', 'TempFolder:C:\\e\\Temp', 'Prefix:debug', 'FileIndex:0', 'Episode:1'], { detached: true })

child.unref()

This then results in CancerCoder completing no matter what (Assuming my quick tests on my unix laptop are correct and it does work as i intended, w/o using child.unref()), even if node crashes.

hanimayahi commented 8 years ago

As for detecting encodes, CancerCoder always puts the completed file into the output folder, using the prefix provided and episode number. This means you could easily detect the completed encodes based off of files from the output folder, since you are storing the prefix in the savefile. From there deleting the files (after confirming that they were successfully uploaded to FTP) should be fine (Source files would be kept on the seedbox), solving the issue of requiring the process to determine what the output is, no?

Yes, detecting finished encodes would be fine that way. But it wouldn't really work if Bot-san is restarted after a crash after just a few minutes, while CC still hasn't finished.

Nikey646 commented 8 years ago

My original statement assumed that you would be polling it, rather than just checking it once.

hanimayahi commented 8 years ago

Well sure I can save the process ID, and then when it tries to encode it again, make a check if the old process is still running, and if it is I'll have it wait until it's finished. But then I'd still be back to square one if the server unexpectedly shut down, because then there'd be no process to wait for..

Nikey646 commented 8 years ago

Uhh what? "Polling it", it being the output folder...

hanimayahi commented 8 years ago

Why should I poll the output folder?

hanimayahi commented 8 years ago

Nikey, I already check that the file exists in the output folder to determine a failed or successfult encode. It's the filesize I want to start checking as well if it's close to an estimation in case something wrong happened.

Nikey646 commented 8 years ago

sigh

Yes, detecting finished encodes would be fine that way. But it wouldn't really work if Bot-san is restarted after a crash after just a few minutes, while CC still hasn't finished.

And i'm saying poll the damn folder, don't just check it once. That way the damn bot could actually detect the file once it's moved there, even if it's restarted in the middle of CancerCoder encoding. Hell you could also do process polling or just completely ignore what processes are running, since you'll get at least enough details to find the series in the savefile from the output's filename.

However, your entire problem is that closing CancerCoder with the bot which presumably causes CancerCoder to move incomplete files to the output folder, which i cannot fix safely, since not all ways of ending a process will trigger any function i can use, and there is no point in partially implementing something.

A completely viable solution that covers every basis mentioned here, and needs only to be done on startup, and continued until no instances of CancerCoder are running:

By combining both the Process and Folder checks, you can ensure any encodes which were started before the bot went offline, or completed before the server crashed / restarted, are detected and the relevant data is updated. Thus, no need to check the filesize.

As for calculating the expected filesize, simply put...The complexity of defining the bitrate is no longer a static Width * Height / 678 for CancerCoder 0.4+, and the additional complexity of determining it would be identical to replicating about half of the Informational Stage of CancerCoder.

I'm done with this just for your information. If you cannot see that it would (imo) be more beneficial to allow CancerCoder to run, and just recover as much as possible from an unexpected exit, then go ahead and waste your time implementing identical functionality that CancerCoder 0.4+ will need to do anyway, and more on top of that. As such, I will no longer be partaking in this issue.

hanimayahi commented 8 years ago

sigh

Still, your suggestion doesn't fix the issue of CancerCoder prematurely moving files during a unexpected system reboot. Your suggestion is valid for what it does, but not what I want to do.

Tyvm.

hanimayahi commented 7 years ago

Closing this because I don't think it's viable anymore. Replaced by #35