Postprocessing even failed / bad downloads as successful

[raysv]

Hi, Recently i have noticed that nzbtosickbeard is postprocessing stuff as "successful" and actually copies over files even though nzbget is marking them as bad and failure.

Kind    Time    Text
INFO    Sun Aug 07 2016 00:42:12    Collection \* added to history
INFO    Sun Aug 07 2016 00:42:12    Post-process-script nzbToMedia/nzbToSickBeard.py for \* successful
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: SickBeard: Successfully post-processed *
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: [00:42:12] [INFO]::MAIN: The /opt/nzbget/scripts/nzbToMedia/nzbToSickBeard.py script completed successfully.
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: [00:42:12] [INFO]::CLEANDIRS: Directory /mnt/nzb-downloads/completed/tv/\* still contains 1 unprocessed file(s), skipping ...
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: [00:42:12] [POSTPROCESS]::SICKBEARD: Successfully processed
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: [00:42:12] [POSTPROCESS]::SICKBEARD: Deleting file: _brokenlog.txt
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: [00:42:12] [POSTPROCESS]::SICKBEARD: Deleting file: _newzNZB.nfo
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: [00:42:12] [POSTPROCESS]::SICKBEARD: Deleting file: dark.matter.s02e06.720p.hdtv.x264-fleet.nfo
INFO    Sun Aug 07 2016 00:42:12    nzbToSickBeard: [00:42:12] [POSTPROCESS]::SICKBEARD: Deleting file: RARBG.com.txt

Any idea of why?or how to fix it?

[flyize]

I noticed the same thing on SAB. SAB killed the download with this status: Aborted, cannot be completed.

Yet, NZBtoSick(Rage) reports a success. Help?

[clinton-hall]

can you show more of the log? i.e all the earlier part that shows how/why this script decided to send this on to SickRage. if the download failed, the script should only mark as good and process if it can actually extract and verify a file as playable.

[flyize]

Here is mine. SAB showed 'Aboerted, cannot be completed'.

http://pastebin.com/yxc11Xsd

[clinton-hall]

yeah, the issue here is that the script successfully extracted the video

EXTRACTOR: Extraction was successful for E:\Downloads\incomplete\Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x\Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x264-CtrlHD.part01.rar to E:\Downloads\incomplete\Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x

but there was another incomplete archive in a sub-directory

EXTRACTOR: Extraction failed for E:\Downloads\incomplete\Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x\__ADMIN__\Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x264-CtrlHD.nzb.gz.

The script then checks the extracted video an verified that it was playable

TRANSCODER: SUCCESS: [Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x264-CtrlHD.mkv] has no corruption.

SickBeard Then successfully processes this episode.

SICKBEARD: Moving file from E:\Downloads\incomplete\Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x\Penny.Dreadful.S01E08.1080p.BluRay.DD5.1.x264-CtrlHD.mkv to D:\Shares\TV Shows\Penny Dreadful\Season 01\Penny.Dreadful.S01E08.1080p.BluRay.mkv

So, if this is the correct, playable, video, the script did as wanted...

[flyize]

Is the incomplete archive what SAB is seeing then?

Also, the video initially plays. However, if I try to skip to the middle, VLC crashes out.

[clinton-hall]

in theory, if any of the included metatdata, subs, samples etc can't be extracted, or any individual file can't be downloaded, then SABnbzd will mark as failed...

However, as long as the video is good, then that is all most of us care about... SickBeard and its derivatives all support metadata creation and some support sub downloads etc.

So the script checks to see if there is a playable video by passing it through ffprobe.

At the end of the day, this video was extracted and is playable. It may be a strange format or have other issues when it comes to skipping etc in VLC... I don't know how to test for that.

Is the video playable to end if you don't skip? At the end of the day, this sounds like a poorly formatted video, so you may want to get another version... but this is a rare set of circumstances that would be hard to catch automatically.

[flyize]

I've tried with a couple different versions of this show. They all seem to be like that. I'll test just letting the video play.

Would it be crazy if the video was designed to pass your test, only to actually fail?

[clinton-hall]

It would not be overly crazy.

I don't pretend to understand the desire to release a bad video or spam. The idea of releasing a video with a virus or trojan is somewhat understandable (in that people try to disseminate viruses and trojans for some perceived gain).

Due to the number of fake/spam releases I created this test process to automatically dismiss the release and search for another. this same process was used in a few other tools (Fake Detector in NZBGet etc) so if someone really wanted to create a corrupt video that would pass my test, this would be possible...

The key here is that my script will simply verify a playable file, and will protect people from keeping actual viruses/trojans (the "if your video doesn't play click here.exe" files). This script will not do anything if someone uploads a playable video that is a commercial, porn, or other than the named show etc.

[Finch106]

Couch has the From folder as D:\Complete\movie, and to D:\Media\Movies. It seems it's extracting and processing partially completed videos. They will begin playing, but any change in the timeline corrupts it and it stops playing.

Is there a setting or option that I set incorrectly? I just moved to a new system so I had to start autoProcessMedia.cfg from scratch.

Ultimately, I'm getting a random section of a movie, as a completed movie in my library.

[19:55:00] [INFO]::MAIN: Loading config from [C:\nzbToMedia\autoProcessMedia.cfg]
[19:55:00] [INFO]::MAIN: Checking database structure...
[19:55:00] [INFO]::MAIN: Checking if git needs an update
[19:55:00] [ERROR]::MAIN: Unable to determine installed version via git, please check your logs!
[19:55:00] [INFO]::MAIN: No update needed
[19:55:00] [INFO]::MAIN: nzbToMedia Version:11.03 Branch:master (Windows 10)
[19:55:00] [INFO]::MAIN: #########################################################
[19:55:00] [INFO]::MAIN: ## ..::[nzbToMedia.py]::.. ##
[19:55:00] [INFO]::MAIN: #########################################################
[19:55:00] [INFO]::MAIN: Script triggered from SABnzbd
[19:55:02] [INFO]::MAIN: Auto-detected SECTION:CouchPotato
[19:55:02] [INFO]::MAIN: Extracting D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part001.rar to D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC
[19:57:03] [INFO]::MAIN: EXTRACTOR: Extraction was successful for D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part001.rar to D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC
[19:57:06] [INFO]::MAIN: Extracting D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC_ADMIN_\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb.gz to D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC_ADMIN_
[19:57:06] [ERROR]::MAIN: EXTRACTOR: Extraction failed for D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC_ADMIN_\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb.gz. Could not call command ['D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC\ADMIN\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb.gz', '-p-']
[19:57:06] [INFO]::MAIN: Removing extracted archive Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part001.rar from folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC ...
[19:57:07] [INFO]::MAIN: Removing extracted archive Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part002.rar from folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC ...
[19:57:08] [INFO]::MAIN: Removing extracted archive Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part003.rar from folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC ...
[19:57:09] [INFO]::MAIN: Removing extracted archive Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part004.rar from folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC ...
[19:57:11] [INFO]::MAIN: Removing extracted archive Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part005.rar from folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC ...
[19:57:12] [INFO]::MAIN: Removing extracted archive Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part006.rar from folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC ...
[19:57:13] [INFO]::MAIN: Removing extracted archive Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.part227.rar from folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC ...
[19:57:14] [INFO]::MAIN: Calling CouchPotato:movie to post-process:Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb
[19:57:14] [INFO]::MAIN: Attemping imdbID lookup for Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb
[19:57:14] [INFO]::MAIN: Searching folder and file names for imdbID ...
[19:57:14] [INFO]::MAIN: Found imdbID [tt0091369]
[19:57:15] [INFO]::TRANSCODER: Checking [Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.mkv] for corruption, please stand by ...
[19:57:16] [INFO]::TRANSCODER: SUCCESS: [Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.mkv] has no corruption.
[19:57:16] [INFO]::COUCHPOTATO: Status shown as failed from Downloader, but 1 valid video files found. Setting as success.
[19:57:16] [INFO]::MAIN: Checking for files to be transcoded
[19:57:16] [POSTPROCESS]::COUCHPOTATO: Starting renamer scan for Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb
[19:57:40] [POSTPROCESS]::COUCHPOTATO: SUCCESS: Finished renamer scan for folder D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC
[19:57:40] [POSTPROCESS]::COUCHPOTATO: Checking for status change, please stand by ...
[19:57:40] [POSTPROCESS]::COUCHPOTATO: SUCCESS: Release Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb has now been marked with a status of [DONE]
[19:57:41] [INFO]::CLEANDIR: Doing Forceful Clean of D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC
[19:57:41] [INFO]::MAIN: Deleting D:\Incomplete\Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC
[19:57:42] [INFO]::MAIN: The C:\nzbToMedia\nzbToMedia.py script completed successfully.
CouchPotato: Successfully post-processed Labyrinth.(1986).2160p.4K.UltraHD.BluRay.(x265.HEVC.10bit).2CH.AAC.cp(tt0091369).nzb!

[battamo]

Hi @clinton-hall

This is a major problem, to which there is a very simple solution.

It would be great if you could make a switch to say that if the path passed by the downloader is not within the watch_dir then the download has failed.

For failed downloads Sabnzbd passes the path of the temporary download folder, which contains the failed files.

It would be flexible if the user could define one or more folder paths, within which all downloads should be failed.

It is not enough to check that the media file is playable. mediafiles with the first part of the file intact are playable, this is the reason that mediafiles can be played while they download, as long as they are downloaded sequentially.

Most failed downloads are playable.

Failed downloads which are passed from Sabnzbd can be easily distinguished by the folder path.

nzbtomedia doesn't recognize that the path passed from sabnzbd points to the temp folder.

Effectively all failed downloads, which exceed the minSize and for which the first part of the file is intact, are passed as completed.

[clinton-hall]

There are a few ways to handle this:

• Add a "Ignore" folder (e.g. E:\Downloads\incomplete)
• Add a switch to "not check failed"

Adding a check to only process from within watch_dir could be problematic as it requires the user to specify watch_dir (currently only required to assist with manual run) and it means any other changes to download location, or a change in category name etc, would break this.

If we ignore the failed download, then with some configs we end up ignoring a download that failed because of an unrar issue which could be extracted by a later unrar binary... or we ignore a download that failed because a sample was not downloadable.... So this definitely needs to be an optional switch.

[clinton-hall]

Actually, look again, I already have the option

[General]
    no_extract_failed = 1

With this option enabled, when the downloader reports this as failed, and therefore doesn't extract, this will not attempt to extract and therefore won't find any media files. This may find a playable file (only when dealing with non rared content, which shouldn't happen that often with nzbs and the script shouldn't even be triggered for a non complete torrent).

I could extend this same option to also disable the media file checking. Or I could specifically create a no_check_failed option as per above.

Can you try with the no_extract_failed option set and let me know how that goes?

[battamo]

I have not tried that option, but I do have extract disabled in the category.

I will report back, once I have had a chance to test it.

[battamo]

Hi,

I just gave it a try. It doesn't work. I don't know whether the file is unpacked by nzbtomedia or sabnzbd, but a media file is compiled in the temporary folder of the failed download, which is then post processed.

It would be fantastic to have a switch to fail the failed downloads.

Sometimes the main media file is definitely intact. However, when it isn't it will eventually fail during playback.

Personally I prefer fetching too many files over the occasional incomplete mediafile.,

[clinton-hall]

thanks for testing. can you show the logs for this so I can confirm what action nzbToMedia has taken and how therefore to best implement this check/switch?

[battamo]

Yes. I have had a couple of thoughts.

If the download has failed, do what you would do if the script had found that the file was corrupted. It might also be an option just to clean out the folder before passing the path on.

This is from re-downloading the failing nzb, which was already processed as completed. It ends up deleting the file, because it has already been processed, but before that, status was set to success.

[23:38:11] [INFO]::MAIN: Loading config from [C:\Users\\user\nzbToMedia\autoProcessMedia.cfg]
[23:38:11] [INFO]::MAIN: Checking database structure...
[23:38:11] [INFO]::MAIN: Checking if git needs an update
[23:38:13] [INFO]::MAIN: No update needed
[23:38:13] [INFO]::MAIN: nzbToMedia Version:697ae370adef98f378ad3b8109c3c2c7e52a18d5 Branch:master (Windows 10)
[23:38:13] [INFO]::MAIN: #########################################################
[23:38:13] [INFO]::MAIN: ## ..::[nzbToMedia.pyc]::.. ##
[23:38:13] [INFO]::MAIN: #########################################################
[23:38:13] [INFO]::MAIN: Script triggered from SABnzbd 0.7.23+
[23:38:15] [INFO]::MAIN: Auto-detected SECTION:SickBeard
[23:38:15] [INFO]::MAIN: Calling SickBeard:sickrage to post-process:name.S01E12.nzb
[23:38:23] [INFO]::MAIN: Attempting to auto-detect sickrage fork
[23:38:18] [INFO]::MAIN: SickBeard:sickrage fork auto-detection successful ...
[23:38:18] [INFO]::MAIN: SickBeard:sickrage fork set to SickChill
[23:38:18] [INFO]::MAIN: FLATTEN: Flattening directory: c:\Users\\Desktop\temp\name.S01E12
[23:38:18] [INFO]::TRANSCODER: Checking [name.S01E12.mkv] for corruption, please stand by ...
[23:38:18] [INFO]::TRANSCODER: SUCCESS: [name.S01E12.mkv] has no corruption.
[23:38:18] [INFO]::MAIN: Found Valid Videos. Setting status Success
[23:38:18] [POSTPROCESS]::SICKBEARD: SUCCESS: The download succeeded, sending a post-process request
[23:38:22] [POSTPROCESS]::SICKBEARD: Processing in folder c:\Users\\Desktop\temp\name.S01E12
[23:38:22] [POSTPROCESS]::SICKBEARD: Processing c:\Users\\Desktop\temp\name.S01E12
[23:38:22] [POSTPROCESS]::SICKBEARD: Processing folder c:\Users\\Desktop\temp\name.S01E12
[23:38:22] [POSTPROCESS]::SICKBEARD: You're trying to post process a video that's already been processed, skipping
[23:38:22] [POSTPROCESS]::SICKBEARD: Skipping already processed file: name.S01E12.mkv
[23:38:22] [POSTPROCESS]::SICKBEARD: Found unwanted files: [u'name.S01E12.mkv']
[23:38:22] [POSTPROCESS]::SICKBEARD: Deleting file: name.S01E12.mkv
[23:38:22] [POSTPROCESS]::SICKBEARD: Deleted folder: c:\Users\\Desktop\temp\name.S01E12
[23:38:22] [POSTPROCESS]::SICKBEARD: Successfully processed
[23:38:22] [INFO]::CLEANDIR: Directory c:\Users\\Desktop\temp\name.S01E12 has been processed and removed ...
[23:38:22] [INFO]::MAIN: The C:\Users\\user\nzbToMedia\nzbToSickBeard.py script completed successfully.
SickBeard: Successfully post-processed name.S01E12.nzb!

[clinton-hall]

Keep no_extract_failed =1 and test the latest nightly build.

[battamo]

Fantastic, it appears that did it!

[battamo]

Btw. I found an additional check that could be foreformed to check whether the file is partial. It won't detect all problems, but perhaps it would increase the hit rate for partial downloads.

mediainfo will try to report whether a file is truncated.

https://unix.stackexchange.com/questions/269342/how-to-check-if-a-video-is-completely-downloaded.