SABnzbd Sonarr response: started instead of Sonarr response: queued.

[DamenQuixotic]

@mdhiggins Im running into this issue now also, but its not constant its random :

_Originally posted by @SudoUser-Sam in https://github.com/mdhiggins/sickbeard_mp4_automator/issues/881#issuecomment-553248224_

---

I am still encounting this exact issue. Its random when it happens but if I download a season/multiple seasons of a show many of them end up with this result. If they get the "started" response Sonarr never picks them up. If I keep retrying to search for the missing episodes they will eventually all work but that's kinda against the point of MP4 Automator to have to babysit it.

[morpheus2n2]

lol no worries will test it out tonight

[DamenQuixotic]

When trying to download most of the jobs get pp-failure, then after sitting for a bit most of them get converted to success. This did not happen before and the only thing I see on the NZBGet side of things is the generic message...

ERROR | Mon Aug 17 2020 10:01:40 | Post-process-script NZBGetPostProcess.py for Doug.S01E09.DSR.x264-REGRET failed (terminated with unknown status)

If I search that timestamp in the sma.log there are no apparent errors there.

[mdhiggins]

Anything in sma.log about nzbget's failure? was nzbgetpostprocess ever working correctly?

[DamenQuixotic]

Yes it was always post processing correctly as far as I could tell.

NZBGet still has episode 24-25 marked as PP-FAILURE, and here that part of the sma.log that looks normal, its also imported into sonarr fine.

https://pastebin.com/MGupfEcn

Here is what my NZBGet history looks like after trying to add a whole season:

Here is what sonarr looks like after:

Nothing else is downloading/all activity is done in both sonarr and nzbget.

[mdhiggins]

Is there anything in the next few lines of sma.log? looks like its working so its probably just a weird return issue with the new technique but it cuts off a little too soon

[DamenQuixotic]

No it just starts the next episode:

2020-08-17 09:59:40 - NZBGetPostProcess - INFO - Script triggered from NZBGet (11.0 or later).
2020-08-17 09:59:40 - NZBGetPostProcess - DEBUG - Path: C:\ProgramData\NZBGet\complete\Doug.S01E11.WEB.h264-WALT-xpost
2020-08-17 09:59:40 - NZBGetPostProcess - DEBUG - NZB: Doug.S01E11.WEB.h264-WALT-xpost.nzb
2020-08-17 09:59:40 - NZBGetPostProcess - DEBUG - Category: sonarr
2020-08-17 09:59:40 - NZBGetPostProcess - DEBUG - Categories: ['sickbeard', 'couchpotato', 'sonarr', 'radarr', 'sickrage', 'bypass']
2020-08-17 09:59:40 - resources.readsettings - INFO - C:\Python38\python.exe
2020-08-17 09:59:40 - resources.readsettings - DEBUG - ConfigFile specified is a directory, joining with autoProcess.ini.
2020-08-17 09:59:40 - resources.readsettings - INFO - Loading config file C:\MP4_Automator\config\autoProcess.ini.
2020-08-17 09:59:40 - NZBGetPostProcess - INFO - Processing file C:\ProgramData\NZBGet\complete\Doug.S01E11.WEB.h264-WALT-xpost\doug.s01e11.web.h264-walt.mkv.
2020-08-17 09:59:40 - NZBGetPostProcess - DEBUG - Process started.

[mdhiggins]

Found the problem a7cfe558be3339dbeda1507ea4574abf18f0785d

That should fix it, forgot to pass the prefixes to the new function

[DamenQuixotic]

Like I said before I know nothing of python, but now that you are returning a function accessAPI at the end of processEpisode do you have to do any sort of await on that?

--I will leave this comment but I did just see you responded with a fix--

[mdhiggins]

Nah no await needed, it was just causing an exception because of that undefined variables, let me know if that fixes things

[DamenQuixotic]

Okay, so I have done some more batch downloads with these changes and it looks promising.

I do see some instances of the paths not matching and it retrying. When it does have to retry it looks like it might be taking all three times so you might want to extend the time between tries, not sure. If you want to leave it as is for now I can bring it up to you if it seems like some are failing too many times for the current parameters you have setup.

All the episodes made it back into Sonarr with these changes, the "complete" folder which downloads get put into didn't get cleared out completely though for some reason. I could potentially circumvent this issue by changing NZBGets history to 1 day which will clear out the remnants after a day. When trying a season without any post processing scripts this folder did not have any remnants sticking around.

What's odd about the files left is that they aren't even from the file that was effected by the retries. Episode 4 is the one that had to retry in the batch above. Maybe the move requests get interrupted by the retries, not sure. I tried to look through the log to see if it had trouble removing any of the folders listed but I didn't see any issues posted.

I did also try a season of something that did require actual post processing from ffmpeg and since there is a minute or so between the calls they don't seem to fail, and the complete folder gets cleared out as expected.

[mdhiggins]

Yeah if you have some logs of the retry I'll take a look, I think I set it for 5 seconds but that might not be enough time

[mdhiggins]

Pushed update that increases retry to 5 and sleep to 10

[powerdude]

hey, i may also be experiencing this, but i'd like to not try and download the whole series again to get logs since they are 16-20GB each. Is there a way to "re-trigger" SMA/Sonarr to pick up the series with this fix in play?

[mdhiggins]

Probably don't need to I believe its been solved with the latest update. The API calls happen as part of the process via the downloader scripts so there's not really a way to recreate it artificially that wouldn't be a lot of work; manual import doesn't do it.

That being said if you just need to get some files into Sonarr just use the manual import option inside Sonarr and point it to your directory and that should bring everything in. postSonarr.py will handle the conversion if that's set up.

[DamenQuixotic]

I have sent you an email with the sma.log for only this season 1 batch where there were some instances of wrong path responses from sonarr. All files got imported into sonarr, nothing looks errored out in nzbget, yet 6 items were left in the completed folder.

[mdhiggins]

Are the items left in the completed folder empty or are all the original files still left behind?

[DamenQuixotic]

The folders all have their original files in them, or what I would think are the original files. Not sure how I can tell exactly because they were already in mp4 format.

[mdhiggins]

Normally I leave the importMode alone which defaults to 'auto' and that usually moves the files for nzb however maybe this glitch is causing 'auto' to copy instead of move.

177461005020e0e5520d7927aefb7eaeb0468712

That update explicitly sets the importMode to move (you'll need to update your NZBGetPostProcess.py file). See if that fixes it. Then hopefully this will all be sorted out.

[powerdude]

@mdhiggins Hmm, i just did an import from my download directory. Sonarr brought in the files, but SMA didn't run. I'm using v3 and i do have postSonarr setup to run "On Import" and "On Upgrade". Is there something else i'm missing?

[DamenQuixotic]

Normally I leave the importMode alone which defaults to 'auto' and that usually moves the files for nzb however maybe this glitch is causing 'auto' to copy instead of move.

1774610

That update explicitly sets the importMode to move (you'll need to update your NZBGetPostProcess.py file). See if that fixes it. Then hopefully this will all be sorted out.

Still having some issues but I do see two instances of permission errors. One of them is for a file that looks like it actually did get moved out of completed. Another instance is for a file that is still there. Then there are 5 other folders/files still in completed that don't have this error appear for them.

2020-08-17 13:17:02 - resources.mediaprocessor - DEBUG - P:\Plex\TV\Doug\Season 2\Doug - S02E09 - Doug Battles The Rulemeister.2.mp4 created from P:\Plex\TV\Doug\Season 2\Doug - S02E09 - Doug Battles The Rulemeister.mp4 successfully.
2020-08-17 13:17:02 - resources.mediaprocessor - DEBUG - Attempting to remove P:\Plex\TV\Doug\Season 2\Doug - S02E09 - Doug Battles The Rulemeister.mp4.
2020-08-17 13:17:02 - resources.mediaprocessor - ERROR - Unable to remove or replace file P:\Plex\TV\Doug\Season 2\Doug - S02E09 - Doug Battles The Rulemeister.mp4.
Traceback (most recent call last):
  File "C:\MP4_Automator\resources\mediaprocessor.py", line 1559, in removeFile
    os.remove(filename)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'P:\\Plex\\TV\\Doug\\Season 2\\Doug - S02E09 - Doug Battles The Rulemeister.mp4'

I have not touched the permission settings in the autoprocess.ini:

[Permissions]
chmod = 0644
uid = -1
gid = -1

I also see some instances of the following error:

2020-08-17 13:18:50 - SonarrPostProcess - INFO - Final state: queued.
2020-08-17 13:18:50 - SonarrPostProcess - DEBUG - {'name': 'RescanSeries', 'body': {'seriesId': 35, 'sendUpdatesToClient': True, 'updateScheduledTask': True, 'completionMessage': 'Completed', 'name': 'RescanSeries', 'trigger': 'manual'}, 'priority': 'normal', 'status': 'queued', 'queued': '2020-08-17T18:17:50.0484843Z', 'trigger': 'manual', 'state': 'queued', 'manual': True, 'startedOn': '2020-08-17T18:17:50.0484843Z', 'sendUpdatesToClient': True, 'updateScheduledTask': True, 'id': 14168}
2020-08-17 13:18:50 - SonarrPostProcess - ERROR - Rescan command timed out
2020-08-17 13:18:50 - SonarrPostProcess - ERROR - Sonarr monitor status update failed.
Traceback (most recent call last):
  File "C:\MP4_Automator\postSonarr.py", line 163, in <module>
    sys.exit(1)
SystemExit: 1

[mdhiggins]

@powerdude That should be all you need

That's mine and it always runs on manual import Does postSonarr.py always run normally? A quick way to know is if you have tagging since that's responsible for tagging

Check sonarr logs and see if there's an error preventing it from starting but it should definitely run on manual import if set up correctly

[mdhiggins]

@DamenQuixotic The first error has nothing to do with NZBGet and looks like its trying to reprocess an mp4 file that's already been converted. Having seen your log files it looks like you have force-convert turned on which you shouldn't if you're using both a downloader script with postSonarr (you'll end up processing everything twice). Turn that off. That permission error is because plex is already working with your file btw not a true permission error but more of a 'file in use' error.

The SonarrPostProcess error is different than what you had, that's a different later sonarr API call for rescanning after conversion and is unrelated to what's going on now (though certainly seems like sonarr is having lots of API failures for you, may want to maybe reinstall or try v3).

Did reprocessing things from scratch with that last update make any difference in what files were left in the download folder?

[powerdude]

v3 looks different:

I'm assuming it runs normally since it works when single episodes are brought in through normal means. I deleted the series and tried manual import again. This is all i see in the Sonarr logs:

20-8-17 15:02:10.3|Info|RefreshSeriesService|Updating Baki the Grappler
20-8-17 15:02:11.2|Info|RefreshEpisodeService|Starting episode info refresh for: [79237][Baki the Grappler]
20-8-17 15:02:11.3|Info|RefreshEpisodeService|Finished episode refresh for series: [79237][Baki the Grappler].
20-8-17 15:02:11.3|Info|MediaCoverService|Downloading Banner for [79237][Baki the Grappler] https://artworks.thetvdb.com/banners/graphical/g1876i-g.jpg
20-8-17 15:02:11.3|Info|DiskScanService|Scanning Baki the Grappler
20-8-17 15:02:12.3|Info|MediaCoverService|Downloading Poster for [79237][Baki the Grappler] https://artworks.thetvdb.com/banners/posters/79237-1.jpg
20-8-17 15:02:13.4|Info|MediaCoverService|Downloading Fanart for [79237][Baki the Grappler] https://artworks.thetvdb.com/banners/fanart/original/79237-2.jpg
20-8-17 15:02:15.6|Info|DiskScanService|Completed scanning disk for Baki the Grappler
20-8-17 15:02:15.6|Info|SeriesScannedHandler|[Baki the Grappler] was recently added, performing post-add actions
20-8-17 15:02:17.1|Info|ExistingMetadataImporter|Found 102 existing metadata files
20-8-17 15:02:17.3|Info|ExistingSubtitleImporter|Found 0 existing subtitle files
20-8-17 15:02:17.5|Info|ExistingOtherExtraImporter|Found 16 existing other extra files
20-8-17 15:02:17.5|Info|ExistingExtraFileService|Found 0 extra files

[DamenQuixotic]

@DamenQuixotic The first error has nothing to do with NZBGet and looks like its trying to reprocess an mp4 file that's already been converted. Having seen your log files it looks like you have force-convert turned on which you shouldn't if you're using both a downloader script with postSonarr (you'll end up processing everything twice). Turn that off. That permission error is because plex is already working with your file btw not a true permission error but more of a 'file in use' error.

The SonarrPostProcess error is different than what you had, that's a different later sonarr API call for rescanning after conversion and is unrelated to what's going on now (though certainly seems like sonarr is having lots of API failures for you, may want to maybe reinstall or try v3).

Did reprocessing things from scratch with that last update make any difference in what files were left in the download folder?

I had force-convert turned on because I wanted to process a file even if it was an mp4 to update the audio codecs to be more universally compatible. I thought when I turned on process-same-extensions it told me that force-convert must also be set to true. I could be completely mistaken about what process-same-extensions is meant for then if that is not the case.

My Sonarr install is only a week or so old, but I can try updating to V3 and see if maybe their API is more mature there. Reprocessing with the forced importMode didn't seem to have an impact on the folder/files leftover in the completed download path.

[DamenQuixotic]

Okay I am seeing something here now. I have turned off force-convert and process-same-extensions and if I search the files that remain in the complete folder by:

NZBGetPostProcess - INFO - Starting conversion.
NZBGetPostProcess - DEBUG - Input directory: C:\ProgramData\NZBGet\complete\Doug.S02E07...

If I search with that sort of method all the ones that are still in the complete folder get two results, everything else only gets one. So its like its still processing some of them twice for some reason.

Let me know if you want the whole log file, this is a consistent thing if I blast the season and try again it still happens.

[mdhiggins]

You don't need force convert.

Process same extensions will evaluate incoming files to see if it's been processed by SMA before (it leaves a tag in the metadata) and will skip those files. Force convert always converts regardless.

[DamenQuixotic]

Okay maybe its something odd with specific downloads, see below this download always has a normal mkv and a -duplicate1 mkv.

2020-08-17 14:30:47 - NZBGetPostProcess - INFO - Starting conversion.
2020-08-17 14:30:47 - NZBGetPostProcess - DEBUG - Input directory: C:\ProgramData\NZBGet\complete\Doug.S02E01.Doug.Takes.the.Case-Dougs.Secret.Song.480p.WEB-DL.AAC2.0.H.264-SA89-xpost.
2020-08-17 14:30:47 - NZBGetPostProcess - DEBUG - File name: Doug.S02E01.Doug.Takes.the.Case.-.Doug's.Secret.Song.480p.WEB-DL.AAC2.0.H.264-SA89.

2020-08-17 14:30:46 - NZBGetPostProcess - INFO - Starting conversion.
2020-08-17 14:30:46 - NZBGetPostProcess - DEBUG - Input directory: C:\ProgramData\NZBGet\complete\Doug.S02E01.Doug.Takes.the.Case-Dougs.Secret.Song.480p.WEB-DL.AAC2.0.H.264-SA89-xpost.
2020-08-17 14:30:46 - NZBGetPostProcess - DEBUG - File name: Doug.S02E01.Doug.Takes.the.Case.-.Doug's.Secret.Song.480p.WEB-DL.AAC2.0.H.264-SA89.duplicate1.

It might be converting both but only one needs to make it into Sonarr of course so the other gets left behind.

[mdhiggins]

@powerdude That's looks to be set up correctly and should be running unless the script is erroring out for some reason after being launched. Are you confident it's running via the usual methods? Is there logs in sma.log for postSonarr running either via manual import or a usual file?

[mdhiggins]

@DamenQuixotic There's no code that would add -duplicate to any file that's not the naming convention that's used by this script so that's gotta be coming from somewhere else.

[DamenQuixotic]

@DamenQuixotic There's no code that would add -duplicate to any file that's not the naming convention that's used by this script so that's gotta be coming from somewhere else.

Yeah I don't think it is an issue with your script, I just don't know why a download would have two video files in it to convert for some reason. I don't know if there is a way around it from the scripts side as its just scanning the dirs and converting what it finds.

I don't know too much about NZBGet but there is something called duplicate handling but I don't know if that is entirely related or not. I will keep messing around with it but the original issue I brought up seems to be fixed now with your changes.

[mdhiggins]

@DamenQuixotic Gotcha good stuff well glad we at least got a workaround for that issue. Hopefully Sonarr fixes things or maybe already did in V3 but at least this should consistently get things processed.

So the importMode change didn't do anything? If not I'll probably just undo that change and leave it on auto but just want to confirm before I revert things

[DamenQuixotic]

Correct, the importMode didn't change anything as the issue is actually these duplicate files being converted and being left around.

[powerdude]

@mdhiggins yes, there are logs in sma.log when it runs normally. Here is a run from earlier this morning:

2020-08-17 04:38:32 - resources.mediaprocessor - INFO - Starting conversion.
2020-08-17 04:38:32 - resources.mediaprocessor - INFO - FFmpeg command:
2020-08-17 04:38:32 - resources.mediaprocessor - INFO - ======================
2020-08-17 04:38:32 - resources.mediaprocessor - INFO - /usr/local/bin/ffmpeg -fix_sub_duration -i /tv/some show/Season 07/some show - S07E21 - Episode 200.mkv -vcodec libx264 -map 0:0 -vb 613k -metadata:s:v BPS=613000 -metadata:s:v BPS-eng=613000 -metadata:s:v title=HD -c:a:0 aac -map 0:1 -ac:a:0 2 -b:a:0 256k -metadata:s:a:0 BPS=256000 -metadata:s:a:0 BPS-eng=256000 -metadata:s:a:0 title=Stereo -metadata:s:a:0 language=eng -disposition:a:0 +default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired-captions -strict experimental -c:s:0 mov_text -map 0:2 -metadata:s:s:0 title= -metadata:s:s:0 language=eng -disposition:s:0 -default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired-captions -f mp4 -threads 0 -metadata:g encoding_tool=SMA -y /tv/some show/Season 07/some show - S07E21 - Episode 200.mp4
2020-08-17 04:38:32 - resources.mediaprocessor - INFO - ======================
2020-08-17 04:47:28 - resources.mediaprocessor - INFO - /tv/some show/Season 07/some show - S07E21 - Episode 200.mp4 created.
2020-08-17 04:47:28 - resources.mediaprocessor - INFO - Tagging /tv/some show/Season 07/some show- S07E21 - Episode 200.mkv with TMDB ID 60694.
2020-08-17 04:47:28 - resources.metadata - INFO - Tagging file: /tv/some show/Season 07/some show - S07E21 - Episode 200.mp4.
2020-08-17 04:47:28 - resources.metadata - INFO - Trying to write tags.
2020-08-17 04:47:29 - resources.metadata - INFO - Tags written successfully using mutagen.
2020-08-17 04:47:29 - resources.mediaprocessor - INFO - Relocating MOOV atom to start of file.
2020-08-17 04:47:29 - qtfaststart - INFO - Removing free atom at 32 (8 bytes)
2020-08-17 04:47:29 - qtfaststart - INFO - Patching stco with 53935 entries
2020-08-17 04:47:29 - qtfaststart - INFO - Patching stco with 54375 entries
2020-08-17 04:47:29 - qtfaststart - INFO - Patching stco with 1187 entries
2020-08-17 04:47:29 - qtfaststart - INFO - Writing output...
2020-08-17 04:47:33 - resources.mediaprocessor - INFO - Plex refreshed: show
2020-08-17 04:47:33 - SonarrPostProcess - INFO - Sonarr response RescanSeries command: ID 1103152 queued.
2020-08-17 04:47:33 - SonarrPostProcess - INFO - Requesting episode information from Sonarr for series ID 315.
2020-08-17 04:47:33 - SonarrPostProcess - INFO - State: started.
2020-08-17 04:47:43 - SonarrPostProcess - INFO - Final state: completed.
2020-08-17 04:47:43 - SonarrPostProcess - INFO - Rescan command completed
2020-08-17 04:47:43 - SonarrPostProcess - INFO - Requesting updated episode information from Sonarr for series ID 315.
2020-08-17 04:47:43 - SonarrPostProcess - INFO - Sonarr monitoring information updated for episode Episode 200.
2020-08-17 04:47:43 - SonarrPostProcess - INFO - Sonarr response RenameSeries command: ID 1103154 queued.

[mdhiggins]

@powerdude Very weird that should definitely be running then. And sorry for a million questions but could you just confirm the process you used for the manual import?

[DamenQuixotic]

For archival sakes, apparently Sonarr is aware of the duplicate file issue and does not plan to fix it as far as 2015 was concerned.

duplicate video issue

[powerdude]

sure, no worries as i may be doing it wrong or not as expected. I simply go through Sonarr and do "Series" => "Import". Is there a different manual import process?

[mdhiggins]

@powerdude Ah yes there is actually. Go to Sonarr > Wanted > Manual Import

It's actually a different process and that's probably why it's not working.

For the files that you need to import they just need to be in a different folder than their final destination folder (causes issues, I usually just import from my downloads folder)

[mdhiggins]

@DamenQuixotic hopefully it was just a fluke from all the bulk testing but appreciate you sharing.

Did you ever open up an issue or forum post for Sonarr about the API issue?

[powerdude]

ahh, LOL, that makes sense. Ok, let me move files back to download folder and try it.

[DamenQuixotic]

@DamenQuixotic hopefully it was just a fluke from all the bulk testing but appreciate you sharing.

Did you ever open up an issue or forum post for Sonarr about the API issue?

I did not open an issue with them

[mdhiggins]

Alright I'll make one at some point but if you beat me to it I'll chime in just link it here.

Gonna close this once we make sure @powerdude is all set

[powerdude]

ok, i think i'm good. Files are importing and converting. I can't believe i never knew that and have been using this for a couple years i think now and doing things the "hard" way.

Hopefully the stuff you guys sorted out will help with the original issue with the season pack not being imported in the first place.

[mdhiggins]

Good stuff. Closing this for now but feel free to continue discussion or reopen with issues. If anyone makes a ticket with sonarr link it here and I'll do the same if I do.

[powerdude]

@mdhiggins sorry if this is a noob question, but when you say "tagging", what do you mean by that?

[mdhiggins]

Adding the metadata and artwork

[powerdude]

through SMA or through Sonarr. I'm using "Kodi (XBMC) / Emby" from Sonarr and i don't think i've changed anything in the autoprocess.ini.

[mdhiggins]

Neither. It writes the metadata and artwork directly into the file. MP4s (and to a lesser extent MKVs) support embedding that information in the file itself.

[powerdude]

How is this behavior configured?

[mdhiggins]

There's a metadata section in autoProcess.ini but it's all enabled by default.

Only scripts coming from media managers (Sonarr/Radarr etc) do tagging though because those get passed the ID of the movie or episode. It's the last step. Post downloaded scripts (SAB/NZBGet/Deluge etc) do not tag by design.