Post Processing Failing

[psychoflame]

Hey, i'm running into an issue where PP seems to be partially fail. If PP runs, it will also go to move files, which get successfully moved but then mylar reports that it failed to move and won't update the status on the file. Going into the comic and manually scanning the comics folder detects it and marks it as downloads so the files are getting properly moved. I'm running Mylar on a Ubuntu machine with the comicRN script being used and completed download handling off. Logs with me trying to manually post process in 2 files are below with debug logging but they don't seem like they'll be much help. Just throws a one line error that it failed to move.

https://pastebin.com/ehXesqBp

[evilhero]

What branch of mylar are you running and a few you up to date? There was a problem with post-processing such as what's happening with yours (failure to mark completed and tidyup old files) - but that was fixed in the last few commits.

Is the issue a story arc or a normal issue (did you initiate the snatch from the arc detail page for arcs)?

[psychoflame]

I'm on the newest commit in the development branch. The issue is with every file that gets PP, they are all normal issues that come from the pull-list, i don't use any of the fancier features like story-arcs or such. It's actually been a problem for me for awhile now every since that update that put in the option for completed download handling. I originally had freenas and it was an issue on that machine as well, but i lost that machine a few months back. Rebuilt my stuff in earlier march, but the exact same problem.

I also know it's not an permissions issue, as i have sonarr and radarr setup the exact same way to handle files like mylar and they have no issues with permissions

[evilhero]

Well you can't compare to sonarr/radarr - different methodologies entirely as well as programming languages. Alot of people set up their stuff permissions-wise for tv/movies and then everything thereafter is an afterthought thinking it's a global encompassing thing, and it's usually not that simple. Add in third-party installers, dockers, VM's, shares ...well, you get the idea, it gets complex right quick.

It honestly sounds like it's a perms issue of some kind - the only failure point during that in the PP is just a simple move operation that it's failing on, which is just doing a copy first then a delete (this is how the ubuntu mv command works). So if it's moving it fine, then it's failing to remove it from the given download location (for sab - usually it will have the sab user perms, so it can copy stuff out fine, but deleting thereafter won't happen due to the improper perm assignment).

Get the latest dev commit that just went up - specifically done to log the exact error thrown back by the OS during a file operation.

[psychoflame]

Sorry i ddn't mean to imply that they were the same on handling files, i meant that the permissions on all of them are the same. I.E they all run as the user plex as does nzbget, so permissions are all for that user. They write into a SMB share that has permissions of nobody:nogroup but plex is a part of that group and has write access to the destination folder.

I updated to the newest commit and tried doing a manual import of another regular issue and PP still failed out same way. Logs are below, but i didn't see any new logging that stands out https://pastebin.com/iuMSiZCp

[evilhero]

Ah crap - I put the catch in the wrong part. New one is up in dev now - update and try it again and it should log the exact error that's being returned.

[psychoflame]

Okay so that worked, I'm seeing this error in the log 2018-04-04 14:33:07 ERROR [MOVE] error : [Errno 1] Operation not permitted: '/home/plex/Remote/Media/Comics/DollFace (2017)/DollFace 1 (2017).cbr'

So it does look like a weird permissions issue then. I don't understand why though, it does get moved successfully, although the copy in the original position doesn't get cleaned up. Could it be because mylar is failing to clean up the copy in the original positions that it fails out?

Full logs https://pastebin.com/vJRHzAgf

[evilhero]

Well at this point, it's not even Mylar - it's just trying to do a mv command, and it's returning that error message. It has even't gotten to the cleanup/tidyup phase yet ;)

Try to do an su - plex from the cli, and then perform a mv command from within that specific location (even perhaps the file) - what it does is the command will copy the file, and then delete the orginal thereafter once the copy is completed. You'll get the same error in the shell that Mylar is returning when it attempts as it seems to be failing on trying to delete the file after the copy succeeded.

What are the permissions on the file it's trying to move (from the original location)? That might be the stumbling block there that the permissions are being enforced by the downloader (ie. it assigns the user/group based when it creates the file when downloading).

[psychoflame]

Yep you were right. Trying to move is throwing this error

mv: preserving times for '/home/plex/Remote/Media/Comics/tarot.cbr': Operation not permitted
mv: preserving permissions for ‘/home/plex/Remote/Media/Comics/tarot.cbr’: Input/output error

Some quick google-fu tells me that this is an issue, cause my destination folder is on a different filesystem so instead i need to use cp to copy file to destination to get around these issues. Using cp from the CL works just fine, i was able to copy the file over to the destination folder with no errors. However changing mylar to copy files over still throws errors. Logs below

https://pastebin.com/jGzqVmDU

https://unix.stackexchange.com/questions/131180/how-to-move-a-file-without-preserving-permissions

Edit: I forgot to mention that the owner is plex:plex and he has r/w on the file from the original download location. nzbget assigns the right ownership on the files it creates

Edit 2: It seems like the file attributes is whats messing up when trying to mv or cp files. running a cp --no-preserve all, cp fails out similar to move because it can't save the attributes of the file.

cp: setting permissions for ‘/home/plex/Remote/Media/Comics/tarot.cbr’: Input/output error

[evilhero]

Hmm, interesting..maybe we should be using copyfile instead of copy. The copyfile option will not attempt to copy existing permission bits, whereas copy will - but there are those that requested the permissions be inherited.....hmmm.

Curious, what is your /etc/fstab entry for the smb mount ? Do you have a uid entry in the mount line, and what is your file_mode, dir_mode set to ?

[psychoflame]

No i don't define any uid entry, i let it mount as nobody:nogroup and have plex as a user in nogroup. Allows me to simplify my permissions across my entire setup. I also don't define file_mode or dir_mode in my mount command, it's pretty simple. I do have full read,write,execute over all files in mount as a ls -l shows.

Mount Command //10.10.10.50/Media /home/plex/Remote/Media/ cifs username=****,password=****,iocharset=utf8,sec=ntlm 0 0

[evilhero]

Have you tried specifying a UID, GID for the mount ? CIFS does not generally have any concept of user and group, so mounting a cifs share will default to showing user and group as nobody. Since you are not nobody Linux will not let you write to anything that doesn't have 0777 permission unless you're use sudo with the command. By adding uid=mylogin,gid=mygroup to fstab, it will set the default UID, GID on newly created files by default or when the UID/GID are wrong. Might also want to try using noperm in addition so that permissions checks aren't being performed when copying/moving files.

Apparently, time and permissions can only be set by the owner, so mounting with the UID would make all files (including newly created ones) on that mount automatically be set to the UID as owner (ie. the UID for plex), so it might fix things just by adding the correct UID for the plex user.

[hjone72]

Hi, I think I am having a similar issue (Happy to open a different issue ticket if it is different). I just pulled the latest Dev commit, see log snippet below.

My issue seems to be happening at the cache folder. For testing I've set the permissions of the downloads folder to 777 and ensured that the downloaded file is 777. When MyLar processes the folder, it creates a temp folder in opt/mylar/cache. The created folder gets the permissions 700 and the files within the folder have 777 but I'm still getting the permission error.

2018-04-05 17:54:28 | ERROR | [POST-PROCESSING] Post-Processing ABORTED
2018-04-05 17:54:28 | ERROR | [POST-PROCESSING] Failed to move /opt/mylar/cache/mylar_oZfGpa/52 037 2007 Secret Identities The Origin of Firestorm.cbz:
2018-04-05 17:54:28 | ERROR | [MOVE] error : [Errno 1] Operation not permitted: '/mnt/media/comics/52 (2006)/52 Week #037 (2007).cbz'
2018-04-05 17:54:27 | DEBUG | [FREESPACE-CHECK] /mnt/media/comics/52 (2006) has 937.1GiB free
2018-04-05 17:54:27 | DEBUG | [POST-PROCESSING] Renaming /opt/mylar/cache/mylar_oZfGpa/52 037 2007 Secret Identities The Origin of Firestorm.cbz ..to.. /opt/mylar/cache/mylar_oZfGpa/52 Week #037 (2007).cbz

Thanks,

[evilhero]

@hjone72 : What do you have your permissions set to within Mylar (are you enforcing permissions using Mylar?) Does the file get moved from the cache to the series directory ok?

Unless you're moving files across different filesystems, yours might be different - as @psychoflame is moving the files involving an smb share.

[psychoflame]

Sorry for the late reply, got sidetracked and couldn't get back to my box. I tried setting the uid and gid with the noperm option enabled and i still get errors thrown when trying to move files over to the mount. it's still throwing this error

mv: preserving permissions for ‘/home/plex/Remote/Media/Comics/tarot2.cbr’: Input/output error

Setting the uid and gid works, checking permissions on the files shows them set as plex:plex

Doing a copy from the cl works fine, however trying to use the copy method from mylar still throws an error. Logs below https://pastebin.com/543ywex9

[evilhero]

does doing a cp -p src dst work from the cli ? I'm trying to figure out which command to use exactly, as there are a few different ones within python - it sounds like you're probably needing copyfile instead of copy (which will just copy the file, and not any metadata attributes).

If you're feeling adventurous, fire up your favourite editor (ie. nano) and edit the helpers.py file in the mylar subdirectory. On line 3742, change: shutil.copy( path , dst ) to: shutil.copyfile( path , dst ) save the file and then try the pp-again. With any luck, that will copy the file and not the attributes which is still the stumbling block.

There's also the shutil.copy2 command which follows the same pattern as the shutil.copy and shutil.copyfile, but the copy2 copies the permissions over thereafter which would probably cause the same problems as the copy command that's in Mylar currently.

[psychoflame]

Okay i tried the cp -p command. Throws the same error cp: preserving permissions for ‘/home/plex/Remote/Media/Comics/deadpool.cbr’: Input/output error

I also edited the helpers.py file changed line 3751 to shutil.copyfile but it still threw up a permissions issues. Logs Below

https://pastebin.com/96TqJDYH

Should i try using copy2 as well. Looking at how the different shutil copy options preserve metadata, i don't think it'd work either.

[psychoflame]

I got it working. I needed to set the smb version to 2.1 to get permissions working correctly. Here's a mount output of all the settings i had to set to get permissions on copy and such to work. Unsure if you want me to close this issue so i'll leave it up to you evilhero. Thanks for all your help in debugging the issue

//*******/Media on /home/plex/Remote/Media type cifs (rw,relatime,vers=2.1,sec=ntlmssp,cache=strict,username=*****,domain=MEDIA,uid=1000,forceuid,gid=1000,forcegid,addr=******,file_mode=0777,dir_mode=0777,nounix,serverino,mapposix,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)

[hjone72]

@evilhero sorry for the delayed reply. I'm using an OverlayFS mount. Would MyLar count that as a different filesystem?

[webheaded]

I started having this issue recently as well. I am on Arch Linux and had just decided to redownload mylar using AUR for whatever reason and I think the package changed. I went through and moved the downloaded AUR version from MASTER to a backup folder and checked out the GIT development branch (which thankfully provided a better error and allowed me to find this page). On a hunch, changed the service file to run as root and now it works again (I think my install was previously running as root).

I then went back and used the method psychoflame mentioned mounting the drive and voila. It was a permissions issue. I think before when it was for some reason running mylar as root, it didn't care about the permissions so I never had issues. I never have the forceuid and forceguid settings active either so it mounted the drive as root. Despite the folder permissions being 777 for the mount, it didn't actually seem to care and still errored out. So this should do it hopefully. :)