search

sabnzbd / sabnzbd

SABnzbd - The automated Usenet download tool
http://sabnzbd.org
Other
2.33k stars 340 forks source link

macos: ignore resource fork files #2380

Closed thezoggy closed 1 year ago

thezoggy commented 1 year ago

user on discord using 3.7.1 reported unrar failing on test download with "is not RAR archive" due to the ._* resource forked version of the rar file being tried which of course is not an actual rar file:

logs:

...
2022-12-19 16:22:53,339::INFO::[directunpacker:315] DirectUnpacked volume 10 for sometestfile-100MB
2022-12-19 16:22:53,549::INFO::[directunpacker:315] DirectUnpacked volume 11 for sometestfile-100MB
2022-12-19 16:22:53,559::INFO::[directunpacker:239] DirectUnpack completed for sometestfile-100MB
2022-12-19 16:22:53,559::DEBUG::[directunpacker:248] DirectUnpack Unrar output 
UNRAR 6.12 freeware      Copyright (c) 1993-2022 Alexander Roshal

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part01.rar

Extracting  /Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB-explanation.txt  OK
Extracting  /Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part02.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part02.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part03.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part03.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part04.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part04.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part05.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part05.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part06.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part06.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part07.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part07.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part08.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part08.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part09.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part09.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part10.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part10.rar

...         100MB.bin
Insert disk with /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part11.rar [C]ontinue, [Q]uit

Extracting from /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part11.rar

...         100MB.bin                                                 OK
All OK
2022-12-19 16:22:54,428::DEBUG::[nzbstuff:1863] Saving attributes {'cat': '*', 'pp': 3, 'script': 'None', 'priority': 2, 'final_name': 'test_download_100MB', 'password': '', 'url': 'https://sabnzbd.org/tests/test_download_100MB.nzb'} for test_download_100MB
2022-12-19 16:22:54,441::INFO::[postproc:370] Starting Post-Processing on test_download_100MB => Repair:True, Unpack:True, Delete:True, Script:None, Cat:*
2022-12-19 16:22:54,441::INFO::[notifier:123] Sending notification: Post-processing - test_download_100MB (type=pp, job_cat=*)
2022-12-19 16:22:54,442::INFO::[postproc:724] Starting verification and repair of test_download_100MB
2022-12-19 16:22:54,443::INFO::[filesystem:1137] [sabnzbd/postproc.py.parring] /Volumes/External 4/Completed/test_download_100MB.3/__ADMIN__/__verified__ missing
2022-12-19 16:22:54,443::DEBUG::[postproc:730] Verified sets: {}
2022-12-19 16:22:54,443::INFO::[postproc:747] Running verification and repair on set sometestfile-100MB
2022-12-19 16:22:54,446::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part07.rar OK
2022-12-19 16:22:54,446::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part10.rar OK
2022-12-19 16:22:54,447::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part02.rar OK
2022-12-19 16:22:54,447::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part06.rar OK
2022-12-19 16:22:54,447::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part09.rar OK
2022-12-19 16:22:54,447::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part03.rar OK
2022-12-19 16:22:54,447::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part08.rar OK
2022-12-19 16:22:54,447::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part05.rar OK
2022-12-19 16:22:54,447::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part01.rar OK
2022-12-19 16:22:54,448::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part04.rar OK
2022-12-19 16:22:54,448::DEBUG::[newsunpack:2001] Quick-check of file sometestfile-100MB.part11.rar OK
2022-12-19 16:22:54,448::INFO::[newsunpack:1123] Quick-check for sometestfile-100MB is OK, skipping repair
2022-12-19 16:22:54,450::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.par2_repair] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.par2
2022-12-19 16:22:54,455::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.par2_repair] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.vol000+001.par2
2022-12-19 16:22:54,458::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.par2_repair] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.vol003+004.par2
2022-12-19 16:22:54,462::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.par2_repair] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.vol001+002.par2
2022-12-19 16:22:54,466::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.par2_repair] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.vol007+008.par2
2022-12-19 16:22:54,472::DEBUG::[filesystem:494] build_filelists(): joinables: []
2022-12-19 16:22:54,472::DEBUG::[filesystem:495] build_filelists(): zips: []
2022-12-19 16:22:54,472::DEBUG::[filesystem:496] build_filelists(): rars: ['/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part11.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part11.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part01.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part01.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part02.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part02.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part03.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part03.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part04.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part04.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part05.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part05.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part06.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part06.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part07.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part07.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part08.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part08.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part09.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part09.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part10.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part10.rar']
2022-12-19 16:22:54,473::DEBUG::[filesystem:497] build_filelists(): 7zips: []
2022-12-19 16:22:54,473::DEBUG::[filesystem:498] build_filelists(): ts: []
2022-12-19 16:22:54,473::DEBUG::[postproc:802] Verified sets: {'sometestfile-100MB': True}
2022-12-19 16:22:54,473::DEBUG::[filesystem:1108] [sabnzbd/postproc.py.parring] Saving data for __verified__ in /Volumes/External 4/Completed/test_download_100MB.3/__ADMIN__
2022-12-19 16:22:54,491::INFO::[postproc:805] Verification and repair finished for test_download_100MB
2022-12-19 16:22:54,492::INFO::[downloader:416] Forcing disconnect
2022-12-19 16:22:54,492::INFO::[filesystem:319] Checking if any resulting filenames need to be sanitized
2022-12-19 16:22:54,495::INFO::[postproc:428] Running unpacker on test_download_100MB
2022-12-19 16:22:54,510::DEBUG::[filesystem:494] build_filelists(): joinables: []
2022-12-19 16:22:54,510::DEBUG::[filesystem:495] build_filelists(): zips: []
2022-12-19 16:22:54,510::DEBUG::[filesystem:496] build_filelists(): rars: ['/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part11.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part11.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part01.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part01.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part02.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part02.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part03.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part03.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part04.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part04.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part05.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part05.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part06.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part06.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part07.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part07.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part08.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part08.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part09.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part09.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part10.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part10.rar']
2022-12-19 16:22:54,511::DEBUG::[filesystem:497] build_filelists(): 7zips: []
2022-12-19 16:22:54,511::DEBUG::[filesystem:498] build_filelists(): ts: []
2022-12-19 16:22:54,511::INFO::[newsunpack:293] Unrar starting on /Volumes/External 4/Completed/test_download_100MB.3
2022-12-19 16:22:54,511::DEBUG::[newsunpack:517] Rar_sets: {'sometestfile-100MB': ['/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part11.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part01.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part02.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part03.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part04.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part05.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part06.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part07.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part08.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part09.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part10.rar'], '._sometestfile-100MB': ['/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part11.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part01.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part02.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part03.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part04.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part05.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part06.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part07.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part08.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part09.rar', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part10.rar']}
2022-12-19 16:22:54,511::INFO::[newsunpack:554] Set sometestfile-100MB completed by DirectUnpack
2022-12-19 16:22:54,512::DEBUG::[newsunpack:577] rar_unpack(): Rars: ['/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part01.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part02.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part03.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part04.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part05.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part06.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part07.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part08.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part09.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part10.rar', '/Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part11.rar']
2022-12-19 16:22:54,512::DEBUG::[newsunpack:578] rar_unpack(): Newfiles: ['/Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB-explanation.txt', '/Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB.bin']
2022-12-19 16:22:54,512::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part01.rar
2022-12-19 16:22:54,517::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part02.rar
2022-12-19 16:22:54,522::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part03.rar
2022-12-19 16:22:54,526::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part04.rar
2022-12-19 16:22:54,531::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part05.rar
2022-12-19 16:22:54,536::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part06.rar
2022-12-19 16:22:54,541::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part07.rar
2022-12-19 16:22:54,545::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part08.rar
2022-12-19 16:22:54,550::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part09.rar
2022-12-19 16:22:54,555::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part10.rar
2022-12-19 16:22:54,559::DEBUG::[filesystem:902] [sabnzbd/newsunpack.py.rar_unpack] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/sometestfile-100MB.part11.rar
2022-12-19 16:22:54,563::INFO::[newsunpack:559] Extracting rarfile /Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part01.rar (belonging to ._sometestfile-100MB) to /Volumes/External 4/Movies/_UNPACK_test_download_100MB
2022-12-19 16:22:54,563::DEBUG::[newsunpack:654] rar_extract(): Extractionpath: /Volumes/External 4/Movies/_UNPACK_test_download_100MB
2022-12-19 16:22:54,564::DEBUG::[newsunpack:722] Analyzing rar file ... None found
2022-12-19 16:22:54,565::INFO::[misc:1166] [sabnzbd/newsunpack.py.rar_extract_core] Running external command: ['/Applications/SABnzbd.app/Contents/MacOS/osx/unrar/arm64/unrar', 'x', '-idp', '-scf', '-o-', '-or', '-p-', '/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part01.rar', '/Volumes/External 4/Movies/_UNPACK_test_download_100MB/']
2022-12-19 16:22:54,565::DEBUG::[misc:1167] Popen arguments: {'stdin': -1, 'stdout': -1, 'stderr': -2, 'startupinfo': None, 'creationflags': 0}
2022-12-19 16:22:54,591::DEBUG::[newsunpack:849] UNRAR output 
UNRAR 6.12 freeware      Copyright (c) 1993-2022 Alexander Roshal

/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part01.rar is not RAR archive
2022-12-19 16:22:54,592::INFO::[newsunpack:297] Unrar finished on /Volumes/External 4/Completed/test_download_100MB.3
2022-12-19 16:22:54,592::INFO::[postproc:430] Unpacked files ['/Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB-explanation.txt', '/Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB.bin']
2022-12-19 16:22:54,592::INFO::[filesystem:319] Checking if any resulting filenames need to be sanitized
2022-12-19 16:22:54,592::INFO::[postproc:434] Finished unpack_magic on test_download_100MB
2022-12-19 16:22:54,593::DEBUG::[filesystem:604] Applying permissions 0o100600 (octal) to /Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB-explanation.txt
2022-12-19 16:22:54,595::DEBUG::[filesystem:604] Applying permissions 0o100600 (octal) to /Volumes/External 4/Movies/_UNPACK_test_download_100MB/._100MB-explanation.txt
2022-12-19 16:22:54,596::DEBUG::[filesystem:604] Applying permissions 0o100600 (octal) to /Volumes/External 4/Movies/_UNPACK_test_download_100MB/100MB.bin
2022-12-19 16:22:54,597::DEBUG::[filesystem:604] Applying permissions 0o100600 (octal) to /Volumes/External 4/Movies/_UNPACK_test_download_100MB/._100MB.bin
2022-12-19 16:22:54,599::DEBUG::[filesystem:866] Renaming "/Volumes/External 4/Movies/_UNPACK_test_download_100MB" to "/Volumes/External 4/Movies/_FAILED_test_download_100MB"
2022-12-19 16:22:54,604::DEBUG::[filesystem:909] [sabnzbd/postproc.py.rename_and_collapse_folder] Removing dir /Volumes/External 4/Movies/_UNPACK_test_download_100MB
2022-12-19 16:22:54,604::INFO::[nzbstuff:1797] [sabnzbd/postproc.py.process_job] Purging data for job test_download_100MB (delete_all_data=False)
2022-12-19 16:22:54,605::DEBUG::[articlecache:150] Purging 4 articles from the cache/disk
2022-12-19 16:22:54,605::DEBUG::[filesystem:902] [sabnzbd/filesystem.py.remove_data] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/__ADMIN__/SABnzbd_article_iolkd8w5
2022-12-19 16:22:54,607::DEBUG::[filesystem:902] [sabnzbd/filesystem.py.remove_data] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/__ADMIN__/SABnzbd_article_z3lpsazi
2022-12-19 16:22:54,610::DEBUG::[filesystem:902] [sabnzbd/filesystem.py.remove_data] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/__ADMIN__/SABnzbd_article_9hw2odto
2022-12-19 16:22:54,613::DEBUG::[filesystem:902] [sabnzbd/filesystem.py.remove_data] Deleting file /Volumes/External 4/Completed/test_download_100MB.3/__ADMIN__/SABnzbd_article_yo4ajvdr
2022-12-19 16:22:54,635::INFO::[notifier:123] Sending notification: Download Failed - test_download_100MB (type=failed, job_cat=*)

per user, /Volumes/External 4 is an exFAT external drive. has been using this. issue started after macos 13.1 upgrade from last night

per https://developer.apple.com/documentation/macos-release-notes/macos-13-release-notes

There’s a change to the implementation of the msdos and exfat file systems. Apps that check for those specific file system formats might not detect them. Please file feedback if this impacts your app. (90768681)

trying to find out exactly what changed

sanderjo commented 1 year ago

due to the ._* resource forked version of the rar file

What do you mean with those words "resource forked"?

/Volumes/External 4/Completed/test_download_100MB.3/._sometestfile-100MB.part01.rar is not RAR archive

Do you understand where that file comes from? Does MacOS 13.1 introduce that, on exFAT? As copy / dummy? Why?

EDIT:

Ah, https://apple.stackexchange.com/questions/14980/why-are-dot-underscore-files-created-and-how-can-i-avoid-them/14981#14981 says "You can't avoid them (but see the dot_clean answer by Saeid Zebardast --they can be removed from a directory if that is what you need). They're created to store file information that would otherwise go into an extended attribute on HFS+ (Apple native) or Unix/UFS volumes; in earlier Mac OS this would be the resource fork."

thezoggy commented 1 year ago

The file downloaded from the nzb is sometestfile-100MB.part01.rar but due to it being on external drive and not using a filesystem that prevents the need of the files when accessed mac created ._sometestfile-100MB.part01.rar. This later rears its head as an invalid rar file and causes sab to fail.

for info about the ._ files: https://www.cnet.com/tech/computing/invisible-files-with-prefix-are-created-on-some-shared-volumes-and-external-disks/

You easily can see this type of issue if you were to insert usb drive with exfat on mac and copy files to it. Then take that drive to another machine you will prob see the extra metadata files that have no use to a non macos machine. We already deal with some of this by ignoring the .dsstore stuff this is just another thing that goes with that all.

macos 13 release notes they did mention:

There’s a change to the implementation of the msdos and exfat file systems. Apps that check for those specific file system formats might not detect them. Please file feedback if this impacts your app. (90768681)

but cant find exactly what that means..

sanderjo commented 1 year ago

And therefore you say "macos: ignore resource fork files", meaning "If on MacOS, do not add ._ files to the unrar command"? Because the problem occurs because the wildcard in "unrar *.rar" is ... too ... wild?

How about: On any OS, do not include files starting with . in the unrar? Because 1) easier and 2) starting with . means hidden, so fishy

thezoggy commented 1 year ago

And therefore you say "macos: ignore resource fork files", meaning "If on MacOS, do not add ._ files to the unrar command"? Because the problem occurs because the wildcard in "unrar *.rar" is ... too ... wild?

when we go to build list of what to extract we should avoid including them. avoiding including ._*.rar in list of stuff that needs extracting might be an easy thing to do for macos. there might be a smarter way to go about it.. i had thought we tried to detect if .rar had rar signature before we tried but maybe not?

so looks like file_is_rar is not used to add files to rarlist in build list.. so might could just use that.

but crudely we could also just pass on ._ files for macos at the start of the build_filelists https://github.com/sabnzbd/sabnzbd/blob/develop/sabnzbd/filesystem.py#L472 add:

if sabnzbd.MACOS and file.startswith("._"):
    pass
Safihre commented 1 year ago

Should probably fix it here? https://github.com/sabnzbd/sabnzbd/blob/e34301fb2fef9e5ab9d0a3fa70a2775babadc6c2/sabnzbd/filesystem.py#L762-L771

sanderjo commented 1 year ago

Should probably fix it here?

Oh? I would think here

https://github.com/sabnzbd/sabnzbd/blob/develop/sabnzbd/filesystem.py#L487-L489

and thus

https://github.com/sabnzbd/sabnzbd/blob/develop/sabnzbd/filesystem.py#L448

sanderjo commented 1 year ago

Or easier:

elif RAR_RE.search(file):

change to

elif RAR_RE.search(file) and not os.path.basename(file).startswith(".")

to exclude files starting "." on any OS. 

>>> not os.path.basename("/bla/xxx/blabla123.rar").startswith(".")
True
>>> not os.path.basename("/bla/xxx/._blabla123.rar").startswith(".")
False
Safihre commented 1 year ago

Now the reported problem is Rar, while this problem could occur in any files. What about par2 for example? Or zip? That's why it should be excluded at the source listing.

sanderjo commented 1 year ago

Ah, OK. And then maybe into IGNORED_FILES_AND_FOLDERS ?

sabnzbd/constants.py:128

IGNORED_FILES_AND_FOLDERS = ("@eaDir", ".appleDouble", ".DS_Store")

thezoggy commented 1 year ago

files that start with dot are valid and we shouldnt be tossing those out. its just the ._ files when your on a mac you prob dont care about. (if were wanting to limit tossing out possibly valid stuff) but just doing it regardless of the os might not be too bad as someone easily could do something like have an external drive mounted on a mac but run sab on another os that happens to access that drive.. (quite suboptimal) but could happen.

but yes, as saf pointed out, thats prob the best place to do it. and its not a fixed name we can do as we need to look with starts with. just need to make sure were not preventing sab from cleaning stuff up after the fact because it gets excluded. havent had a chane to dig through to make sure of that.

thezoggy commented 1 year ago

just noticed github cut off a previous update,

here is the relevant drive from the output of: diskutil info -all

   Device Identifier:         disk7s2
   Device Node:               /dev/disk7s2
   Whole:                     No
   Part of Whole:             disk7

   Volume Name:               External 4
   Mounted:                   Yes
   Mount Point:               /Volumes/External 4

   Partition Type:            Microsoft Basic Data
   File System Personality:   ExFAT
   Type (Bundle):             exfat
   Name (User Visible):       ExFAT

   OS Can Be Installed:       No
   Media Type:                Generic
   Protocol:                  USB
   SMART Status:              Not Supported
   Volume UUID:               F0EC9A2C-C013-<removed>
   Disk / Partition UUID:     0BEC372C-C53E-<removed>
   Partition Offset:          210763776 Bytes (411648 512-Byte-Device-Blocks)

   Disk Size:                 18.0 TB (17999996125184 Bytes) (exactly 35156242432 512-Byte-Units)
   Device Block Size:         512 Bytes

   Volume Total Space:        18.0 TB (17999694397440 Bytes) (exactly 35155653120 512-Byte-Units)
   Volume Used Space:         1.1 TB (1086299045888 Bytes) (exactly 2121677824 512-Byte-Units) (6.0%)
   Volume Free Space:         16.9 TB (16913395351552 Bytes) (exactly 33033975296 512-Byte-Units) (94.0%)
   Allocation Block Size:     512 Bytes

   Media OS Use Only:         No
   Media Read-Only:           No
   Volume Read-Only:          No

   Device Location:           External
   Removable Media:           Fixed

   Solid State:               Info not available

I plugged in a usb thumb drive on my mac which has macOS 12.6.2. Formatted USB drive with exFAT and placed a .mkv file on it. No ._ file from that action. But running "open x.mkv" on it, caused the file to open in mkv and resulted in ._x.mkv being created. Tried again with a .png and same thing. looking at the ._ file can see its a binary file and says its a resource fork within it.

Okay so I think I can replicate the user scenario, just need to get sab source and see if the fix I suggested works.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Safihre commented 1 year ago

Should we just always ignore files starting with ._, no matter what OS? Since this could be a macOS disk mounted on Windows, etc.. We do the same for the other macOS specific filesystem tricks like .DS_store.

thezoggy commented 1 year ago

I couldn't replicate this so far (sab download triggering the resource fork creation) on my Mac,. I can get them to generate by playing videos on external fat32/exfat drive.

I don't think it would hurt to just skip them generically

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.