Support ISO9660 file format

[blackwind]

This isn't *arr-centric so I expect it won't get any traction, but it'd be tremendously useful to me and my fellow game-hoarders if Unpackerr had an option to extract and clean up .iso files as well, recursively from an originating rarset if needed.

The 7z CLI app is capable of extracting .iso, but I assume the Go library you're using isn't. I recommend this library, as the other available one apparently doesn't support files larger than 4GB, a major no-no:

https://github.com/kdomanski/iso9660

[davidnewhall]

I downloaded the large Dreamcast DCRES pack floating around out there. I tried to extract it with xt, and all hell broke loose. The 7z library sucks, and I already have plans to change it. Thanks for your suggestion, I'll check it out. I also found that the rar files were packed in some format my (pretty damn good) rar library doesn't seem to work with. I mention this because I wonder what other sorts of 'junk' will come from the game-hoarding scene. Hopefully it's just an "old stuff" problem and your use will be better.

Extracting ISOs seems fine. When I have some time I'll dig into this more.

[blackwind]

Cheers! I know there are a number of edge cases like Xbox and Xbox 360 ISOs that aren't standard ISO9660 files, but while there are reasons one might want to extract those individually, I can't think of any reason one would want to extract them all automatically. Expect to run into stuff like that for sure, but I wouldn't worry too much about it. My use case is just modern PC games, packaged pointlessly into ISO9660 files by release groups after downloading them from Steam and copying in their generic crack (thanks guys, but what exactly am I supposed to burn this 150GB ISO onto?), then packaged again into RAR4 or RAR5 sets. Nothing fancy.

[davidnewhall]

Still putting some polish on this, but it works.

2022/12/11 01:54:38.501623 folder.go:260: [Folder] Extraction Started: /Users/david/go/src/golift.io/xtractr/test_data/test_isos, items in queue: 0
2022/12/11 01:54:38.912676 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Angry Birds (Europe) (Chillingo) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:39.296143 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Ancient Game Treasures Mill (USA) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:39.432270 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission II (USA) (NG) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:39.660598 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Ambassador Kane (Europe) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:40.073118 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alien Zombie Death (Europe) (En,Fr,De,Es,It) (v2) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:40.391780 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission II (Europe) (NG) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:40.586384 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Angry Birds (USA) (v2) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:40.951896 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/1000 Tiny Claws (USA) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:41.235238 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alien Zombie Death (Europe) (En,Fr,De,Es,It) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:41.549490 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission (Europe) (NG) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:41.723926 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission (USA) (NG) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:41.893598 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alien Zombie Death (USA) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred
2022/12/11 01:54:42.172999 queue.go:291: [DEBUG] Extracting File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Angry Birds (Europe) (Rovio) (minis) (PSN).iso to /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred

2022/12/11 01:54:42.588423 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/1000 Tiny Claws (USA) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/1000 Tiny Claws (USA) (minis) (PSN)
2022/12/11 01:54:42.588512 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Alien Zombie Death (Europe) (En,Fr,De,Es,It) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alien Zombie Death (Europe) (En,Fr,De,Es,It) (minis) (PSN)
2022/12/11 01:54:42.588580 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Alien Zombie Death (Europe) (En,Fr,De,Es,It) (v2) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alien Zombie Death (Europe) (En,Fr,De,Es,It) (v2) (minis) (PSN)
2022/12/11 01:54:42.588643 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Alien Zombie Death (USA) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alien Zombie Death (USA) (minis) (PSN)
2022/12/11 01:54:42.588707 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Alpha Mission (Europe) (NG) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission (Europe) (NG) (PSN)
2022/12/11 01:54:42.588765 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Alpha Mission (USA) (NG) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission (USA) (NG) (PSN)
2022/12/11 01:54:42.588829 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Alpha Mission II (Europe) (NG) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission II (Europe) (NG) (PSN)
2022/12/11 01:54:42.588889 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Alpha Mission II (USA) (NG) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Alpha Mission II (USA) (NG) (PSN)
2022/12/11 01:54:42.588948 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Ambassador Kane (Europe) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Ambassador Kane (Europe) (minis) (PSN)
2022/12/11 01:54:42.589160 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Ancient Game Treasures Mill (USA) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Ancient Game Treasures Mill (USA) (minis) (PSN)
2022/12/11 01:54:42.589229 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Angry Birds (Europe) (Chillingo) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Angry Birds (Europe) (Chillingo) (minis) (PSN)
2022/12/11 01:54:42.589290 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Angry Birds (Europe) (Rovio) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Angry Birds (Europe) (Rovio) (minis) (PSN)
2022/12/11 01:54:42.589352 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Angry Birds (USA) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Angry Birds (USA) (minis) (PSN)
2022/12/11 01:54:42.589412 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/Angry Birds (USA) (v2) (minis) (PSN) -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/Angry Birds (USA) (v2) (minis) (PSN)
2022/12/11 01:54:42.589468 files.go:250: [DEBUG] Renamed Temp File: /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred/_unpackerred.test_isos.txt -> /Users/david/go/src/golift.io/xtractr/test_data/test_isos/_unpackerred.test_isos.txt

2022/12/11 01:54:42.589515 files.go:270: Deleted (recursively): /Users/david/go/src/golift.io/xtractr/test_data/test_isos_unpackerred

2022/12/11 01:54:42.589590 folder.go:276: [Folder] Extraction Finished: /Users/david/go/src/golift.io/xtractr/test_data/test_isos => elapsed: 4s, archives: 14, extra archives: 0, files extracted: 15, written: 1849MiB

[blackwind]

Awesome, cheers!

[blackwind]

Hope to get around to setting this up sometime this weekend, but a question first that came to mind: Unpackerr won't recurse into ISOs, right? I can imagine scenarios where a game might have archives among its data, Unpackerr unpacks and deletes them, and the installer or game itself then fails.

[davidnewhall]

It's a great question. It probably will. And that's the type of feedback I need.

[davidnewhall]

I have not release a version with this feature in it yet, but I can send you a test build. What OS?

[blackwind]

I'll be using Docker, and I noted a few nights ago there's an unstable tag. It should already be in there, right?

[davidnewhall]

hah yup. You know what to do. :) Sorry, I talk to a lot of folks about a lot of problems every day.

[blackwind]

Ah, the curse of the developer who writes software primarily for the type of person who shouldn't be using it. 🤣

[davidnewhall]

This feature is present in the latest release. https://github.com/Unpackerr/unpackerr/releases/tag/v0.11.1

[blackwind]

Alright, finally sat down to check this out. Here's what I found:

• First and foremost, the ISO extraction algorithm needs work. I tested two ISOs, both of which WinRAR identifies as "UDF 1.02 archives". ISO 1 (the first one in my log) has a 6GB file inside. Unpackerr only extracts 2GB of it, no matter how many times I re-run the test. ISO 2 also has a very large file inside, but Unpackerr had no problem with it. Although the extraction of ISO 2 produced files whose contents match that of a WinRAR extraction, the filenames were truncated and in all caps. Example:

$ ls -l extracted-by-unpackerr/
total 4432364
drwxr-xr-x 2 docker everyone       4096 2023-02-01 21:43 ARTBOOK
drwxr-xr-x 2 docker everyone       4096 2023-02-01 21:43 GUIDE
drwxr-xr-x 2 docker everyone       4096 2023-02-01 21:43 MANUAL
drwxr-xr-x 4 docker everyone       4096 2023-02-01 21:43 OST
drwxr-xr-x 2 docker everyone       4096 2023-02-01 21:44 POSTER
-rw-r--r-- 1 docker everyone  243729046 2023-02-01 21:44 SETUP~01.BIN
-rw-r--r-- 1 docker everyone 4294081022 2023-02-01 21:44 SETUP_X-.BIN
-rw-r--r-- 1 docker everyone     896112 2023-02-01 21:44 SETUP_X-.EXE
drwxr-xr-x 2 docker everyone       4096 2023-02-01 21:44 WALLPAPE

$ ls -l extracted-by-winrar/
total 4432364
drwxr-xr-x 2 docker everyone       4096 2022-12-16 08:20 artbook
drwxr-xr-x 2 docker everyone       4096 2022-12-16 08:20 guide
drwxr-xr-x 2 docker everyone       4096 2022-12-16 08:20 manual
drwxr-xr-x 4 docker everyone       4096 2022-12-16 08:20 ost
drwxr-xr-x 2 docker everyone       4096 2022-12-16 08:20 poster
-rw-r--r-- 1 docker everyone 4294081022 2022-12-16 08:20 setup_x-blades_hd_1.0_(60327)-1.bin
-rw-r--r-- 1 docker everyone  243729046 2022-12-16 08:20 setup_x-blades_hd_1.0_(60327)-2.bin
-rw-r--r-- 1 docker everyone     896112 2022-12-16 08:20 setup_x-blades_hd_1.0_(60327).exe
drwxr-xr-x 2 docker everyone       4096 2022-12-16 08:20 wallpapers

• I expected the ISO to be extracted to the root _unpackerred folder, but it's currently creating an undesirable extra subfolder based on the ISO's filename.

• I expected the ISO to be deleted after extraction along with the other archives, but currently it isn't.

And finally, two minor unrelated quibbles that aren't worth opening an issue for:

• An ignore_path config parameter might be worth adding to ignore the folder for in-progress downloads. I nest my incomplete folder inside my download folder, and I assume I'm not alone. In the meantime, I mounted an empty folder over it so Unpackerr can't see what's inside.

• Let us set log_queues to 0 to disable the queue messages entirely. In the meantime, I set it to over a year.

I'll make every effort to test the fixes you push to unstable same-day. Sorry for taking my sweet time up to this point.

[davidnewhall]

to ignore the folder for in-progress downloads.

This is a Docker problem with fsnotify/inotify not working. Wish I knew how to make it work. If you run unpackerr native on Linux/Windows it will not extract in progress downloads.

I set it to over a year.

That's the way to do it. I can probably make it print less stuff when you have no starr apps configured too, then it might be reasonable to set it to once a day or something.

For the rest of this.. We can definitely work on it. Do you mind transferring the comment above to a brand new issue, so have a clean slate? You may omit the log; honestly, it's not very useful without debug. Don't worry too much about it though, you've given me a few things to tackle, so give me a few days. Thanks!