Feature Request: The ability to extract the No-Intro sets.

[TomTurbine]

With the official CDNs set to shut down next year, the No-Intro group has largely datted the entire CDN at this point and each of them includes the tmds even versioned as well.

Basically having an application that would be able to extract them as needed from the set even being able to extract the different version of the updates and all would be a welcome thing.

Edit: Also, since their encryption scheme is being used by their other systems, an possible extension of this feature request would be to allow it to also be used on the other CDNs of the No-Intro collections that use it.

[coreyemtp]

I disagree, I think it's No-Intro that needs to fix their DAT. It should be 1:1 with what is downloaded from the CDN, but it's a complete mess right now.

[TomTurbine]

I disagree, I think it's No-Intro that needs to fix their DAT. It should be 1:1 with what is downloaded from the CDN, but it's a complete mess right now.

If they just did a file name match, it would either be missing a ton of files as they have multiple versions of updates that just build off the previous updates or you would have massive amounts of repeating files more than doubling the size of the set because of it as they would have multiple full updates of the exact same game sometimes.

Remember how many Smash Brothers updates they have, now imagine having to store the full version of each one of them even though you know that every file of the older updates are also in the newer ones.

The way they have it setup allows the set to have each version of each update in it. It is the equivalent of having a merged Mame set versus of non-merged set.

I know it's still a work in progress, but the files are still there with a set naming scheme and can be built off of and the servers will go down eventually. So best to have them as a fall back.

I see where you disliked the .h3 files on their site because they were just hash checks put there by the software that dumped them for verification and not the server itself but having those files extra doesn't prevent you from using the other files.

And you have to admit, the files are still there, have an understandable naming scheme, and still can be used.

[coreyemtp]

Agreed, of course a simple filename match would be useless, but I don't really see the relevance since CRC32/SHA1 is used.

Ultimately there is a final version of each game, and they will never see an (official) update. Therefore, the definitive set would be the main game and the final update. That is what is downloaded when you request the game. I understand you can download a prior version, but that is a niche use for the "maybe I'll need it for a hack some day" crowd or those who inexplicably believe that every byte of data generated deserves preservation. That falls more in line with the TOSEC/GoodSets way of thinking. Don't muck up the main DAT files with a bunch of garbage that is useless to 99.9% of users.

[TomTurbine]

Agreed, of course a simple filename match would be useless, but I don't really see the relevance since CRC32/SHA1 is used.

Ultimately there is a final version of each game, and they will never see an (official) update. Therefore, the definitive set would be the main game and the final update. That is what is downloaded when you request the game. I understand you can download a prior version, but that is a niche use for the "maybe I'll need it for a hack some day" crowd or those who inexplicably believe that every byte of data generated deserves preservation. That falls more in line with the TOSEC/GoodSets way of thinking. Don't muck up the main DAT files with a bunch of garbage that is useless to 99.9% of users.

Not entirely true. With the current setup, it allows you to keep virtually every update with minimal space as the last update is built off the older updates anyways. And if they exclude the older stuff, then it isn't a proper CDN set. That's like asking them to start purging all the older revisions of SNES games as well as their betas and prototypes.

Sure, most people won't use them but that still means its an incomplete set in the DAT file.

Then it comes to games like Smash Brothers where they change balance from patch to patch where some players liked it before.

Now, don't get me wrong, I also believe that 99.9% of all users will only care about the last update, but that isn't what the No-Intro group is about doing. They are trying to preserve it all in their sets and were actually debating on whether they should include every single aftermarket and homebrew app in their sets.

Just glad they finally came to the decision to split those dats in the near future where all the official, unlicensed games, and virtual console/mini-console games get one dat for that system and then all the aftermarket and homebrew will go into another. Will leave one relatively stable platform of the actually released stuff and one constantly updated with every homebrew and aftermarket release in their collection.

Edit: Also, given what you just described, you might actually support a suggestion I put on the No-Intro forums for a new tag for the sets "Superseded"

https://forum.no-intro.org/viewtopic.php?t=6253

[coreyemtp]

We definitely agree on the latter part, splitting the DATs is much needed!

---

From: Fugus @.> Sent: Saturday, December 30, 2023 11:19:57 PM To: Xpl0itU/WiiUDownloader @.> Cc: Corey Davis @.>; Comment @.> Subject: Re: [Xpl0itU/WiiUDownloader] Feature Request: The ability to extract the No-Intro sets. (Issue #78)

Agreed, of course a simple filename match would be useless, but I don't really see the relevance since CRC32/SHA1 is used.

Ultimately there is a final version of each game, and they will never see an (official) update. Therefore, the definitive set would be the main game and the final update. That is what is downloaded when you request the game. I understand you can download a prior version, but that is a niche use for the "maybe I'll need it for a hack some day" crowd or those who inexplicably believe that every byte of data generated deserves preservation. That falls more in line with the TOSEC/GoodSets way of thinking. Don't muck up the main DAT files with a bunch of garbage that is useless to 99.9% of users.

Not entirely true. With the current setup, it allows you to keep virtually every update with minimal space as the last update is built off the older updates anyways. And if they exclude the older stuff, then it isn't a proper CDN set. That's like asking them to start purging all the older revisions of SNES games as well as their betas and prototypes.

Sure, most people won't use them but that still means its an incomplete set in the DAT file.

Then it comes to games like Smash Brothers where they change balance from patch to patch where some players liked it before.

Now, don't get me wrong, I also believe that 99.9% of all users will only care about the last update, but that isn't what the No-Intro group is about doing. They are trying to preserve it all in their sets and were actually debating on whether they should include every single aftermarket and homebrew app in their sets.

Just glad they finally came to the decision to split those dats in the near future where all the official, unlicensed games, and virtual console/mini-console games get one dat for that system and then all the aftermarket and homebrew will go into another. Will leave one relatively stable platform of the actually released stuff and one constantly updated with every homebrew and aftermarket release in their collection.

— Reply to this email directly, view it on GitHubhttps://github.com/Xpl0itU/WiiUDownloader/issues/78#issuecomment-1872666654, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOH5SHSAI4OKBJFP5NIKQB3YMDRW3AVCNFSM6AAAAABAV3LUWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGY3DMNRVGQ. You are receiving this because you commented.Message ID: @.***>

[TomTurbine]

Any word on this? I have checked and the files do match up perfectly between what your app downloads and what No-Intro verifies it as.

All they do is remove the app extension and rename the tmd to show what version of tmd it is for those entries that have multiple versions of them.

Honestly though, this could actually work really well as a standalone app. Just have it scan a zip file or a folder and if it finds a tmd file it generates the missing files in memory and then decrypts them using those files "as is" and if it finds multiple tmd files it asks which version you wish to decrypt and if you were to setup a batch session you can have it setup to either automatically decrypt the highest version found or ask.

Bonus if it can be done without fully extracting the actual zip to another location first since no file in the archive is ever more than a few GB so they could be extracted to memory individually as they were being processed. Double bonus if it could also work on the Wii, 3DS, or any other consoles CDN files.