[Question]: Scope of this wiki?

Basically the title. What is the scope of this wiki? Is this only intended to document modern file formats (those used on the Switch), or is this intended to be more of a catch-all for all file types used across various generations? If the latter, how far back are we talking?

If file formats from older generations are allowed, even if you are not personally interested in them, would you accept contributions for adding those types? Asking because there have been no pull requests here yet so it's not clear what your policy is on that

It is ok to document older file formats as well. Age does not matter, but the file format must be exclusive to Nintendo games (and the game must be published by Nintendo, not an indie). For example, Pokemon games are ok, but Minecraft is not. I've added a clarification to the readme.

I would accept contributions as long as the above conditions are met.

Thank you for the clarification, that's much more clear. There is one thing which is a bit unclear however, what would you say counts as a "file" in this context? So far it looks like this covers documentation for files which are physically part of the game files, but I can think of a handful of cases where a file may live on the server only, or when maybe "file" is not the best word?

For example BOSS files on the Wii U/3DS. The files, as stored with the game files, are not the same as how they are stored on the server. The server stores an encrypted version of the files, with a header to define some configuration options. The server provides the encrypted file to the client via SpotPass, which it decrypts and stores locally. Would this server-side file be within the scope of this wiki?

Another example would be in Super Mario Maker, where course data is stored on the server as a single file made up of 4 other compressed files stitched together. The client downloads the metadata about the course from DataStore, which contains the sizes of each of the 4 compressed sections, then downloads the single stitched file and uses the data from DataStore to split the file, decompress it, and store the decompressed files locally. Would this be within scope?

The BOSS files question is also relevant for the "published by Nintendo and exclusively available on Nintendo systems" requirement as well. SpotPass was usable by all developers, not just Nintendo, does this publishing requirement also restrict files to those ONLY found in 1st party games? If so, would the BOSS file format be excluded since it was used in games not published by Nintendo?

It's a bit difficult to come up with precise rules. I think I want to restrict the documentation to files that could be physically part of a game and save files. I'm not exactly sure how BOSS works, but it sounds like the local format would be in scope, but not the server format.

From high to low priority:

Files that are seen in multiple Nintendo games
Files that are seen in a single Nintendo game
Save files

The primary purpose of this repository is to facilitate game modding and data mining, so stuff like BOSS would not have a high priority, even if in scope.

does this publishing requirement also restrict files to those ONLY found in 1st party games?

No. For example, contributions for NintendoWare are more than welcome. Perhaps this is more clear:

The format must be seen in at least one game that is published by Nintendo
The format must be designed by Nintendo, or be exclusively seen on Nintendo systems

The second condition is important because I do not want to document Unity assets or MP4 movies here. Documentation for those can be found elsewhere.

I think I want to restrict the documentation to files that could be physically part of a game and save files

Sounds good, this is super fair :+1: just wanted to make sure

I'm not exactly sure how BOSS works

BOSS is the system which powers SpotPass (it can be used for other things too but for this context only SpotPass matters). BOSS is a system on the Wii U/3DS which essentially just lets the system manage background tasks relating to data on a server (the Wii U uses this to download WWP data from Miiverse for example, though it doesn't use the traditional SpotPass system for this)

SpotPass is a feature on the Wii U/3DS which lets the consoles download (and sometimes upload) additional data for the title without going through a title update or DLC download. Things like the costumes in Super Mario Maker, the stage rotations in Splatoon, system notifications for the notifications app, etc. all use SpotPass. It's kinda similar to BCAT on the Switch IIRC?

The way it works is the application registers a task with BOSS, using a unique ID for the title and the name of a tasksheet/file list, and BOSS will download the data in the background. The Wii U uses the concept of a "tasksheet", which is an XML document defining data about the file such as its size, name, type, and then a URL to the file content. An example would be https://npts.app.nintendo.net/p01/tasksheet/1/zvGSM4kOrXpkKnpT/schdat2?c=GB&l=en, which is the tasksheet for the Splatoon (zvGSM4kOrXpkKnpT) rotation schedule (schdat2). A tasksheet can have multiple files associated with the task (like with SMM and it's costumes). The 3DS uses the concept of a "file list", which is a plain-text style document defining similar data as a tasksheet. The notable difference is that the "file" in the list does not actually need to have a real file on the server, and so no download URL is provided. Some games use the file list as a way to store in-game passwords, for example, rather than actual content files. An example would be https://npfl.c.app.nintendowifi.net/p01/filelist/LUX1uLHXK82phCiX/FGONLYT?c=ES&l=es, which is the file list for Team Kirby Clash Deluxe (LUX1uLHXK82phCiX) (the meaning of FGONLYT is unknown, and is common among many titles, though game-specific list names also exist)

The data on the server is encrypted in one of 2 ways, the Wii U and 3DS have different encrypted content structures. BOSS will download these files when needed in the background, decrypt them, and save the decrypted file locally for future use

but it sounds like the local format would be in scope

These are almost always game-specific files, so it sounds like these would be low priority :+1:

The primary purpose of this repository is to facilitate game modding and data mining, so stuff like BOSS would not have a high priority, even if in scope

The content used by BOSS/SpotPass can be used in this way, which was the main reason why I asked. Since the server format is the way this content is delivered to the client, it wasn't clear if that should be documented as well for people who wish to make modifications. A bit personal, but we also have an archive of nearly 300GB of SpotPass data from prior to the shutdown which we plan to release on the Internet Archive. But it's all stored in the encrypted BOSS formats, so anyone who would want to data mine it would need to decrypt it first

Some games rely very heavily on this data for in game content (such as the costumes in SMM, which can be changed this way, and the stage rotations in Splatoon), and some games even rely entirely on this data (Nintendo Badge Arcade stores basically ALL gameplay data inside the files it downloads with SpotPass. Everything from the crane design/structures, prizes in each crane, the badge designs themselves, everything is downloaded from the server). So making mods for these games would rely on knowing about these files (at the very least, they should know that if the files are modified locally then the server content will overwrite them, unless the modified content is also served from the server)

I definitely understand not wanting to support some of this stuff, and I understand it's hard to set hard rules that cover everything, so no worries there if none of this is within scope :+1: I just wanted to get an idea of what is and isn't covered (and also layout the reasons for asking and what these files are used for) before jumping into contributions

No. For example, contributions for NintendoWare are more than welcome. Perhaps this is more clear:

Much more clear, thank you!

The second condition is important because I do not want to document Unity assets or MP4 movies here. Documentation for those can be found elsewhere

Also very fair, I agree that this is kinda useless here :+1:

kinnay / Nintendo-File-Formats

[Question]: Scope of this wiki? #8