Proposal for a more userfriendly homepath/basepath

[dGr8LookinSparky]

This is a rough proposal for @Chomenor 's new file system, in regards to reorganizing the directory structure inside the homepath/basepath in a way that would hopefully be a lot more user friendly than the old file system, it involves moving the fs_game directories into a sub directory.

First, as a note, the original intended roles of the basepath and the homepath, the basepath was meant to be shared by by all user accounts on a common system, while a unique homepath was meant to be owned by each user account.

The proposed “root contents” of the homepath/basepath would be as follows:

• mods: this would serve as the home of all fs_game directories, can contain fs_game specific downloads from game servers.

• maps: only non-default maps would be downloaded to here, and would work fs_game independent.

• dependencies: not sure if this directory is necessary to serve its intended purpose, but basically it would be for things like texturepacks, and other kinds of pk3 files that would only be loaded if they contain something that is actually used currently by the client. would work fs_game independent.

• overrides: would contain things like custom HUDs, custom skins, custom texture packs, etc. Would be fs_game independent at the root of this folder, but maybe could allow for fs_game specific overrides with sub directories named for the corresponding fs_games.

• defaults: would only be used in the basepath, cannot be overridden in the homepath, and would contain all default assets of 1.1, 1.2, and 1.3. Nothing can be downloaded to this folder, and this folder doesn’t represent any fs_game. We could maybe specify by protocol which default assets would be utilized. Onlye the pk3 files in this directory are loaded when loading the main menu or when loading the download menu. Switching mods from the main menu would only load non-default pk3 files at the time a server is created from the main menu, or when loading a map from the client with /map or with /devmap, otherwise while in the main menu, only the files in the default directory are loaded.

• demos: would contain all demos

• screenshots: would contain all screenshots.

• condumps: would contain all console confdumps

• configs: would contain the universally used config files in its root, we could consider allowing for fs_game specific config overrides by placing sub directories with names corresponding to fs_game names.

• qkey: would still be in the root of the homepath.

[dGr8LookinSparky]

Your idea of buttons that open the specific files would make this part of the directory organization moot, and would be preferable to have. Especially considering even getting to the homepath can be a difficult task if you haven't done it before, even more so on some platforms (i.e. macOS) than on others. I did look into what might be involved in having a client command that would open the homepath in a default file browser:

https://stackoverflow.com/questions/24983834/open-file-folder-in-the-directory

https://linux.die.net/man/1/xdg-open

https://stackoverflow.com/questions/9115574/how-can-you-open-a-file-with-the-program-associated-with-its-file-extension

https://stackoverflow.com/questions/6631299/python-opening-a-folder-in-explorer-nautilus-mac-thingie

I don't think the directory should be called trem13 though (what about future versions?), I think a more suitable name would be something like user-data.

[dGr8LookinSparky]

Oh, I made a typo, the buttons would open the folders, not the files (although that could be done too, but not really needed). I think the way to go would be a system function (customized per platform) that would open any specified path would be sufficient, and then the specific paths can be specified in the buttons.

[Chomenor]

The idea of trem13 is that it would hold any general-purpose pk3s compatible with (only) 1.3 and up. Putting the data files saved by 1.3 under the same location would be consistent with typical idtech3 conventions, and avoids an extra directory that doesn't really fit the usage of other mod directories, which could be confusing to new users trying to understand the layout. You could say that since the file layout is introduced in 1.3 you keep using that directory even if you release a version 1.4, provided the file layout is effectively the one from 1.3.

If you do want to use a separate directory like user-data, you might want to be careful about being too generic with the name. I think being able to coexist well with other past and future Tremulous applications, including mods by other groups, is a good feature and having your particular file layout under a distinctly named directory helps keep things organized when different applications share the same install location.

[Chomenor]

The way you break it down logically is that each mod directory has a dual purpose as a pk3 repository plus potential data store for a certain mod/application. This model is compatible with existing files from other applications and versions, and since each directory at the root has similar behavior it is not too hard to understand.

• trem13: general pk3s compatible with 1.3 and up, and data storage using the layout introduced by 1.3
• gpp: general pk3s compatible with gpp and up, and data storage used by gpp applications (or the old 1.3 client that you have released currently)
• base: general pk3s compatible with 1.1 and up, and data storage used by old applications
• other mods: pk3s for the mod, and potentially data storage for files created by the mod vm itself

[Chomenor]

Note that in the simplest case with a clean 1.3 install, this can reduce to almost all config/screenshot/demo/etc. data being stored in trem13 (or user-data) and the other directories just being used for pk3s. But this model also works in more complex situations with mods and coexisting applications storing data in other directories without requiring users to learn a new directory structure or move to a new homepath location.

[dGr8LookinSparky]

from a server perspective, a common file (stored in the current fs_game) is admin.dat. That file stores admin level definitions (with admin ability flags), registered player information (registered players can individually be flagged abilities, but also have their admin level specified), and bans. Because admin commands and levels might be defined differently, admin.dat would probably have to remain in each fs_game separately. However, when I do change the fs_game, since our reasons for changing it so far doesn't result in admin changes, I would just copy the admin.dat over to keep the player registrations. Is there a better way of handling something like that?

Also each fs_game has a separate copy of a games.log (the server game log), should that instead be in one place for the same server?

I know that servers can greatly benefit from the same dedicated configs folder that the clients would use (as I specified aboove). Some settings you might want to keep universal independent of the fs_game (like the name of the server, the master servers being pointed to, the download url, etc), and other settings might only make sense for the specific fs_game.

[dGr8LookinSparky]

In that context, by "common file" I mean commonly used file.

[Chomenor]

If admin.dat is handled by the game vm rather than the engine, it probably makes sense to be in its own fs_game. Is live switching of server fs_game or game vm supported, or intended to be supported in the future? If not, it probably doesn't matter much which mod server files are stored in, since I think most production servers should have their own full working directory.

[dGr8LookinSparky]

the VMs can be swapped live without having to restart the tremded, it just requires a map change, the fs_game currently requires a full tremded restart (to my knowledge).

Changing pk3 files live is not ideal as it should be however. If you swap or remove any pk3 files live, clients that would attempt to download the previous pk3 files directly from the game server won't be able to, and in some cases the server can crash if it tries to access a pk3 files that is no longer there.

I think perhaps a preferable approach would be someway for the swapping/removal/adding to be prepared while a map is played, and then take place automatically upon the next map change. This would especially be useful for servers that update often.

[Chomenor]

There are probably ways to deal with these server issues, but I think we should deal with the client file layout first, especially the question of whether to keep the current mod layout that is compatible with existing versions. What are your thoughts on that at this point?

[dGr8LookinSparky]

I wrote an initial implementation for "folder browsing buttons". It needs some more work and testing on all the supported platforms, but so far it does function on Linux at least. There is more information here: https://github.com/GrangerHub/tremulous/issues/79#issuecomment-386692623 .

Given that functionality, we could try the most recent approach with the layout you suggested @Chomenor , with the dedicated subdirectories I proposed (i.e. configs/demos/screenshots/condumps ) in the same folder. As long as all demos 1.3 saves and uses are in that one dedicated demos subdirectory, all client config files written and used by 1.3 are in that one dedicated configs subdirectory, etc... Then we can add buttons in the "Browse Folders" menu for those particular dedicated subdirectories.

[Chomenor]

Sounds good, I'll start working on the filesystem side of the implementation shortly.

Just to confirm, do you like the idea of the name "trem13" for the new directory, or would you prefer some other name? It can be changed later of course. Also do you think a name like this should be hardcoded, or is there a use case for making it cvar controlled?

[dGr8LookinSparky]

If we don't have to include pk3 files in that directory, I think it would make sense to give it a non-version specific descriptive name (something like user-data), in which case I think hardcoding would be the way to go. I think it would be good if starting from this point, that data is always in the same place (unless would it be more beneficial in the long run for that directory to be versioned?)

Since the intended use of the data in both trem13 and gpp folders would behave differently from the data in all other non-base fs_game folders, Would there be issues with old systems if we put a trem13 subdirectory inside of the base folder, and kept the 1.3 default assets in there? likewise if we put a gpp subdirectory in base as well (although we could call it something like trem12 to keep with a better naming convention, and for all practical purposes, gpp assets can be considered 1.2 assets (1.2 never had a full release, and likely never will at this point)).

[wtfbbqhax]

@dGr8LookinSparky @Chomenor, please consolidate idea's into a concise 1-2 page design document for the rest of us

[dGr8LookinSparky]

@Chomenor , does the edit button work for you on this wiki page (idk if editing on that wiki requires special permission)? https://github.com/GrangerHub/tremulous/wiki/New-homepath-layout

[Chomenor]

Yes, the edit button appears to work.

@dGr8LookinSparky What did you think of my points above in favor of using the trem13 directory for both data and pk3s? It would follow existing conventions better and having the version in the name helps distinguish it especially to users from directories that may be used by other applications.

I don't know why you would want to put any subdirectory under base. In general you want to follow the mod dir conventions as much as possible, both because users will be able to understand it more easily and because it allows other Tremulous versions to access the same files.

[Chomenor]

To sum up my design, you just add the "trem13" directory, with the new configs/demos/screenshots/etc directories under it, and which can also be used for pk3s that other clients shouldn't load because they are only compatible with 1.3.

Everything else stays as it is, and you don't need much documentation because the extra directory is very apparent in meaning to anybody looking at the install directory, which is good because most users won't read documentation anyway.

[dGr8LookinSparky]

Lets consider how future major client versions might fit into this layout. What our general overall plan is currently for post 1.3, starting with 1.4 is to have the new servers (1.4 and above) break compatibility with older clients (bellow 1.4), but still have the newest clients keep backwards compatibility with the older servers. We may take the same approach with 1.5 and so on.

So lts say someone has a 1.3 installation, and they have used a trem13 sub-directory for awhile, where all the demos, screenshots, confgis, and config statements they ever made/used with the 1.3 client (whether they play on a 1.1, gpp, or 1.3 server, or from loading a match from their client) would only be found in that trem13 sub-directory. Then later, when 1.4 clients would be available, such a new client is installed on top of the existing 1.3 install. a trem14 subdirectory is created which would be the new home of the demos/configs/screenshots/condumps made/used by the 1.4 client, even from the 1.4 client connecting to 1.1, gpp, and 1.3 servers. So if you wanted to use any of the demos/configs from 1.3 in the 1.4 client for example, you would have to copy them from the trem13 folder to the trem14 folder., as the 1.4 client wouldn't be touching the demos/etc in the trem13 folder, however, the 1.4 client still may use the default pk3 files in trem13 at least for connecting to 1.3 servers, same with the pk3 files in the gpp folder.

I do believe that it is essential that at least for what a given client would create/use, all demos/configs/screenshots/condumps should be available each in their own dedicated folders, that can be opened by dedicated folder buttons (regardless of which fs_game and server version that client might be connected to at the time). But for new version installs on top of old installs, there is potential for confusion with the layout in regards to the user data, without using the in-game folder buttons, although maybe that might not be as much of an issue having the in-game folder buttons.

There is another potential problem of clutter to consider, what if older version servers created fs_game folders with the naming convention of future versions? Like a 1.3 server (or even a 1.1 server) using an fs_game name trem13. Or for that matter, what if a 1.3 (or older) server used a trem13 fs_game with modded pk3 files that should rightfully be in a seperate fs_game? Sure with this new file system, that shouldn't interfere with the correct functioning of the new clients, but there would be clutter of pk3 files that should be in "modded" fs_game folders, but are in the versioned "baselike" folders. Is there anyway we could guard against such potential clutter that would be intuitive, or is that a necessary evil?

[Chomenor]

My interpretation of the "trem13" directory is that is for data and pk3s for version 1.3 and up. So when you release 1.4 you probably can keep storing most data in trem13, and only create a trem14 if you have new data that is only compatible with 1.4. The thing to remember and convey to users is that trem13 stands for 1.3 and up, not 1.3 exactly.

base - compatible with 1.1 and up gpp - compatible with gpp and up trem13 - compatible with 1.3 and up trem14 - compatible with 1.4 and up

Older versions putting pk3s in a newer version directory isn't a problem, because the hash precedence system in newer version will take care of any conflicts. On the other hand a newer version server putting incompatible files in an older versioned directory could be a problem, for people who still run an old version client, but that's not really our responsibility since the potential for incompatible pak downloads is an inherent issue for older clients.

[Chomenor]

I can also understand that continuing to use trem13 for data storage on versions past 1.3 might seem counterintuitive, but for me it is actual more clear than something generic like 'user-data' because it conveys the history of the directory, i.e. the file storage location introduced in version 1.3, and disambiguates it from any storage locations used by unrelated projects.

[dGr8LookinSparky]

I updated the wiki page with a design based on our discussions so far: https://github.com/GrangerHub/tremulous/wiki/New-homepath-layout

Feel free to comment regarding it, and adjust where appropriate.

[dGr8LookinSparky]

I edited the wiki page further switching to the MediaWiki format so that it now has a nice autogenerated table of contents (that the markdown format didn't have).

[Chomenor]

This looks really good. Now that we have at least a provisional file layout is established, I should be able to start working on the hash precedence system. Do you think we should have a separate issue or other chat area to discuss questions about that effort?

[dGr8LookinSparky]

Yes, I think we should have a separate issue for that discussion. We probably should also create a few more separate issues for some of the other things discussed in this issue.

[dGr8LookinSparky]

Hmm, I was thinking, should the "versioned base like" folders instead have a prefix of "base_" for extra clarity of its function (also to ensure they stay near the top of the folder when things are ordered alphabetically)? That is: base_X-Y.

[jkent]

Perhaps I missed the part of the discussion, why is it base_X-Y instead of base_X.Y?

[Chomenor]

@dGr8LookinSparky I don't see any reason why not. I went with "trem" in my examples I guess because it reflects the name of the project associated with it, "Tremulous 1.3", but I don't know any technical reason to favor either name.

[dGr8LookinSparky]

Since the entire homepath is dedicated to the Tremulous game, it is kinda redundant to use the prefix trem instead of base .

@jkent It wasn't discussed before, but I wasn't sure if using "." in the folder name would present any issues. Would there be any issues with using a period in folder names for either new or old clients/servers @Chomenor ?

[Chomenor]

"trem" and "base" kind of have the same meaning to me. It probably comes down to whatever you think looks nicer.

I don't know offhand if there are any issues with periods in mod names.

[dGr8LookinSparky]

I like the idea of these other "versioned base" folders being next to base alphabetically, as well as close to the top of the homepath alphabetically, and having their prefix further indicating that they have similar properties to base. Regarding the period, lets try with it for now, and if it is found to be an issue with either the new system or with compatibility with old systems we can change it back to using hyphens instead easy enough.

I updated the wiki page accordningly: https://github.com/GrangerHub/tremulous/wiki/New-homepath-layout

[jkent]

thumbs up

[wtfbbqhax]

@dGr8LookinSparky @Chomenor, A comment from Jkent above reminded me of something we need which is pk3 sorting based on semantic versioning (particularly against the code we already use external/semver) as this is how we tag our releases and subsequently our paks.

@Chomenor Does your system support pk3dir's?

[Chomenor]

@wtfbbqhax How does this semantic versioning work for pk3s? Is there any documentation for this feature?

My filesystem does currently support pk3dirs.

[dGr8LookinSparky]

@Chomenor , what if we were to have a user-data directory that would be top level in the homepath and contain the configs, demos, screenshots, condumps subdirectories, and like the basemod directory, not allow pk3 files to be downloaded to it, and not allow the fs_game to be set to it?

[Chomenor]

It's technically possible. IIRC I favored using trem_1.3 for this purpose because it would be more clear to users which particular version was creating this particular file layout, when multiple versions coexist(ed) with the same homepath location, and is consistent with the role of other directories in the homepath.

[dGr8LookinSparky]

It would be consistent with the file layout, it would be like user-data was an fs_game directory, but we wouldn't let the client nor server use it as an fs_game, and if the unlikely situation that an old client uses that as an fs_game, there still wouldn't be any conflict, the old client would actually save the demos and screenshots in the same folders in that case. Additionally, it is the plan to make the config files functional for multiple versions, having the option to set cvars and binds for specific version(s), so it should be future proof if/when there might be a need for a new base folder version after 1.3.

[Chomenor]

trem_1.3 is presumed to be used for future versions anyway, so being future proof shouldn't be an issue. Whether it is confusing is another matter, but personally I think storing data in trem_1.3 for a hypothetical version 1.4 is still better than a generic name like user_data as long as the homepath is shared with other Tremulous clients.

[Chomenor]

I understand this is less elegant, but to me different types of Tremulous clients should avoid using commonized directory names in the shared homepath folder whenever possible, especially when the layout of the directory is specific to a particular client. Otherwise things can get really messy.