[Feature]: Centralize campaign storage

[kwvanderlinde]

Describe the Problem

As far as campaign files go, MapTool only understand .cmpgn ZIP files. These files are great as they can easily be shared around, but they limit our ability to improvement MT in some ways. It also means the user has to make a decision about where to save their campaigns - this isn't onerous, it is an unnecessary burden given that .cmpgn files are specific to MT.

The Solution you'd like

I would like a user's campaign to be centralized into a Campaign Manager (final name TBD). The Campaign Manager would just be a directory under ~/.maptool-rptools (on linux; I don't know the equivalent for other OSes).

The Campaign Manager would have a UI to browse the user's campaigns and open them. This UI would not just be a file chooser, but would be something that isn't directly tied to the file system or the format of the internal campaigns. It would show pertinent information to the user about the campaigns stored in the Campaign Manager. A bare minimum in my mind would be showing the campaign name and MT version it was created in.

The current paradigm of "opening" and "saving" .cmpgn files would be removed. Instead, campaigns could only be opened from the Campaign Manager and saved to the Campaign Manager. Existing .cmpgn files could be "imported" into MT either by the File menu + file chooser, or by dragging the .cmpgn file into the Campaign Manager UI. This would copy the campaign into the Campaign Manager where it would be available for use. The reverse would also be possible: "exporting" a campaign from the Campaign Manager to a .cmpgn file of the user's choice.

The files in the Campaign Manager would be considered "off limits" for a user. Not that we could actually enforce that, but I think it's fair to say that if you mess with these files you're on your own.

Alternatives that you've considered.

None.

Additional Context

This was discussed a while back on Discord as part of campaign directories. This FR is just about one point in that discussion and is just my take on how we can achieve it.

And thanks to @thelsing who first came up with the concept (though I don't claim to accurately reflect his ideas).

[kwvanderlinde]

What I'm hoping for with this FR is to take one relatively small step to get to a place where lots of other improvements are possible. I don't want this FR to become about those possible future improvements, but I also think it's important to think a bit about them as part of the motivation for this idea. So here's a few:

1. We could change the format of campaigns stored in the Campaign Manager, avoiding the need to unzip & zip campaigns just to use them.
2. We could share static data (assets) across campaigns in the Campaign Manager to avoid redundancy and make better use of the file system. Maybe consider Library resources as well...
3. We could add a place for GM prep tools that could do things that aren't appropriate for a shared campaign. Exported .cmpgn files would not include these tools.
4. We could have easy backup and restore options that go beyond the autosave feature.

[taustinoc]

Given the amount of grief saving .cmpgn files under the ~/.maptool-rptools directory has caused over the years, what happens to this Campaign Manager directory when MapTool is uninstalled and reinstalled?

There would need to be a bulletproof protection of it during the uninstall process, particularly since most people are never going to export any kind of backups.

(And then, there's the issue of what could easily be multiple gigabytes of data files left over when MT is uninstalled with no intention of reinstalling it.)

[kwvanderlinde]

Given the amount of grief saving .cmpgn files under the ~/.maptool-rptools directory has caused over the years, what happens to this Campaign Manager directory when MapTool is uninstalled and reinstalled?

There would need to be a bulletproof protection of it during the uninstall process, particularly since most people are never going to export any kind of backups.

What issues have there been with ~/.maptool-rptools/? I'm familiar with the problems of saving to the install directory, especially on Windows, but that is a different location and is forbidden now. Does Windows also remove ~/.maptool-rptools/ when uninstalling MT? If so we should fix that.

(And then, there's the issue of what could easily be multiple gigabytes of data files left over when MT is uninstalled with no intention of reinstalling it.)

We can either have our cake or we can eat it. We can't both clean up the cruft and not lose the data on uninstall.

On Linux (DEB-based) when we uninstall a package we have the option of "Removal" (just remove the package itself) or "Complete Removal" (remove associated files). I don't know how it applies to MT today, nor do I know how prevalent such an option is on other systems, but that sort of option would allow either leaving user data in place or cleaning it up, as the user desires.

[Pmofmalasia]

What I'm hoping for with this FR is to take one relatively small step to get to a place where lots of other improvements are possible. I don't want this FR to become about those possible future improvements, but I also think it's important to think a bit about them as part of the motivation for this idea. So here's a few:

Another nice side effect could be the ability to "clone" a campaign file, specifically for use in making a new campaign from a framework with some required maps/tokens/macros. I know some of this stuff is intended to be rolled into addons, but not sure how feasible having something like a library map rolled into the addon is.

With this in mind, perhaps campaigns could be flagged somehow as "base framework files" so they don't get lost in other campaign files meant for actually playing (terminology to change because that name sucks).

[taustinoc]

What issues have there been with ~/.maptool-rptools/? I'm familiar with the problems of saving to the install directory, especially on Windows, but that is a different location and is forbidden now. Does Windows also remove ~/.maptool-rptools/ when uninstalling MT? If so we should fix that.

I am, perhaps, confused as to exactly what goes where. I don't know if uninstall removes the data directory or not, but I have had things go wrong with the resource directory (as in, have it all completely disappear for no reason), which is in the maptool-rptools folder. Should that happen to the campaign manager directory, it would be bad.

The real issue, as I see it - and I don't know how common it would be - is that now, saving a .cmpg file requires a deliberate, conscious act on the part of the user, and pretty much everybody does it at some point. So they're aware of the concept, at least. If you automate the whole process, built in, a lot of people are never going to make any kind of backup because it doesn't occur to them that such a thing is possible.

(And then, there's the issue of what could easily be multiple gigabytes of data files left over when MT is uninstalled with no intention of reinstalling it.)

We can either have our cake or we can eat it. We can't both clean up the cruft and not lose the data on uninstall.

On Linux (DEB-based) when we uninstall a package we have the option of "Removal" (just remove the package itself) or "Complete Removal" (remove associated files). I don't know how it applies to MT today, nor do I know how prevalent such an option is on other systems, but that sort of option would allow either leaving user data in place or cleaning it up, as the user desires.

One possibility would be to ask on uninstall if the user wants to remove saved files. I've seen many programs that do so. But there will be people, who knows how many, who either don't read or don't understand the prompt, and click on either yes or no at random. And, at the moment, the strong recommendation is that when upgrading to a new version, one should uninstall the old version first, so uninstalling and reinstalling will be a regular thing for many users. Some of them will click on the wrong thing.

Another possibility (if it is possible, maybe it's no for a Java app) would be to have an upgrade process built in. Instead of just a prompt saying "there's a new version, do you want to download it" have it give the option to download and install with an assumption that nothing in the maptool-rptools directory should be touched.

Another question: How would this deal with having multiple versions of MapTool installed?

There's hazards in any approach.

(Just to be clear, I quite like the idea. I just think there's question it is far better to ask before anyone starts coding it, rather than after it's in a newly released version.)

[FullBleed]

What issues have there been with ~/.maptool-rptools/? I'm familiar with the problems of saving to the install directory, especially on Windows, but that is a different location and is forbidden now. Does Windows also remove ~/.maptool-rptools/ when uninstalling MT? If so we should fix that.

The "data directory" is not removed on uninstall in windows.

But, given that it defaults to a hidden User directory (where I don't want programs installing stuff) I intercept its initial build and always set an alternative location by editing the cfg file before first run on new installs.

If it's also going to become the principal repository of campaigns, then we need to be able to select the data directory on install.

On Linux (DEB-based) when we uninstall a package we have the option of "Removal" (just remove the package itself) or "Complete Removal" (remove associated files). I don't know how it applies to MT today, nor do I know how prevalent such an option is on other systems, but that sort of option would allow either leaving user data in place or cleaning it up, as the user desires.

There is only one "remove" option in windows that removes the install location. The data directory has to be manually removed.

[cwisniew]

I suggest allowing the user to choose the location of the directory. We can save that location in a config file in the ~/.maptool-retools directory. There are various reasons users might want to store the campaign files somewhere else (the ability to find them quickly being only one of them).

[cwisniew]

Given the amount of grief saving .cmpgn files under the ~/.maptool-rptools directory has caused over the years, what happens to this Campaign Manager directory when MapTool is uninstalled and reinstalled?

This was due to people saving campaign files in the install directory, ~/.maptool-rptools is not removed on uninstall, but I still wouldn't suggest we keep it all there.

[kwvanderlinde]

I suggest allowing the user to choose the location of the directory. We can save that location in a config file in the ~/.maptool-retools directory. There are various reasons users might want to store the campaign files somewhere else (the ability to find them quickly being only one of them).

Agreed about the option. Not sure about that particular reason as these are not intended to be user-facing files, but as you say there are also other reasons to do it.

[cwisniew]

I suggest allowing the user to choose the location of the directory. We can save that location in a config file in the ~/.maptool-retools directory. There are various reasons users might want to store the campaign files somewhere else (the ability to find them quickly being only one of them).

Agreed about the option. Not sure about that particular reason as these are not intended to be user-facing files, but as you say there are also other reasons to do it.

Backups, not storing it on a tiny C: drive :) Come to mind, we can put a readme in there saying not to touch, but I think having the directory somewhere the user is aware of makes it easier to back up, even if they are only copying it to a USB Flash drive. Hmm, we will probably have to warn them not to choose their USB drive for this folder :)

[kwvanderlinde]

The real issue, as I see it - and I don't know how common it would be - is that now, saving a .cmpg file requires a deliberate, conscious act on the part of the user, and pretty much everybody does it at some point. So they're aware of the concept, at least. If you automate the whole process, built in, a lot of people are never going to make any kind of backup because it doesn't occur to them that such a thing is possible.

This is a case of me not being clear. I did not intend to suggest that we would have no explicit save operation. I was still picturing an explicit save, it just wouldn't allow the user to pick a location or produce a .cmpgn file, instead storing the campaign in the Campaign Manager. I'll try to reword the FR for clarity.

(Just to be clear, I quite like the idea. I just think there's question it is far better to ask before anyone starts coding it, rather than after it's in a newly released version.)

Absolutely, there's no reason to shoot ourselves in the foot if we can avoid it.

[kwvanderlinde]

Another detail to think about: as a developer, I wouldn't want my dev builds of MT to use the user's Campaign Manager at all. It should use a completely separate one.

[kwvanderlinde]

For inspiration, an analogue of the proposed Campaign Manager for MT can be found in Minecraft. There's a lot of similarities between MC worlds and the proposed paradigm for MT campaigns, including:

• a single configurable location for worlds
• a UI for loading worlds which shows the user information about the worlds
• an internal world directory format that the user should not generally care about
• a way to export a world in the form of a ZIP file that can be shared

And this is what that UI looks like:

[cwisniew]

Another detail to think about: as a developer, I wouldn't want my dev builds of MT to use the user's Campaign Manager at all. It should use a completely separate one.

Possibly an easy solution to this is to add the ability to create more than one campaign storage, if there is only one, then MT starts with that one selected, if there is more than one it asks the user to select which, with possibly a command line flag to select so people could even create differing icons for each location if they really wanted (or your run command in ide could provide the flag :) )

The question is, do we want one storage and partitions in that, since even for a dev storage, you may want to share common image resources.

[kwvanderlinde]

Another detail to think about: as a developer, I wouldn't want my dev builds of MT to use the user's Campaign Manager at all. It should use a completely separate one.

Possibly an easy solution to this is to add the ability to create more than one campaign storage, if there is only one, then MT starts with that one selected, if there is more than one it asks the user to select which, with possibly a command line flag to select so people could even create differing icons for each location if they really wanted (or your run command in ide could provide the flag :) )

The question is, do we want one storage and partitions in that, since even for a dev storage, you may want to share common image resources.

That's not an easy solution, it requires extra complication in MT logic and UI to handle detecting / partitioning / managing these storages.

The easy solution is that MT only has one place to look for campaigns, no partitions. For devs, gradle configs will point MT somewhere away from the standard location, say, an ignored directory in the MT repo. For users, MT will use whatever location they had requested (during installation or preferences or however we do that).

[kwvanderlinde]

Given the amount of grief saving .cmpgn files under the ~/.maptool-rptools directory has caused over the years, what happens to this Campaign Manager directory when MapTool is uninstalled and reinstalled?

This was due to people saving campaign files in the install directory, ~/.maptool-rptools is not removed on uninstall, but I still wouldn't suggest we keep it all there.

Why not in ~/.maptool-rptools/? It seems a natural place for it, at least as the default.

[cwisniew]

Given the amount of grief saving .cmpgn files under the ~/.maptool-rptools directory has caused over the years, what happens to this Campaign Manager directory when MapTool is uninstalled and reinstalled?

This was due to people saving campaign files in the install directory, ~/.maptool-rptools is not removed on uninstall, but I still wouldn't suggest we keep it all there.

Why not in ~/.maptool-rptools/? It seems a natural place for it, at least as the default.

I think it's better if we let the users decide where they want the directory with the campaigns so they can back it up how they see fit. Afterall today they have a say where they can save their campaigns, and I can already see the first feature request "allow use to move where campaign storage is saved" :)

[FullBleed]

Why not in ~/.maptool-rptools/? It seems a natural place for it, at least as the default.

I agree that it kind of has to be the default... with the caveat that we're given the opportunity to chose the location on install.

Right now, the data directory location can only be set through editing the cfg file... which is in the install directory... which is removed on uninstall and not seen with a new install (since users can not install overtop their old installs without breaking stuff.)

So, my question is where can we store the user's chosen data directory so that it's accessible between updates? I think it will have to go in the OS registry.

[cwisniew]

Why not in ~/.maptool-rptools/? It seems a natural place for it, at least as the default.

I agree that it kind of has to be the default... with the caveat that we're given the opportunity to chose the location on install.

Right now, the data directory location can only be set through editing the cfg file... which is in the install directory... which is removed on uninstall and not seen with a new install (since users can not install overtop their old installs without breaking stuff.)

So, my question is where can we store the user's chosen data directory so that it's accessible between updates? I think it will have to go in the OS registry.

We can create a config file in the ~/.maptools-rptools directory that contains the location of the campaign storage. To be honest, the less we store in the Windows registry, the better. Just let the user pick where the campaign storage lives when they start MT if that file doesn't exist / is empty.

[FullBleed]

That will work, but if the data directory is not in the default location it'll have to be manually set with each install.

Actually, it looks like the data directory location is already stored in the Registry: HKEY_CURRENT_USER\Software\JavaSoft\Prefs\/Map/Tool\prefs\asset/Roots

[kwvanderlinde]

That will work, but if the data directory is not in the default location it'll have to be manually set with each install.

I don't really follow. ~/.maptool-rptools/ is not removed on uninstall, so the proposed file would be kept around and the location would not have to be set on each install.

Unless you're talking about relocating ~/.maptool-rptools/ in its entirety? But that goes beyond this FR and I don't really see the motivation for doing so.

[kwvanderlinde]

One concern I have with users choosing the location is doing things like pointing it at some file share. That would be bad news. If Dropbox or whatever is syncing files there while MT is also updating them, that's just asking for broken campaigns. They'd get away with it if we stuck with the .cmpgn format internally, but if we ever move away from that (and I hope we do) it could get dicey real fast.

As mentioned in the FR, I don't want users to expect to be able to fiddle with the internals of the campaign manager and still have everything work. Dropbox, etc., count as "fiddling".

[kwvanderlinde]

As for how a user changes the location, definitely not an install-time option. New users won't care and established users need to be able to change it for their existing installations.

Also not a fan of asking the user when they haven't yet made a choice. Same issues with install-time, but worse user experience. Every new user would have to answer a question that is meaningless to them, and 99% of them won't have any need - certainly not right away. Established users should be able change their minds at any point in time and shouldn't have to hunt down a config file to accomplish that.

So... just use the default location and don't ask the user about it. New users will be able to open up their newly installed MT and get going right away, no niche questions to answer beforehand. Users that need or want to change the location should be able to do so the same way they change anything else about MT behaviour: via preferences.

[FullBleed]

That will work, but if the data directory is not in the default location it'll have to be manually set with each install.

I don't really follow. ~/.maptool-rptools/ is not removed on uninstall, so the proposed file would be kept around and the location would not have to be set on each install.

Unless you're talking about relocating ~/.maptool-rptools/ in its entirety? But that goes beyond this FR and I don't really see the motivation for doing so.

The data directory location is not fixed (and we're talking about giving users the choice to set it on install). We even used to be able to set it in the app (Preferences>Startup) before we lost that ability (the field is still there we just can't edit it) and we now have to manually edit the cfg file (i.e.):

java-options=-DMAPTOOL_DATADIR=X:\Maptool\Data Directory\

On Windows, MT currently defaults creating the data directory on first run to a hidden directory in the User's root directory on their c drive (C:\Users\UserName.maptool-rptools). There are many reason why people might not want MT choosing that location (it's a nonstandard location to install things, it's on their boot drive, etc.)

As you know, once we change the location of the data directory (in the cfg file located in the install directory), that change is currently lost when we uninstall/reinstall MT.

And putting the location of the data directory inside a non-default located data directory won't help, because a new install of MT would only know to look in the default location.

Right now, many (most?) users probably don't even know about the data directory given where it's installed... at least not until they have a problem (like needing to load a backup). But if we're all the sudden going to give them the option to select its location on install (because that's also where their default campaign manager is going to be), then it becomes a very flexible folder... and MT needs to know where it is on updates or people will have to manually enter non default locations each time. Hence my suggestion to install the location in the registry (which, it turns out, MT already does) so on install it can just pull the key I noted earlier.

I hope I'm being clearer... As a user who on every new install of MT has needed to first go in and manually edit the cfg file to add a non default data directory location before running the first time, I know that this is a bit annoying. And while being able to set the directory on install will be better, having it know what I previously set would be even nicer (especially now that a consistent Campaign Manager location, which you've suggested should default to the data directory (and I agree), will be essential.)

[FullBleed]

As for how a user changes the location, definitely not an install-time option. New users won't care and established users need to be able to change it for their existing installations.

Also not a fan of asking the user when they haven't yet made a choice. Same issues with install-time, but worse user experience. Every new user would have to answer a question that is meaningless to them, and 99% of them won't have any need - certainly not right away. Established users should be able change their minds at any point in time and shouldn't have to hunt down a config file to accomplish that.

So... just use the default location and don't ask the user about it. New users will be able to open up their newly installed MT and get going right away, no niche questions to answer beforehand. Users that need or want to change the location should be able to do so the same way they change anything else about MT behaviour: via preferences.

IMO, MT currently does something very bad on install: It doesn't ask where users want the data directory installed. Few users are fans of programs that install stuff in non-standard locations... especially programs that leave the "hidden" directories (potentially full of gigabyte of cache data) behind after uninstall.

So I just don't see giving users the choice to change the "default" location for their "Campaign Manager/Data Directory" on install as being a bad thing even if it appears to be an "extra question" on install. Worst case scenario, MT should do as some programs do during install and provide a "Custom Install" button that gives users the choice to set those locations. Users that don't care can just gleefully click through and accept everything the program wants to install wherever it wants to install it.

But, connecting to the discussion above, there is little reason for MT to not be able to remember where users have set their Campaign Manager/Data Directory between updates. Until now I've assumed that the jpackage installer just was just too limited in allowing for such a thing... but if we're going to have a Campaign Manager (default to the data directory) I think now would be a good time to fix this.

[cwisniew]

One concern I have with users choosing the location is doing things like pointing it at some file share. That would be bad news. If Dropbox or whatever is syncing files there while MT is also updating them, that's just asking for broken campaigns. They'd get away with it if we stuck with the .cmpgn format internally, but if we ever move away from that (and I hope we do) it could get dicey real fast.

As mentioned in the FR, I don't want users to expect to be able to fiddle with the internals of the campaign manager and still have everything work. Dropbox, etc., count as "fiddling".

I understand the downsides, I agree with not wanting the users to interfere with the internals, and we can warn them against that. I also wouldn't expect the directories/filenames to be anything but the internal representation (so for example in the log run maps would be in a directory that was the ID of the map not the name, but ah that's future :) ).

I also agree it's a problem if they choose dropbox or worse SMB, which we can also warn them not to do that.

But -- and in my mind this is a big but -- at the moment, it's easy for people to backup their campaign files. It should be easy for them to backup their campaign storage in the future. Squirrelling it away in a hidden directory is antithetical to this, for all those people that ask "where are the files so I can back them up" there will be 2 or 3 that don't and therefore won't back them up. If a user doesn't back up their directory they create its on them, if they don't know they need to backup a hidden directory it's on us.

So... just use the default location and don't ask the user about it. New users will be able to open up their newly installed MT and get going right away, no niche questions to answer beforehand.

I don't think asking people on first run where they want to save their information (campaigns) is a niche question, so many applications already do that. I think this would be less surprising than finding out my information is stores somewhere hidden I can't easily find.

[cwisniew]

Thinking more about this and just throwing it out as an idea it might be beneficial to use git or something at least for the text files in a campaign. Saves (including autosave) can be just a write, git add, git commit, git tag with with the save version. (this is just a quick off the cuff example it obviously will need more thought behind it, especially when you consider non text assets). Saves us having to keep autosave backup 1 to x lying around in there.

[bubblobill]

Less concerned with the location and more with the Campaign Manager.

4818

It seems the ability to rebuild campaign files would fall under a Manager's scope.