Support grain patching as an advanced feature

[ndarilek]

One issue I'm encountering with the DokuWiki app is the need to introduce config files and theme modifications into the grain's mutable storage. Unfortunately, Doku expects to find its themes/plugins in its own source tree, and there's no easy convention for storing user modifications in a separate directory. Poor design on their part, but that's what we're stuck with.

What this means is that I'm copying out parts of Doku's installation files into directories in /var, then linking those directories into the source tree. This lets users download a grain backup to tweak their theme, install new plugins, etc. then repack the backup and upload their modifications. I've also had to tweak the stock theme to add Sandstorm support for path and title changes.

Unfortunately, this has a few downsides. If someone modifies their theme or installs a new plugin into the backup, they can only create a new grain. They can't upload a backup to replace the existing grain, or better yet, upload a diff containing the metadata and only the new files. Also, some of the Sandstorm changes I've introduced require that files be added to /var. Short of writing complex migrations to check for these changes, I don't know how to resolve this. Further, someone may have customized their own versions of these files, so I don't want to just replace them.

I'd be interested in seeing an Upload Patch feature of some sort. Patches would only contain files you'd like to modify, which would be replaced completely and in-place. The directory structure would be identical to that produced by backups, so to make a patch you'd start with a backup and then remove all files you don't modify. I suppose you could also just leave the files in place, since the patch would just replace them with their identical copies.

I'd consider this an advanced feature. If it was not included in the toolbar and had to be accessed in some documented but hidden way, that's fine. I expect the need to keep tweaking Doku's files to level out over time, but I'd like to offer folks an easy upgrade/tweak path, and short of just trashing their files I don't know how else to do that.

Also, I suppose this need is mitigated somewhat by sharing a grain via a collection. I could take a backup, make my tweaks, upload a new grain, delete the old, then share the new grain with the collection. But I can't guarantee that someone hasn't edited the wiki between me making a backup and uploading the modified archive, particularly if the archive is large and takes a while to upload. In a traditional setup, I could shut down the service for maintenance. Patching would be quick, and would guarantee continuity.

Thanks.

[kentonv]

Hi Nolan,

Hmm. Ideally, what we would really prefer here is for the app to provide a user interface to configure it. If people have to edit config files to make the app do what they want, then non-technical users are not likely to be able to use the app. To some extent it was an intentional decision on my part to make this difficult, in order to force apps to provide UI.

I know that's not really what you want to hear when you are merely packaging an existing app which, unfortunately, does not provide any such UI. Implementing one for them may be a bit much.

Perhaps as a stopgap, though, you could implement a page which is basically a big textarea preloaded with the content of the config file, and lets the user modify it and submit it?

[ndarilek]

My hands are so full that I probably don't have time to add such a feature to every app I maintain, then continue maintaining that code perpetually. I'm a bit apprehensive at all the changes I've made to get Doku working, and even so I haven't resolved all the issues there. Even if I added a config file feature, as you suggest, I'd then have to do the same for plugins and themes. As of now, I don't think Doku lets you update themes remotely. They need to be uploaded onto the server.

I understand that you're all busy right now, and I wouldn't call this a critical feature. Worst case, I suppose I could manipulate the files on the server directly. But I hope this feature gets considered. I understand the target you're aiming for, but am starting to feel like the opinionated nature of Sandstorm restricts me much more than it frees me. For instance, I get the benefit of not letting apps communicate to the outside network, but in the case of Doku, that also limits my ability to easily install plugins or detect plugin updates. I'm willing to accept some of these tradeoffs because I understand the benefit, but I worry about hitting a point where these opinions limit me in such a way that I just can't get my job done. Having the ability to patch existing grains would mitigate the fact that I can't easily use Doku's plugin manager anymore. Without it, I think I'll worry that we may hit some requirement that I can't meet without jumping through lots of Sandstorm-specific hoops.

[funwhilelost]

This is an interesting train of thought. Your current method of downloading the backup, editing, and restoring it with new configuration hadn't occurred to me before.

It would be interesting to have a Sandstorm-powered facility for editing certain config files as a "big textarea". If you could just specify an .ini/.php/.json file in the pkgdef, for instance, then have an editor that's provided by Sandstorm it would be great!

It would give people an "under the hood view" who needed it.

[ocdtrekkie]

@funwhilelost Such an editor would need to be built into the app, so it can modify things in the grain storage. And obviously app packagers could implement that sort of thing if they wanted to, but it sounds like @ndarilek doesn't really want to do that.

I do think there's a certain point, and this is very personal opinion territory, where some apps make less sense for Sandstorm because of the difficulty of putting them in the Sandstorm model. On a different tack to this particular one is something like ownCloud, which is such a large multi-user multi-faceted app that it would be very hard to fit into the Sandstorm model well. My hope is that in time, Sandstorm-native and Sandstorm-targeted apps will replace these.

[kpreid]

a Sandstorm-powered facility for editing certain config files as a "big textarea".

Such an editor would need to be built into the app,

I would suggest that this is something that can and should be done as inter-grain functionality. All that is needed is a standard protocol for exporting a filesystem over Cap'n'Proto; the config-bearing app can use a drop-in library to implement the protocol (possibly with parameters/customizations to restrict what can be edited), and the editor app is a client for the protocol.

This approach needs no new features in Sandstorm itself, allows there to be more than one editor UI, and does not break the encapsulation of the grain while it is running.

[ndarilek]

I would be fine with a Sandstorm-native grain editor that could edit /var of another grain, maybe some sort of file manager. I wouldn't be fine with it limiting what I can edit. I'm starting to really worry about some of Sandstorm's limitations. I think that lots of them make sense, but many are in place with no current workaround (I.e. powerbox) for those of us who need them. I'm starting to hit a point where Sandstorm certainly makes things easy, but also imposes a glass ceiling whose limits I don't anticipate until I need to surpass them, and there really is no current way to do so. It's lock-in, but of a different sort.

[kpreid]

The “unlimited” version could exist as a feature in Sandstorm itself which allows one to edit a grain filesystem using the same FS protocol I proposed — this is “just” a more convenient version of “download the backup and edit” which works without involving local filesystem. If you like, read me as saying not “don't allow editing grain FS” but “don't put the editor UI in Sandstorm”; I actually thought of mentioning this option when I wrote the first comment but left it out as non-central.

The motivation for my proposal above is not to limit what people can do, but to promote these goals:

• Access control that is not all-or-nothing — in particular, “edit what the grain exposes” can work for non-grain-owners, but “edit the grain's filesystem unrestrictedly” cannot.
• Keep Sandstorm-per-se, the “kernel” of the system, simple.
• Allow nontrivial UI components to be apps, and therefore replaceable.

[ndarilek]

Got it. Sure, I'd support some mechanism for selectively exporting a grain's FS while allowing owners edit access to everything. Though at this point I can imagine several use cases for the latter and none for the former. Actually, maybe a Gitweb grain could export its repository so another app could consume it, then maybe an online IDE could connect directly to a repository for commit access. So that's one for the former.

In any case, I'd support this as an alternative to grain patching. I just don't want lock-in that makes some fairly practical and otherwise simple tasks impossible, and the "modify the app to support this" line doesn't really scale when one person is maintaining half a dozen ports. Maintaining multiple modifications is probably one of the biggest factors I consider when porting an app to Sandstorm, and I worry that I'll start something I won't be able to finish or maintain, and then folks are depending on me to keep delivering.

[kentonv]

Shouldn't it be "pretty easy" to drop in a .php file which implements a full-page textarea for editing the config?

[paulproteus]

For @kentonv: I think @ndarilek 's perspective can be understood by considering the difference between O(1) and O(N). @ndarilek would need to add such a file to each package he maintains. It could get out of sync/old. He could mess up the permission checks for it.

One way around that might be if vagrant-spk ships with something to this end, but vagrant-spk would presumably want to make sure that the permissions checks for it are correct, and maybe other things. Then it'd be O(1) for Nolan since "the tooling provides it for him".

My perspective is that I very-much like @kpreid 's "grain filesystem editor app" idea.

[paulproteus]

Also such a PHP file would either need to integrate properly with the existing UI/authorization/etc. of the app (e.g. DokuWiki) which substantially increases the cognitive load over time (upstream changes might break it) and increases the cognitive load per app, or Nolan would need to decide that it does not need to integrate properly with upstream's UI/authorization/etc and it could be a a separate file like /adminbackdoor.php . The latter is a somewhat sad decision for an app author, I imagine. Such a file would either need to be customized per-app to reference the specific file that needs to be changed, or it would be a 100% identical-between-apps generic PHP file manager app, in which case it would feel somewhat frustrating (I presume) to untar the same generic PHP file manager app into each separate Sandstorm package. Additionally, this is a delta that upstream would not want to merge.

[kentonv]

I mean, I hate to say this, but while all the ideas in this thread are reasonable, this just isn't something we're likely to be able to prioritize anytime soon. :/ So I was trying to suggest a stopgap...

[ndarilek]

Thanks, Asheesh got it exactly right. The other fact to consider is that I don't know PHP. I mean, I know it in the sense that I know any simple programming language and can pick up the specifics quickly on learnxinyminutes.com. But I can't guarantee that any PHP file uploader I wrote wouldn't be full of bugs/security issues. Sure Sandstorm shields me from that somewhat, but maintaining an uploader that honored Sandstorm permissions in a language I don't use daily or care to invest time learning because I won't use it for anything more than this one thing, not the simplest. :) Then factor in Haskell, which I can't wrap my brain around easily, used by HLedger. Maybe someone wants to upload an existing ledger, but the current app doesn't let them do so yet. Writing an uploader in Template Haskell-filled Yesod and learning how a dozen poorly-documented compiler directives augment the base language, not really my idea of a fun weekend. :) Or Rust, which I'm learning and like but don't yet feel solid in. Or Ruby, for my Loomio port, and I'm not terribly interested in relearning complex Rails with a million gems, all monkeypatching each other and doing all sorts of magic. ...

Kenton, sorry if I misread your message. I assumed that the plan wasn't to support this, and that modifying each app was the ultimate solution. Like I wrote originally, I'm fine with this being lower-priority, but could see myself hitting this down the road at some point. If I won't ever get it, it might be a major concern now. Thanks for making a good suggestion. I apologize if me shooting it down seems ungrateful--it was me reacting to my misperception that this was all we'd ever get. :)