Improve the skeleton for shared hosting

[franzliedke]

I'd like to talk through ways that we can improve the experience for people on shared hosts.

References:

• https://github.com/flarum/docs/pull/42
• https://discuss.flarum.org/d/17779-a-big-mess-after-updating-to-beta-8

My favorite idea would be to add an index.php with the correct paths in the root directory as well. It wouldn't be reachable in a "normal" virtual server setup, and shared hosts would only need to remove the public/ folder and be ready.

What do you think? Any other ideas? Other small improvements we can make to reduce the amount of customization necessary?

[tobyzerner]

They'd also have to move .htaccess and assets out of the public directory. I'd prefer not to duplicate index.php to be honest, because it could be confusing to someone (especially devs) looking at the skeleton without any context – they'd be like, why are there two front controllers? Which is the real one I'm supposed to use? It seems a bit... sloppy?

What if we made index.php somehow auto-detect where it is, so it works both in the root and in the public dir without any changes?

Or include commented-out versions of the paths in index.php, so all people have to do is move it out of the public dir and uncomment the correct lines?

Or as per my suggestion in flarum/docs#42, we could have a site.php file which returns the Site instance (with all the correct paths), and then flarum and index.php just include that, and then people would only need to update one path (or it could be auto-detected).

Or we could just revert to the old no-public-directory skeleton, and talk about creating a public directory in a "security hardening" doc. (I suppose if we just accept that shared hosts are always gonna be our biggest support burden, this is probably the wisest solution)

[franzliedke]

it could be confusing to someone (especially devs) looking at the skeleton without any context – they'd be like, why are there two front controllers?

Easy enough to fix with a friendly, explanatory comment, don't you think?

Or as per my suggestion in flarum/docs#42, we could have a site.php file which returns the Site instance (with all the correct paths), and then flarum and index.php just include that, and then people would only need to update one path (or it could be auto-detected).

Definitely, this would reduce the duplication between flarum and index.php. This doesn't necessarily exclude other measures, though. If we go for the site.php thing, we should also create it in such a way that only the public path needs to be changed for shared hosts.

Or we could just revert to the old no-public-directory skeleton, and talk about creating a public directory in a "security hardening" doc. (I suppose if we just accept that shared hosts are always gonna be our biggest support burden, this is probably the wisest solution)

Dunno, I thought we had agreed to go secure-by-default, easy-enough for shared hosts (as we don't recommend them).

---

In any case, instead of explaining that certain changes need to be made in certain files, we should simply provide an alternative index.php that can be fully copied when the root directory is the public dir.

[luceos]

What if we create a post install cmd script for composer that asks (not sure that's possible) where the public directory is and patch the files based on that information?

[tobyzerner]

Dunno, I thought we had agreed to go secure-by-default, easy-enough for shared hosts (as we don't recommend them).

True.

My preferred solution then is to simply have the site.php as we've talked about, and then spell out the rest very clearly in the docs (including the new contents of index.php to copy and paste).

By the way, regarding site.php... it might be nice to merge it with extend.php:

<?php

use Flarum\Extend;

require 'vendor/autoload.php';

$site = Flarum\Foundation\Site::fromPaths([
    'base' => __DIR__,
    'public' => __DIR__.'/public',
    'storage' => __DIR__.'/storage',
]);

$site->extend([
    // Register extenders here to customize your forum
]);

return $site;

[franzliedke]

I would leave extend.php separate, this makes it slightly easier to keep changes there intact in case that site.php is ever changed in the "official" skeleton.

[luceos]

In a recent implementation I actually added a bootstrap/site.php and use that from inside both the flarum binary and the public/index.php. I'm not sure we can simplify this further though..

[andreherberth]

What if we create a post install cmd script for composer that asks (not sure that's possible) where the public directory is and patch the files based on that information?

Would it be possible to create a custom command instead, and let users run it with composer run-script after installation if they want?

Would provide some quality of life, and not take away from the security first argument, as far as I know.

[tobyzerner]

All those in favour of bootstrap/site.php and bootstrap/extend.php say aye

[luceos]

Aye. What about a post cmd script and/or some directory identification?

[franzliedke]

Aye.

some directory identification

Can you expand on that, @luceos ? Do you mean writing these files in such a way that they can deal with both typical setup scenarios from the get-go?

[luceos]

Yeah. The same way your studio binary works. This makes most sense for the index.php as we can assume (and potentially recommend) to keep the flarum binary in the root path so we can use that to identify the root path from the index.php..

[relax4o]

I am intersted in contributing in this project and just started reading some of the issues to get familiar with the software.

I haven't really read the whole conversation here but I've got an idea about the shared hosting. Many providers are using Softaculous for installing wide range of software.

I was just wondering, what do you think about adding Flarum into the software list of Softaculous? It will be the easiest and most convenient way for the users of shared hosting to use this forum.

I am not completely familiar with the process but I was thinking that'll be something worth to look at.

[franzliedke]

@relax4o Thanks for your suggestion. As I think Softaculous is a bit off-topic for this ticket, feel free to discuss this on our forums.

[franzliedke]

The skeleton changes (flarum/flarum#63) alone are not good enough. What we came up with in today's meeting:

• Add a console task that can set up the recommended skeleton for either shared hosts, or "real" servers (i.e. those that cannot change their document root and those that can).
• Add a post-install message with Composer that recommends running this command.

Depending on how long this will take me, I might also look into making the (web?) installer:

• warn when there's public into the URL, and
• complain when AJAX requests to endpoints like vendor/composer/installed.json succeed (those must be hidden, either through a secure document root or additional server configuration, .htaccess or differently)

[askvortsov1]

So I've been thinking a lot about the current approach we're taking, and I'm not sure the approach we have planned (https://github.com/flarum/core/issues/1669, https://github.com/flarum/core/pull/2008) is the best way to approach this. I'm honestly not that big a fan of moving folders and files around the skeleton during operation. It's possible that this has been discussed before, but I was thinking of the following approach instead:

1. Get rid of the public directory completely to standardize 1 skeleton structure. I realize this has potential security implications, but plenty of systems do this (e.g. wordpress), and this is mitigated by the following:
2. Add the 403 response to /vendor, /config.php, etc to the .htaccess and .nginx.conf by default, don't comment it out.
3. In prerequisites to install, require that all these sensitive paths are restricted with appropriate file permissions, and won't be served by the web server regardless of what server is used. Also, add a middleware that shows admin users a big red warning if those paths are accessible on an existing installation.

This solution would be much cleaner, wouldn't require parts of flarum/flarum to be duplicated in core (I think decoupling the skeleton and the core system is a good thing tbh), and wouldn't rely on renaming/moving files at runtime (which I think we should avoid outside of possible extension management). It would play nicer with alternate deployment mechanisms (e.g. Docker), require less core modifications and less maintenance overhead, and it would be a lot simpler for users to manage, and could make 1-click installation more feasible. Even more importantly, it could provide a better base for internal routing that might improve support for hosting flarum on a subdirectory (plus this could literally be used to host Flarum on a subdirectory by moving it).

The only perceivable downside is potential security implications, but those can be mitigated via (2) and (3). Additionally, it has been done before (by wordpress no less), so I'm 100% confident that we can do this without losing security.

I realize that Franz is on vacation and rest of team is busy with release, so no rush to discuss, but I think this could be a much better way of handling this.

[askvortsov1]

So to recap key points from today's discord conversation with myself, Clark, David, and Tanker:

1. There is still not clear agreement on what to do with the skeleton / one-click install
2. Restructuring the skeleton from inside the application via console is clunky and probably not the best approach. The vast majority of forums set up their skeleton exactly once, and after that, they only care about updates. If we can try to make each skeleton as forward-thinking and updatable as possible (possibly creating shell scripts for installing / updating), and ideally that can be easily replaced, that could be awesome for our users.
3. There is strong opposition to using index-in-root for all forums by default
4. There is strong support for making Flarum compatible with one-click install systems like softaculous
5. One viable option is having 2 separate skeleton repositories. The current one could be used for VPS/dedicated/docker/etc, and the new one would have index-in-root, and would be tailored towards a one-click install for non-technological users. Core doesn't care where index.php is, which is awesome!
6. Composer is still an issue for shared hosting one-click install (does Bazaar use composer?) It might make sense to (as we near stable, not now) make extensions uploadable via a ZIP file.
7. OctoberCMS has a very nice installer, could act as inspiration for our shared-hosting focused installer (1-click).
8. For shared hosting one-click, we could package the shared skeleton, core, and bundled extensions in a zip file that people could download and unzip.

[askvortsov1]

Here's a POC of my idea for this: https://github.com/askvortsov1/flarum-ez-install. As of right now, the biggest barrier is Bazaar being slow (because it depends on composer).

This (or something like it) essentially fixes the issue of installation and core/bundled extension updates on shared hosts. I've also put some pretty basic logic to support systems that might already be installed with older skeletons. But essentially, installation no longer requires that composer is available, rather, you just need to copy-paste 1 line of code into a terminal.

It works for updating as well. It will update the skeleton, core, bundled extensions, and bazaar to their latest versions. If skeleton/file permissions/etc changes in a given release, this repo's flarum.sh script can also be updated to automatically make those changes, meaning that for shared users, upgrading is as simple as copy-pasting 1 line of code (or 1 click, if we can partner with the shared hosting providers!).

Additionally, the security settings effectively block access to resources that should not be accessible (e.g. config.php). You can find an example of a site installed this way at http://iyna.dreamhosters.com/.

Anyway, that's my proposed solution for shared hosting installation! Open to any feedback/suggestions/ideas.

[askvortsov1]

Now, given an easy-to-user system, the major bottleneck for shared hosting is extension install/remove/etc speed. This is something we're going to get regardless of whether flarum is installed via composing or something like above.

From discussion on Discord, setting up our own Satis server for flarum extensions only should massively reduce effort needed by composer to resolve/install dependencies. We could use something like extiverse to register new extensions and update the satis. Testing by Matt confirms the efficiency of this approach. Of course, this would be a post-stable project, but imo it would be very valuable to any flarum installation, especially on shared hosting.

Also important to note that this composer issue is a complete tangent to the "improve shared hosting installation" matter that this particular thread is for, but still something to consider.