On OctoberCMS' file-based philosophy

[harmenjanssen]

This is not an issue with a clear solution but a starting point of discussion.

At our company we're on the brink of committing ourselves to OctoberCMS for all our CMS-needing clients in the foreseeable future (and willing to become active community-members and offer PRs and plugins).

We do however, continually wonder about the file-based approach of the system.
It has a couple of downsides that we just can't solve, that are none-issues in systems where content is part of the database instead of disk:

• Deploying to a cluster of webservers. This would be the main one: how on earth can content be managed when it ends up on one of the disks of a webserver, but not another?
• More generally: every deploy would overwrite htm files generated on the server. You could symlink the pages directory outside the deploy target dir to solve this but that would disconnect the page from updates to the layout coming from version control.

And at the other end of the spectrum, a lot of things that I traditionally put on disk are in the database, such as backend settings.

I know there's the Static Pages Plugin that solves this, but it's a general philosophy that trickles down to plugins from the community as well. Since file-based is the flavour of the framework, a lot of plugins are not usable for us, because they adopt the same strategy, or extend the native file-based CmsPages.

I would like to discuss these points with the core developers, to get a better understanding for these design choices because it feels like we're going against the grain 80% of the time and I'd rather work with the system than around it.

Thanks in advance!

[fuzzybaird]

I think you are approaching October CMS a little wrong if you are thinking of it as a pure flat file CMS. Yes it gives developers the capacity to do a very basic site that is "pure" flat file, but its real power in my opinion for larger scale sites is in the plugins and components design patterns. What I love about October across our sites is that each piece of functionality we want to give to our business partners is neatly tied into a plugin, that has fully capable models, controllers, and views that elegantly extends the backend with easy WYSYG & Crud Capability.

So in essence if you think about october more like a Content Management Framework instead of a Content Management System the builds come a little more naturally. You can in minutes build exactly the functionality your clients or partners need to edit, and then just integrate it into your views.

So in answer to your first thought, we deploy in clusters of up to three servers per site, and simply choose to use a database driven plugin architecture so that we don't have to deal with flat file at that scale. And that feels incredibly natural in october.

We have done a business user implementation where they can edit the flat file CMS portion of October and our solutions to both of your bullet points were first, Git Version control. second, Having the business user edit on stage (a single server) and then when we are ready to go live we simply pushed those changes. We toyed with writing a plugin that was a deploy button that the partners could push to get their changes on prod but abandoned it in favor of the database editing approach.

The CMS part in our mind is a developer tool not a partner tool. It allows us to work with the plugin community, and then extent that functionality using our own backend CRUDs for partners.

I honestly don't know what you mean about the settings on database vs filesystem. If you have ever worked with Drupal, Wordpress, or god forbid AEM, 100% of settings are in the database. The fact that october has a valid .env structure, along with overridable configs is a solid step in the right direction.

I am not a core developer, but I am 2 years enterprise october developer and these are my thoughts on your questions.

In short. October is going in an amazing direction for my team's needs. KEEP IT UP OCTOBER TEAM!

[that0n3guy]

@harmenjanssen I definitely understand where you're coming from. Here are my thoughts:

CMS on filesystem

• We treat the CMS portion more like a "theme" editor. Theme files are kept in GIT and clients wouldn't mess with those... I mean they could, but it would just get wiped next deploy (and they know that). If you like the static pages plugin, you could still use it, but mount the "content" of those pages on the filesystem (see next bullet). Its probably just easier to use the database though.
• We also use a distributed file system. We use docker and don't want to care where our containers are or what host they are on. Distributed file systems have come a long way in recent years, because of the very problem you'r seeing. The easiest way to do this is with NFS (or EFS on AWS), but the real neat stuff is like gusterFS or longhorn (I've not tested this, but someday as we use rancher for docker management) or a bunch of other options, I've not tested..

If you don't want to do a distributed filesystem, I mostly say just don't fight it. Think of the CMS portion of the backend as off limits (use permissions) since its for theme editing and just use it on your local (or ignore it 90% of the time and use your IDE). If you want to add components to DB plugin content, I think you can use https://octobercms.com/plugin/toughdeveloper-richeditorsnippets (not tested yet).

Most (id probably say all, except static pages as I've not seen to many others) plugins store their data in the DB.

Settings in DB

As for settings being in the DB, that is pretty common for most CMS's. Some ways around it:

• use config files for all of your plugins since you have control over it
• for other plugins, write migration files for any DB changes you want in your own plugin.
• or, I'm guessing you could create a simple plugin the that would convert a custom config.php/yaml file to write to the Settings model of other plugins. This could be set to push everytime you deploy. This would save you from all the overhead of the migrations file idea. This is what I would tinker with first and would probably be very easy to do.

[harmenjanssen]

Thanks both of you for your insights.

@fuzzybaird I appreciate your cheering the October team on, but just to clarify: this is by no means an attack, merely the start of a conversation. I want to find the natural way of things and adapt a workflow that fits the system, and feel like a discussion about the design choices helps this along. When in Rome, do as the Romans do, and all that.

You both basically describe the route we're going now: just ignore all file-based plugin in favor of database-backed ones. We're actually working on a Pages plugin that allows you to add something like Wordpress' advanced custom fields, and actually embed components in the database-backed content. This helps a lot.

So you guys don't feel half of the plugins on the marketplace can't be used because of a file-based approach?

Settings in DB

To be honest, maybe I've just had bad luck with the plugins I've used, but I feel like a lot of the plugins store settings exactly the wrong way around. Not to name-and-shame, but to give two examples:

• ErrorLogger plugin stores my Sentry API key in the database: makes much more sense in .env
• CacheRoute plugin stores all configuration in the database. A plugin that's meant to give a great speed boost shouldn't have to look for config in the database, honestly.

I understand this is not OctoberCMS' responsibility, but I feel like the ecosystem somehow favors the above design, where someone can click-to-install, click-to-configure, all from the web browser, over programmatically installing and configuring.

This becomes clear as well when installing a theme with its required plugins. The documentation promises all required plugins will be installed when installing a theme. I'm sure it works when installing thru the marketplace, but running php artisan theme:install most definitely does not install all required plugins.

Also a bit unpolished is the unit testing environment. I've spend a lot of time debugging a plethora of problems in the testing environment. First plugins are not registered, then plugins are not booted, and right now I'm having problems because the app instance of a plugin seems to be corrupted after running the first test. (I will offer up a PR to fix this this weekend)

It's okay to have to debug of course, but all this stuff together feels like the intended audience of October primarily is a developer who clicks a website together in a browser, versus one doing things in a more automated, programmatic way.
We're unit testing, deploying to clusters, auto-deploying, continuous integration, and we have to fight to make things work sometimes, which is frustrating.

Having said all that, it's good to meet developers who seem to be doing all that stuff in October 🙂

[harmenjanssen]

Oh, to be precise: I was assigning tasks to my team members and wanted someone to modify the WYSIWYG editor settings and backend branding to match our needs, but then realised this wouldn't do anything for my system because everything would be in their local database.
That's also what fuelled the Settings in db point.

Right now I'm thinking of writing a console command that populates these settings from a yaml file which is run on deploy. To me it doesn't make sense for content managers to brand or configure (most of) their backend, it's part of the product we build for them, and therefore should be in version control where we can automate and repeat it.

[munxar]

@harmenjanssen Just an idea: If we would write an eloquent driver that uses the filesystem with a human readable format like json, yaml or xml, we would end with a full file based cms. even plugins that use the db would work. the amount of work would be huge, but not impossible (example: https://github.com/jenssegers/laravel-mongodb). Imho if want to go pure file base octobercms is NOT the way to go, for that I'll use kirby or grav.

[LukeTowers]

@munxar it's not as human readable as YAML or JSON, but SqlLite is already supported as a database option.

@harmenjanssen in regards to your settings complaint, I currently use the following structure in my plugin's configuration options:

• Attempt to load it from .env
• Load it from config/myplugin/config.php
• Load it from myplugin/config/config.php (these two steps are handled automatically by October)

If I include options that are available in the database, then I provide the file config for them but allow the database settings to override the file based config options.

In theory, this could be supplied as a plugin that does that for you I suppose, which might make your life easier.

[harmenjanssen]

@LukeTowers Great, yeah, that's exactly right, I'd say 🙂

[CSNWEB]

We have also used it for cluster deploys and had two several single server development environments one for content editors and one for developers. Then the content editors could change the page with the static pages plugin and afterwards push and deploy their changes via git. One clear advantage of a file based approach vs. database I see is that you can easily merge different updates together thanks to git (e.g. (local) changes by devs and changes by content editors or if you working on several drafts at the same time)

[harmenjanssen]

@CSNWEB I've read other people's accounts about a solution like that, but honestly... it feels just way overcomplicated to have to commit changes back to the repo from the webserver.
It's also a matter of resilience: you want the web servers in a cluster to be redundant and to be able to die at any time and just be booted up again or cloned or whatever without losing content.

A release should be a package, checked in to source control, which deploy can be repeated on one or more target machines, scripted, and yield the same results every time.

[LukeTowers]

@harmenjanssen we've got some changes planned to the RainLab.Pages plugin (which could in theory be extended to the CMS section) that would store page content in the database and only pull from files when a database version didn't exist; populating the database in the process. Then you could still have the best of both worlds.

The idea being using the database as a non-mandatory augmentation to the existing system.

[harmenjanssen]

@LukeTowers Sounds good, I'll keep an eye out for that.

[that0n3guy]

Attempt to load it from .env Load it from config/myplugin/config.php Load it from myplugin/config/config.php

@LukeTowers I like this, its something that should be built into the settings model. Then existing plugins and sites would keep working (as there is no file config), but could be used by us who want the settings in File. Personally I would make file config override DB though.

[LukeTowers]

@that0n3guy why would you make file config override DB? File config would be stored in a repository that gets deployed everywhere to everyone on the team, if you need instance specific settings changed then that should be handled in an instance specific data store, which the database is.

[that0n3guy]

@LukeTowers. Most of the time I don't want specific settings on a per instance basis. I want my local devs to have same as staging and same as production. The DB makes that more difficult. In a local->function testing->staging->production deployment type of situation, I want all the configs the same.

If DB overrides and someone goes in and changes something (lets say on staging), my team would be looking at the config saying "why is this not working on staging?".

[harmenjanssen]

@that0n3guy How would you turn that around though? How would someone make an exception on staging when the file has the final say?
There's no way to not have the file right? Either it's on the server or it isn't, and it is, therefore changes to your DB would be moot. Right? Maybe I'm missing something. 🙂

(Don't get me wrong – it's a real problem. Therefore I would chose to have some stuff only in DB and other stuff only on disk. And invest in a handy database-migration tool.)

[that0n3guy]

@harmenjanssen You are right, we do want different config on local then staging/production for some things like sandbox api keys, etc... but its environment specific or sensitive, neither we would want in the DB. So environment vars or .env files. Why do I want my staging different then production? Thats the point of staging :).

For the <1% use case where I need to modify a setting in staging, I'll just login to docker console and change the config file (or add a .env) next deploy will destroy my change, but it lets us test and see things.

[CSNWEB]

I agree with @that0n3guy @LukeTowers @harmenjanssen. We generate our environment files based on the system environment variables (set by kubernetes or docker-compose) and at least we have a dedicated db cluster for staging so it doesn't affect production. Getting up a different cluster with a different environment variables and a dumped db is pretty easy, having to go to the database to change settings adds an extra step. But more importantly, it's more about making the config fixed and the files authoritative so you know exactly how the system is configured.

[LukeTowers]

@that0n3guy I would say that settings suited to the DB being authoritative are moreso along the lines of cosmetic changes or per user preferences.

[milankubin]

This is the typical CMS/SCM conflict and not really unique to October. The WordPress approach is actually way worse to version-control because more stuff resides in the DB. I'm fairly new to October. Pretty much started my first project this October :))

Settings in DB

Just coming from a phpBB contract project which uses symfony components and plugin based architecture, this was pretty much a no-brainer to write a plugin that versions migrations and DB seeders ( for third-party plugins with model based settings). Even better with October contrary to phpBB, you can mix migrations and seeders in your updates folder and version up/down accordingly. The downside of this approach is that it makes it difficult to make a composer package out of this simple plugin, because the updates would be project specific, and thus they need to be moved or symlinked into the plugin folder post install.

I do feel this stuff could use some documentation, because talking to some devs this wasn't all that obvious to them.

Pages / static content.

I agree fully with @that0n3guy I feel pages/layouts in general should be left alone once they moved to a post dev stage. Layouts and pages structure content. I see content as blocks of editable text with maybe some images or media, that should not impact layout. Client should be able to easily edit his contact details or some other parts of his site that are not managed by a plugin, or require a new deployment. NFS/EFS seems like a good approach for this.