Scaling enduro server

[adrianObel]

First of all great work @Gottwik and the rest of the contributors. This truly is one of the simplest CMSs to use and has plenty of great features.

My concern is with scaling enduro, and specifically enduro admin, past a single process. I'm using enduro in production and this is a big point we need to solve. It seems like by design it wasn't meant to scale but we've made a handful of changes that gets enduro one step closer.

The first problem you run into almost immediately when scaling past a single node instance is that sessions no longer work. Sessions are stored in memory as can be seen here. This doesn't work for obvious reasons. We forked enduro and implemented JWT based sessions. briefly discussed in this thread.

The second problem is serving the actual content. Enduro being a flat-file CMS means it actually needs to write the file to disk and unless you're running some sort of distributed FS this will be a serious problem. It seems to me that juicebox was created with the intent of running enduro admin on PaaS, like Heroku. Juicebox and the remote FS works great for these volatile services where your instance is taken down and restarted every so often. This is a great step forward but, again, won't scale past the single instance. Have you or anyone run into this issue?

Sticking with the flat-file and remote filesystem architecture in enduro and juicebox a cool feature might be to, on publish from the admin the generated file(s) be pushed up to s3. Not only as the tarball'ed backup but the generated files themselves to some other bucket that can serve the files.

These features of course increase the complexity of enduro but if people plan on putting their enduro sites to real use it's be worth considering.

Let me know what you think. We're eager to contribute to the project!.

[Gottwik]

wow :-) Sounds like you though it thought more than I did. Thank you for your comment, psyched with your ideas.

Originally I wanted to make things as simple as they can be. You are right that the juicebox was added for cases when you don't have persistent storage and to make this work on heroku.

(1) feel free to contribute. I am happy to add you as collaborator.

(2) I am starting to feel building separate layers/addons/plugins is better than adding functionality. Enduro started super lean and mean and got somewhat cluttered over time. Could we do something like enduro cdn syncer(cool name needed) that would serve generated website for cutting edge performance?

[adrianObel]

Great! :)

I feel like there are two problems we need to solve.

1. How do we serve content from a CDN.
2. How do keep multiple enduro instances in sync.

One might be the solution to the next but it's not exactly the same issue.

To accomplish the second point anything that falls under cms/ would need to be stored in a bucket. On admin page load we could pull the remote cms directory to power the admin panel. Any changes or new pages would then be pushed up to the bucket. Basically, treating the bucket as the filesystem. This is very close to what enduro does currently so I think it's more than half way there.

Now for actually generating the content I'm not entirely sure what would be the best solution. Since the pages have the same name as the the json files we could potentially pull the json from the bucket and compile the template at runtime. Pages can be cached so I don't think it would be terribly slow.

The solution would keep all enduro instances in sync and allow you to run a cluster without any issues. I also feel like this might be the successor of juicebox since you would no longer need to keep another set of backups. If necessary, versioning could be done on the vendor side.

A follow-up feature/addon/plugin could then be to push all of the content to a CDN.