Running in multi-container/node/zone deployment

[jakubknejzlik]

Hello, I'm new with strapi and it looks very interesting for many of our projects. One thing that I'm struggling with is how to run containerized strapi with multi-container/multi-node deployment (k8s, ecs etc.). The issues/questions I've discovered so far:

1. each container needs local volume to install plugins and dependencies, is it safe to run multiple containers mounted in same volume? Wouldn't it break if all containers try to start at once and install dependencies?
2. i think the best scenario would be running the container without any volumes, no problem with node_modules installation (in custom image), but is it possible to do this with plugins (preinstall plugins and possibly disable installation from administration) so the app starts much quicker and it's no problem to drop container and start another one?
3. is the strapi app ok with running same app in multiple instances/containers behind load balancer? For example after updating the model, the app restarts. Can't imagine how to do that with multiple instances.

Thanks for any answer/s.

[luisfavila]

Any news regarding this? Also interested.

[manhhailua]

Interested in this, I'm also aiming to deploy Strapi on K8S cluster and expect it to be able to run on multiple pods/containers. One more point is Strapi is causing downtime on configuration changes (code changes), this's awkward on production.

[Aurelsicoko]

@lucaperret @msgeissler @synth3tk any insights?

[msgeissler]

Hm, I don't have any experience with K8S and only minimal experience with running stuff in (docker) clusters. For the strapi-container you should basically follow the normal strapi-cluster setup first. Sadly I cannot find any documentation for that. So I do not know what kind of settings and setup strapi itself expects in a cluster environment. Without that, I cannot say whether it is OK to just point multiple strapi-installations to the same database and just place a load-balancer in front of it (maybe with sticky sessions).

As far as volumes go, I would also go with the idea to create a custom "final / production" image that already has everything installed. That way you can test your final image, you get faster start times and you can make sure (unless strapi does some fancy under the hood auto-updates) that all instances run the same version. Otherwise a new node / pod might suddenly download a newer plugin version and behave differently. That way there is also no need to mount an external volume so you do not run into any problems with the mount point not existing on a node / not being persistent (so it has to re-download everything) / or having to mount it on all nodes (and re-use the same for all pods, which I do not think is a good idea). I currently cannot remember if strapi requires a persistent storage for media uploads though, as I haven't used it in a while. That might require a cluster-ready persistent storage (whatever strapi supports, NFS, S3 etc. and support from strapi itself for concurrent access).

But I would need more info on strapi-clustering itself first to help more. If anyone has a link please post it, I was not able to find one quickly. :)

[techguydave]

So at first thought, this will almost certainly require some core changes. I believe the server restart for model/plugin changes will be the biggest pain point, as others have pointed out. If Container A updates a model, it's saved in the database for others to see, but Container B doesn't. Same with the plugins. I'm not familiar with how other CMSes handle this in a container.

I haven't touched alpha-14 yet, but I think that may have taken care of running it behind a loadbalancer via strapi/strapi#1813. I'm sure there are other challenges on top of these, too. After November 4th I'll be free to start working on my Strapi project again, so I'll make a note to research it a bit more in-depth then, if no one else has started.

[techguydave]

Looks like @msgeissler replied before I had a chance to post my comment (I thought GitHub autoloaded comments). Anyway, I wasn't familiar with a Strapi cluster mode, but that sounds like it's a step in the right direction.

I also agree that installing plugins and creating a "golden" image might be a good way to take care of the plugins side of the equation. We'd still need to address models, though. Hopefully that's what this cluster mode will allow.

As for file uploads, that's easily taken care of by using a storage plugin for network storage (cloud or local). However, and I'm new to k8s, I believe having a persistent volume claim could also solve this? k8s would take care of where it goes (S3, Azure, NFS, etc) and you'd just add the claim to the pods.

[msgeissler]

I really haven't worked with strapi too long. :-D I completely forgot about the models, I was thinking only of strapi and node plugins. Models might be a problem if they should be changeable at runtime, but I think that can only be solved by a proper cluster mode and possibly API in strapi itself first. Maybe they have to be stored in the database as well and created from there (they still have corresponding JavaScript code right?), otherwise you might run into problems with models being changed while containers are running and new ones being created after that (how do they know about the change?). Sharing the models folder between the containers / pods might not work either, as strapi has to know that it should reload them but also has to allow other instances to change them at any time.

I also don't know how other CMS systems handle that.

[jakubknejzlik]

@synth3tk I feel little bit confused. Could you clarify what you mean by "cluster mode". All what I was able to find about strapi and cluster is node cluster module support (not sure if this will do any good for this issue) and issue about MongoDB cluster :)

[techguydave]

@jakubknejzlik I was referencing @msgeissler's comment. I don't recall hearing about it, either, so I was curious if it was something introduced since I stopped working on my project.

[msgeissler]

I was just mentioning it because I think it would be a requirement to get proper multi-container support. If strapi does not support a cluster environment, neither can strapi in a container. But as I mentioned earlier, I also was not able to find any documentation on that. So for now, I assume that it does not exist yet. The problems mentioned above would most likely be stuff that such a cluster-mode would fix / have to deal with (including many other things probably).

[timoa]

Hi guys,

I'm also new with Strapi, and my first idea was to make my Strapi container immutable as much as I can by developing locally and commit the model changes that trigger a new build (container).

It works well at the moment if I build for the same environment, but I can't reuse the same container for all the environment because it needs to rebuild the plugins with the npm run setup (https://strapi.io/documentation/3.x.x/guides/deployment.html#_2-setup-optional) because I use a different path for the admin on staging and production.

For the database, I use mLab, but I will try MongoDB Atlas before my first release in production.

For the files/media, I will use the S3 plugin also to share most as I can between all the containers.

I don't like to have a container for each environment because that means more testing instead of using only the environment variables to change the behaviour of a unique container, but at the moment, I see only this way to release a Strapi app under a container :(

[therebelrobot]

Hey y'all, running into the same issue here. Trying to run this in a multi-instance setup (specifically on Zeit Now) with a single Mongo instance (mLab), with the s3 and graphql plugins installed.

I tried doing the "golden image" path, running my own dockerfile and shell script to install the additional pieces (basically waiting until strapi.sh is done running and then installing the plugins), which was working great, but now everytime I start a fresh instance, it blows away the field that has the content models in it (resetting it back to just Users in the UI). The rest of the db looks to be intact, but it kinda "forgets" that the rest exists... does that make sense?

Is there a way of setting up strapi / install plugins that doesn't reset the install's settings of content types?

[therebelrobot]

I think I found the root of the problem I'm describing above: I was unaware that Strapi creates physical files on the server when generating new content types, I previously thought it was all dynamically created based on what was present in the database. This explains why new installs don't show the previously created content types; the files just don't exist in a fresh install.

With this in mind, the docker image in this repo may not suit my needs, since it's start up script is installing a fresh version of the app each time, with a clean slate as per api files (regardless of what's in the db). I'm going to try and make a custom install of Strapi proper to make it work for my particular use case.

[Aurelsicoko]

@therebelrobot You're right! Strapi generates new files every time you create a new API. Everything is physical, like if it was you who developed the API. We think it's the cleanest way to do it instead of storing everything in the database. It avoids you to be dependant from the UI which is not good as a developer.

[jakubknejzlik]

@Aurelsicoko imho that's very good architecture (better than black-boxing the source codes in some magically loaded dependencies). But there's no easy/possible way how to create final container that I want to distribute...or is there anything like that? Currently I'm using the UI to generate my model, install plugins, configure the backend stuff etc. If I can do this locally, then wrap the whole generated code and use it to build docker image which will be distributed, we are half way there. But I'm not sure, if this flow is even possible.

[Aurelsicoko]

No idea, I don't know how this can be done with Docker. We've hired a new lead back-end developer, he'll certainly find a way to solve this issue 👍

[vinayagg]

@Aurelsicoko I am also interested in this, but I can help too. I have been a enterprise architect for years, I am an expert in docker and kubernetes and I have also done startups. I can have a call with your backend developer (if it helps) to come up with a solution that will work with docker and kubernetes. (message me if interested) My interest is only that I want to use it in docker/kubernetes.

[jakubknejzlik]

@vinayagg it would be awesome if I can join this call too (if it happens). We are using kubernetes in large portion of our project and this is the only issue that discourage us from using the strapi.

[alexandrebodin]

HI,

We just released a full rebuild of our docker image 🎉 We are closing the issues and open PRs to start fresh.

If you feel like something is missing or want to add things feel free to open new PRs ! 💯

[jakubknejzlik]

@alexandrebodin thanks for update 🎉 . As this comment has been added to multiple recently closed issues, would you mind adding link/explanation how issues mentioned in this conversation are resolved? 🙏

[stanleyyuenyiu]

@alexandrebodin may i know where PR? im facing the same issue too, strapi needed to deploy docker/kubernetes, there is no persistent storage at all, so when user create a new content type, it will generate code for the API, but if somehow we restart the docker, everything are gone, how do we solve?

[YosuCadilla]

As a Systems Administrator and DEVOPS, I have some knowledge of Docker and Kubernetes and would be happy to team with someone with Strapi experience to achieve the goal of multi-node, self-healing, auto-scaling Strapi on K8S.