Obversity
commented
7 years ago Maybe choose 3-5 different ones and choose a time frame to setup and migrate and a time to evaluate.
I am completely in favour of this. I think the goal when doing the first one should be to write a server setup script that will do most of the hard work of setting up the server for us, and write a koi generator to generate capistrano config that allows us to deploy to this setup very easily. We'd need to:
- Pick a VPS service. I'd suggest Vultr, given their pricing and given that they have Sydney locations, meaning we'd get similar latency to our current EngineYard setup.
- Pick a linux distro. I'd suggest Ubuntu or CentOS or something else well-maintained that we'd be comfortable with and that has plenty of guides and documentation.
- Build a 'boot script' that would:
- create a deploy user
- install and configure all the things you need for a rails VPS and for Koi, like rbenv, ruby-build, ImageMagick, openssl, redis, nginx etc.
- set up a folder structure similar to what we're used to working with in EngineYard, for consistency
- secure the server, perhaps configuring fail2ban / iptables, and maybe changing SSH access from the default port
- (I got started writing this setup script the other day, and I've tested it line by line, but it's unfinished and I haven't tested it as an actual boot script)
- Build a generator for koi that generates capistrano/nginx config that works with this server setup, perhaps choosing what our default server should be (unicorn vs puma—puma seems to be the default in Rails 5 now, but I think our only site that uses puma [ysa-v2] has a memory leak, and we're not sure whether that's a koi/puma issue or an issue with code in the app)
- Pete's capistrano setup for SALC could probably be abstracted for this purpose!
The goal should be to make deploying to something like Vultr as painless as possible for a new project. And given that we control everything in the stack, we should be able to make it even less painless than EngineYard with a bit of upfront effort.
We should consider using LetsEncrypt with this setup. As I understand it, Peter has some experience setting this up, so that it's essentially a free SSL service for our customers. (We could charge them a fee for this, essentially offering a cheaper service than comodo / other SSL providers.)
pvawser01
jsidoryn
heathamos
Summary
It may be worth exploring alternatives to shared hosting with EngineYard for small clients and single instance hosting at EngineYard for medium clients.
This is also the continuation of the following RFC started by Peter Alternative hosting · Issue #308 · katalyst/koi · GitHub
Motivation
We have been with EY for 5+ years now and it is prudent to review whether this is the best setup. There have been problems with mismanagement in the past that has been cleaned up by hard work from Peter and Heath.
The foundational reason for going with EY is the belief that by going through a PaaS we spend less time on dev ops (which is not our expertise) and spend more time on building Rails projects (which is our expertise).
This only makes sense if going with a PaaS actually does lead to Katalyst making more money overall or significantly reducing risk. After discussing this with both Peter and Bill it seems that there is still significant time investment in working with EY and our dev ops may not be significantly reduced as well as other pros and cons.
First, some background to provide some context to the decision making process. This is primarily provided so we are coming from the same place in this discussion. Please ask or challenge me on any of this if it doesn’t make sense or you feel it is incorrect.
This first section discusses some things that I don’t believe should be considerations in making the decision to move away from shared hosting with EY. I’m not saying that these may not be good side effects, just that they shouldn’t be drivers in the decision making process.
Shared resources across sites
With multiple sites on a single instance, if one takes up more resources it affects the others. This doesn’t matter as it hasn’t been a limiting problem in the past 10? years of doing this and also we can get around it with better setting customers expectations around this price point. By problem, I don’t mean that it doesn’t happen, just that it hasn’t been enough of a problem from our customers perspective.
Absolute cost from hosting company
The cost of hosting is much wider than just the absolute cost that we are charged. There are many indirect costs that need to be taken into account to arrive at the true cost of a hosting service. The main ones are the risk of having our own setup and therefore alone in solving our problems, the extra expertise needed and the opportunity cost we give up by spending time here rather than elsewhere.
Decreasing cost to clients
Our strength isn’t with what we provide for small clients and providing a great hosting service for them isn’t a priority.
Too many sites on an instance
What we have now has too many on a single server but we could plan a much better execution. EY have indicated that this isn’t optimal but that other customers take this approach.
Below is a list of the areas that I believe should be seriously considered when reviewing the EY platform
Locked to a single version of Ruby
For each instance, we only get to choose a single version of Ruby. This has the side effect of creating consistency but it seems to be a big hindrance. If a single client on a shared server wanted to pay to upgrade to a later version of Koi, then we couldn’t do this without updating them all or creating a new instance and moving it to that.
Quirks and ongoing issues
There seems to be a number of issues like deploying a new app, IP’s not re-attaching themselves when rebooting an app and ongoing Redis issues that don’t seem to be adequately resolved. We have work around's but they were likely time consuming to work out in the first place and should be better.
Opaque system
This one seems to be the crux of much of the frustration. On one hand, any PaaS has its own ways of doing things and part of our job is to learn it. This can be frustrating as we may already understand the way to do it if we had complete control and it’s redundant learning/knowledge as it’s only useful in this particular context.
That being said, the value in this is if it works well. From what I’ve seen their support has mostly been excellent however we are the first to investigate a problem anyway and our ability to troubleshoot when problems do inevitably arise is difficult as there’s a level that we just can’t know anything about.
This is both frustrating and requires an over reliance on a 3rd party that may limit our ability to address an issue in a timely manner.
Better hosting offering for single instance clients
While a great offering for shared clients which are generally lower value is not important, being able to offer a better hosting offering for single instance clients is of benefit. Clients such as Oliveri, Lifecare, Medehealth, Southern Cross Care, Minda, Guide Dogs, Beaus, Combat Sports are good clients for us and we can offer them a much better hosting service if we had an EngineYard alternative such as Digital Ocean or Vultr for lower cost hosting.
Detailed design
Based on the above I think there’s more than enough reasons to investigate alternative hosting options. I think there’s limited value in getting more data or further discussion and it’s best to test it out on a few sites and evaluate the benefits. Peter has also already gone through this process building SALC on a Digital Ocean box and the experience seemed to be pretty positive.
So I think we should start to pull sites off the shared environment. Maybe choose 3-5 different ones and choose a time frame to setup and migrate and a time to evaluate.
I would expect that a separate RFC should be created to better define a more detailed plan to move forward.
Drawbacks
Here is a couple of the main drawbacks of going down this path.
Every server may have a different setup
One unintended side effect of having a shared server is that it keeps many things reasonably consistent. With each server being separate, it’s much easier to create custom configurations. As we have more staff or staff change over time these problems can magnify.
This could be solved primarily through a process. I’m sure you already have ideas about how we can solve this and in addition to these, one may be to peg things against versions of Koi.
Increased dev ops knowledge
We’ll be on our own a lot more and will need to work things out ourselves. This includes setting up the hosting environment, managing deployment and maintaining. The bet here is that we likely know a reasonable amount of this already and what we don’t know we can figure out. There’s no one to help us if and when things go wrong, but on the other hand, we’ll have full knowledge of the system and will have more power to fix things ourselves.
There’s nothing really to say here except that it’s a thing, but we acknowledge that the benefits may be greater than the costs. We’ll just need to keep in mind that we document things better and so when new people start they can quickly be productive in this space.
Time spent researching, testing, integrating and documenting the new setup
This is largely unknown and the responsibility that was on EY is now on us and this will likely take a reasonable amount of time to successfully implement.
Multiple hosting options
As we will likely still have larger sites that require multiple servers hosted with EY (such as the Fringe) we’ll still need knowledge and understanding of the PaaS, however, we’ll also have a second system to manage and understand.
Alternatives
Use different PaaS
A different PaaS doesn’t really solve the problems although the experience may be better. It’s worth noting that ThoughtBot host everything on Heroku and so push off all of their dev ops. Heroku are expensive and it is likely it will be more expensive than EY.
External DevOps team
Another approach is to outsource the DevOps to an external team such as ReInteractive (reinteractive OpsCare | 24/7 Ruby on Rails Operations). This is a strong approach, however, most are set up for managing a single large site rather than many smaller ones. This makes ReInteractive very expensive and I started discussions with them about looking at it as a group of sites and it’s open for further discussion with them if we feel it’s needed.
We also previously hosted at Anchor who offers some DevOps on the stack and their service was also strong.
Use Amazon stack directly
Rather than going through EY we could setup a single instance directly using the Amazon stack directly. There’s also Google Cloud as an alternative to Amazon that I’ve heard decent things about.
Stay with EY and do better
This may include a better strategy for managing shared servers (fewer clients, clear upgrade tasks) and to renegotiate a better deal with them.
Hire to setup and then we run
We could hire a consultant to be responsible for managing this transition so we can keep the focus on our clients and Koi. Once set up we could be responsible for managing.
Unresolved questions
The main unresolved questions are the amount of time, expertise and risk that is associated with going for a single instance hosting. I believe the best way to answer this question is to just do it.