Open kfatehi opened 9 years ago
kfatehi
commented
9 years ago I wonder if it's a good idea to just combine efforts into "meteor up" and then maintain a very simple ansible role that leverages it...
I don't think it is a good idea btw, I don't like that meteor up does a whole bunch of things without the sysadmin really knowing about -- unlike in Ansible where we can clearly see all the steps involved as ansible primitives in Yaml format
settinghead
commented
9 years ago @keyvanfatehi Yes, I am aware of Meter Up. :)
One of the reasons I'm extending this Meteor+Ansible effort is precisely what you've described: scalability. My plan is to put more components into docker. I believe this has so many benefits that it deserves a more detailed write-up to explain the pro's and con's.
Another among the reasons is I like Ansible's declarative approach: you declare the machine to be in a certain state (i.e. presence/absence of a certain package, etc), as opposed to imperative approach where I find it more difficult to solve issues on servers.
kfatehi
commented
9 years ago I completely agree with you. My issue is that your readme basically tells me to sit on my hands. I want to know your full vision on this and where you are with regards to implementing it -- your roadmap, etc. I would like to help develop it because I badly need it (well, I don't /badly/ need it yet, but very soon several separate meteor projects my friends and I maintain are about to outgrow digital ocean...)
I need this on multiple projects, so I want to do whatever I can to help make this exist (assuming maybe there aren't others that have potential, and maybe they should be helped if the vision is similar or better, I didn't google that deeply).
I happen to be pretty experienced w/ docker... here's some stuff i've done with it
I'm also really interested in metrics and the current solution (made by the same guy that made meteor up) is closed source and super expensive... we need to make a proper open source thing -- I dont think NewRelic and the other usual suspects are going to step up to it anytime soon. I think people are still just waiting to see if meteor is legit (me included, but I've been nothing but impressed with meteor since day 1)
settinghead
commented
9 years ago @keyvanfatehi It's good to know you have a lot experience in Docker. :) I'd be happy to collaborate with you if what I'm proposing suits your needs and vice versa. Here are a few items on on TO-DO list which I will soon need for my own projects:
Here are a few things that I think is worth implementing (which I might need in the near future):
Even more ambitious:
If you are interested, I'd suggest you list some of the feature that you think you will most urgently need (along with a rough timeline), and we can see if we can combine our efforts from there. Also feel free to critique some of my goals listed above.
settinghead
commented
9 years ago @keyvanfatehi
For metrics, I agree with you that there needs to be an open source effort, and it's a nice-to-have item for me. However TBH I don't have much thought on it for now but I'd be interested in participating.
kfatehi
commented
9 years ago @settinghead thanks for the detailed response! This became really interesting for me, so thanks for that :) Here are my thoughts:
First off we should get clear what I think this project is so you can correct me. It would be for an ansible role that knows how to build (presumably by calling meteor build under the hood or something, whatever "meteor up" is doing) and transmit the artifact to the host (or some other way, like ensuring meteor is installed on the host, then using git to update the app source on the host, then running meteor build there), configuring a reverse proxy. Users then use this in their playbooks and can add to the role if they want, on their side.
MongoDB replica sets & oplog tailing support (This would speed up reactivity significantly and is also good for scalability).
I am not so experienced with MongoDB scaling, and less so how Meteor deals with it. Ideally we have sane defaults and good docs so that people like me can get educated on the decisions (e.g. sharding vs replica-sets vs hybrid?) made by the ansible role.
Dockerize Node.js app
When you say Node.js app do you mean the node.js server that meteor builds for us to deploy? I assume that's what you're talking about. Ok honestly, I feel like Docker is a bit overrated and overused. I'd like to have a deeper conversation about why use Docker, and why not, especially when Ansible has built so many nice primitives for us for dependency management. I'd rather contribute hiqh quality ansible content than shoddy Dockerfiles that go stale and unmaintained. In my experience it is far easier to change a playbook in production than to change a docker container. And no conversation about docker in production is complete without touching on stuff like CoreOS, fleetd, and others... I wonder if going this way might prematurely add unnecessary complexity and get away from the fact that we're already in the ansible environment, where all our needs are already met better, probably better than a Docker environment.
Perhaps another refactoring of the current repo to make it more user-friendly (the goal here is to type as few key strokes as possible to enable the initial deployment)
That would be nice! I have not analyzed the repo or the rest of the README because it seemed like you were going to radically change it. Is that not the case? Does it work? Meteor Up works, so I should probably analyze that one. I feel like just porting Meteor Up to ansible primitives is what I'm looking for.
Here are a few things that I think is worth implementing (which I might need in the near future):
Offer the option to install plain MongoDB (instead of Docker), or the option to integrate with a 3rd-party DaaS (e.g. MongoLab). I heard Dockerized MongoDB has log overgrowth issues (at least with Docker version below 1.4). This needs further investigation.
I agree that MongoDB needs to work properly, easily, and scalably if desired. However I don't think that Docker is necessarily the right way to achieve this or that we even need to concern ourselves with MongoDB. It feels out of scope. Instead we could simply recommend a good meteor ansible role that handles single-instance or multi-instance mongodb deployment and configuration.
A nicer way of inserting and updating deployment scripts into existing Meteor project (Currently one would need to use either git subtree or git submodule. I feel it is too complicated and daunting, even for some advanced users. Ideally I'd like a CLI tool that auto git-pull the scripts into your project).
Agreed -- I'd rather avoid git submodule voodoo too when simpler things are available (rm -r whatever && git clone whatever). But I really am not familiar with what submodule (deployment scripts? why are you cloning these? Doesn't ansible handle all this file transfer hassle for us? I must review your code I guess, maybe a line number to start from would be helpful).
Redis integration (this shouldn't be too complicated)
Yeah I mean as a user creating your playbook you should be able to add a redis role to your playbook and then set the env var accordingly, which then your meteor app code uses and connects to. It should be absolutely no extra work on our part in developing the ansible meteor role. Right? Do you anticipate otherwise? Why?
Support for RabbitMQ Same as above. Actually "Kudu" the CICD tool I've been working on has a meteor FE and relies on AMQP, so I'd actually use this. I see Mongo, Redis, and Rabbit all as being no extra work on our part at this time due our ability to leverage the roles already built for ansible by others.
Basically, put as much into Docker as possible so that the server will not need to install any extra libraries
Although Docker seems to make installation easier, I feel like it makes long-term maintenance much more complicated and risky compared to using ansible's tools for config management. Here's how i think of it: the Dockerfile is not a very sophisticated configuration management format & rich ecosystem compared to Ansible's role format & ecosystem. Docker images tend to be low quality and their maintenance over the long term tends to be more difficult due to the added layer.
Perhaps a better strategy for others to contribute and extend its capabilities. This may either involve writing better documentation, or a little more restructuring to make the codebase more modular and extendable. Yeah I love this idea. I feel like ansible roles get us there. My main attraction to ansible compared to chef and salt was that it seemed to get the modularity "right" in a config management tool. It was easy as hell to get started with a simple recipe or whatever it's called, but then graduate to a playbook. I still miss salt's speed though -- but I remember ansible was working on something competitive to the 0mq solution salt uses (as opposed to ssh).
Even more ambitious:
A GUI dashboard that displays deployment status?
This would be an external tool and related to metrics, in my opinion. I am interested in this, but it's not the same as the meteor web role I think we're trying to envision. I feel like it's part of continuous integration / continuous deployment / metrics... in that realm. Definitely into it, once we've got more meteor stuff in production.
If you are interested, I'd suggest you list some of the feature that you think you will most urgently need (along with a rough timeline), and we can see if we can combine our efforts from there. Also feel free to critique some of my goals listed above.
Ability to deploy a meteor app to a single instance, for now! We need a solid base to start from. I will spend some free time (however limited right now) looking at what other solutions are out there, how meteor up works, and how some very popular ansible roles (e.g. django) are structured and made interoperable with other tech like Mongo, Redis, etc.
For metrics, I agree with you that there needs to be an open source effort, and it's a nice-to-have item for me. However TBH I don't have much thought on it for now but I'd be interested in participating.
It'll come, after we gain some traction on this initial stuff and start using it to maintain our production meteor stuff. What I envision as a first step would be to have logging, we can probably make recommendations for ansible roles that provide a nice ELK stack (elasticsearch, logstash, kibana), and make a little atmosphere package that takes variables we document in our meteor role that hooks it all up.
After that, we could look at Sensu (ruby) or Shinken (python) monitoring stacks. There's also statsd... really a lot of places we can go with monitoring, but I would love to be able to provide recommendations, docs, and easy configuration to get all these things with as minimal initial effort as possible.
First and foremost, though, I feel that we need a pure ansible approach to deploying a meteor app as easily as possible, like you said, and then figure out how we're going to support other tech (mongo, redis, rabbit) in the smartest ways possible, then lastly thinking about maintenance mode stuff like logging, metrics, etc, if it makes sense. Also you can probably tell from this lengthy response that I am wary of falling victim to Docker hype due to lack of critical thinking regarding if Docker is right here or not (it may be, I am experienced with the remote API, not much else... i.e. i have never run a coreos fleet, etc) and end up having a trash codebase as a result of a passing fad or bad decisions -- I think that the conservative/safe choice is to keep everything in Ansible/YAML/Python and good documentation.
Thoughts?
kfatehi
commented
9 years ago By the way, if you do a role search for "meteor" on the ansible galaxy, you get one result: https://galaxy.ansible.com/list#/roles/3354 which is this repo https://github.com/carlosacp/ansible-meteor
this is nice as it installs meteor and nothing else (from what i can tell). We could depend on this role or fork it and make a more in-depth solution. Curious what you think.
settinghead
commented
9 years ago It would be for an ansible role that knows how to build (presumably by calling meteor build under the hood or something, whatever "meteor up" is doing) and transmit the artifact to the host (or some other way, like ensuring meteor is installed on the host, then using git to update the app source on the host, then running meteor build there), configuring a reverse proxy. Users then use this in their playbooks and can add to the role if they want, on their side.
Yes, I agree with the definition of the project. In fact, I already have a working version of most of what you described here. I have been using this repo to do all my new deployments to Digital Ocean, with continuous integration support (I use Circle CI mostly; an example can be found here: https://github.com/settinghead/easy-meteor-deploy/blob/master/circle.example.yml).
Just to be clear, let's break it down on the implementations.
how to build (presumably by calling meteor build under the hood or something, whatever "meteor up" is doing)
You can already use ansible-playbook -i inventories/[inventory_name] playbook-build-app.yml --connection=local to build the app and copy the tarball to remote machine via SCP. See https://github.com/settinghead/easy-meteor-deploy/blob/master/roles/build_app/tasks/main.yml
The caveat here is you need a linux machine that runs the builds (Mac OS X is also OK as long as you don't have npm library dependencies that has platform-specitic binary executables).
and transmit the artifact to the host
This is currently done via SCP. See the file above.
or some other way, like ensuring meteor is installed on the host, then using git to update the app source on the host, then running meteor build there
This is supported by the original repo that I forked from, but I haven't used it for a while and probably needs an update. You can find the tasks here: https://github.com/settinghead/easy-meteor-deploy/blob/master/roles/deploy_app/tasks/main.yml#L24
Ideally we have sane defaults and good docs so that people like me can get educated on the decisions (e.g. sharding vs replica-sets vs hybrid?) made by the ansible role.
Agree. I actually found pretty good Ansible MongoDB scripts here which we might be able to use: https://github.com/ansible/ansible-examples/tree/master/mongodb
When you say Node.js app do you mean the node.js server that meteor builds for us to deploy? I assume that's what you're talking about. Ok honestly, I feel like Docker is a bit overrated and overused.
OK I probably overstated it a bit by saying this is on my immediate implementation plan. The biggest advantage I see for Node.js+Docker isolation is to avoid NPM cache lock conflicts and Node version conflicts, which in my experience are both possible, but rare. We can come back to this when we have an actual problem which can be solved by Node.js + Docker.
I'd like to have a deeper conversation about why use Docker, and why not, especially when Ansible has built so many nice primitives for us for dependency management. I'd rather contribute hiqh quality ansible content than shoddy Dockerfiles that go stale and unmaintained. In my experience it is far easier to change a playbook in production than to change a docker container. And no conversation about docker in production is complete without touching on stuff like CoreOS, fleetd, and others... I wonder if going this way might prematurely add unnecessary complexity and get away from the fact that we're already in the ansible environment, where all our needs are already met better, probably better than a Docker environment.
Agree. Docker is not the best solution to all things. Ansible is more precise, declarative and replicable (BTW I'm also a fan of NixOS in this regard). I haven't played with CoreOS & fleetd, but would like to try them and test things out.
I have not analyzed the repo or the rest of the README because it seemed like you were going to radically change it. Is that not the case? Does it work?
Yeah It actually works, and what the steps described in README is currently accurate.
I agree that MongoDB needs to work properly, easily, and scalably if desired. However I don't think that Docker is necessarily the right way to achieve this or that we even need to concern ourselves with MongoDB. It feels out of scope. Instead we could simply recommend a good meteor ansible role that handles single-instance or multi-instance mongodb deployment and configuration.
True. I Dockerized MongoDB because of a specialized Meteor project I worked on that requires multiple databases that are isolated from each other. This however is not a normal requirement.
I'd rather avoid git submodule voodoo too when simpler things are available (rm -r whatever && git clone whatever). But I really am not familiar with what submodule (deployment scripts? why are you cloning these? Doesn't ansible handle all this file transfer hassle for us? I must review your code I guess, maybe a line number to start from would be helpful).
I ended up with git subtree because I can't think of a better way to update easy-meteor-deploy across multiple Meteor projects. The goal here is to constantly refine this repository and easily propagate its changes to all existing Meteor projects. (You can find git subtree instructions in the README).
The current set up is you make this repo a subree/submodule of our Meteor project (e.g. as /.deploy folder), so that you can do git subtree pull or git submodule sync whenever this repository updates.
Another simpler strategy is just copy all files in the repo to your Meteor project, but you'd need to do manual updates every time easy-meteor-deploy updates. (in comparison git subtree has the same benefit plus the ability to sync with this repo more easily). IMO git subtree isn't that bad once you get familiar with it, but it's still not friendly enough for impatient first-time users (most of whom probably are).
That said, I'm also looking for a better solution. If you can think of something please do let me know.
Feel free to add me on IM if you feel like discussion the philosophy and goals further. I'll email you my contact shortly.
Yeah I mean as a user creating your playbook you should be able to add a redis role to your playbook and then set the env var accordingly, which then your meteor app code uses and connects to. It should be absolutely no extra work on our part in developing the ansible meteor role. Right?
Exactly.
Although Docker seems to make installation easier, I feel like it makes long-term maintenance much more complicated and risky compared to using ansible's tools for config management. Here's how i think of it: the Dockerfile is not a very sophisticated configuration management format & rich ecosystem compared to Ansible's role format & ecosystem. Docker images tend to be low quality and their maintenance over the long term tends to be more difficult due to the added layer.
OK point taken. Over-engineering is a sin. :)
I still miss salt's speed though -- but I remember ansible was working on something competitive to the 0mq solution salt uses (as opposed to ssh).
Haven't used salt, but I was able to get reasonable speed out of this repo (usually takes less than 10 minutes for first-time deployment into a clean Ubuntu Vagrant instane, and much less time for subsequent deployment). I've also enabled pipelining to speed things up. Not sure how much still I can squeeze out of it.
What I envision as a first step would be to have logging, we can probably make recommendations for ansible roles that provide a nice ELK stack (elasticsearch, logstash, kibana), and make a little atmosphere package that takes variables we document in our meteor role that hooks it all up.
Good idea. I'd love to talk more about a plan for this.
After that, we could look at Sensu (ruby) or Shinken (python) monitoring stacks. There's also statsd... really a lot of places we can go with monitoring, but I would love to be able to provide recommendations, docs, and easy configuration to get all these things with as minimal initial effort as possible.
Not entirely familiar with the monitoring tools (only used monit and New Relic before). But again would like to chat more about it.
I think that the conservative/safe choice is to keep everything in Ansible/YAML/Python and good documentation.
OK. Your response convinced me to use Ansible as much as possible, and use Docker only when necessary. :)
By the way, if you do a role search for "meteor" on the ansible galaxy, you get one result: https://galaxy.ansible.com/list#/roles/3354 which is this repo https://github.com/carlosacp/ansible-meteor
Nice. This is really good to know. In fact, I've been looking for an ansible meteor module for a while and even thought about writing it myself. Looks like the repo is fairly new. I'll try to incorporate soon.
Also feel free to add me on IM if you want to chat about the plans further. I'll email you my handles shortly.
settinghead
commented
9 years ago @keyvanfatehi re leveraging Meteor Up:
Since this repo (easy-meteor-deploy) is in a usable state (i.e. you can deploy to a complete empty box with the current version), I think the fastest way is probably start with this repo and borrow ideas from Meteor Up & elsewhere. Just a suggestion.
kfatehi
commented
9 years ago @settinghead awesome. thanks for the deep replies. I'll take some time to study those links, and get acquainted with this repo in its current state / continue the research. I only had one comment, for this part of the conversation:
The current set up is you make this repo a subree/submodule of our Meteor project (e.g. as /.deploy folder), so that you can do git subtree pull or git submodule sync whenever this repository updates.
Another simpler strategy is just copy all files in the repo to your Meteor project, but you'd need to do manual updates every time easy-meteor-deploy updates. (in comparison git subtree has the same benefit plus the ability to sync with this repo more easily). IMO git subtree isn't that bad once you get familiar with it, but it's still not friendly enough for impatient first-time users (most of whom probably are).
That said, I'm also looking for a better solution. If you can think of something please do let me know.
The solution that came to mind is making it a meteor package and distributing it via atmosphere. Do you see any cons to that? I see these pros:
Thanks for the personal contact info, will add soon -- very interested to continue to discussion so long as I can keep myself educated enough and try it out enough to be adding value and not detracting or distracting :)
settinghead
commented
9 years ago @keyvanfatehi have to say I really like this idea and had a how-come-i-never-thought-about-that moment. It is a little ambitious, but a little googling suggests there are some existing projects that can facilitate the development of a Meteor GUI package + Ansible solution:
I'll let you know if I find out more. Thanks for the suggestion!
kfatehi
commented
9 years ago I thought I should mention that the meteor up developer is also thinking to use docker, check this https://github.com/arunoda/meteor-up/issues/390#issuecomment-97680067
Just curious if you know about this: https://github.com/arunoda/meteor-up
I also prefer Ansible as my tool for configuration management / continuous infrastructure and very much support a meteor role.
The "Meteor Up" project works well for small stuff, but I'm looking at your repository because I am thinking about scale...
I wonder if it's a good idea to just combine efforts into "meteor up" and then maintain a very simple ansible role that leverages it...