search

Strider-CD / strider

Open Source Continuous Integration & Deployment Server
http://strider-cd.github.io/
4.59k stars 432 forks source link

Strider 2.0 Brainstorming #667

Open knownasilya opened 9 years ago

knownasilya commented 9 years ago

Just putting this out there, want to list out some of the "must have"s and "nice to have"s for 2.0.

Must Haves

Feel free to disagree, and contribute! Discussion welcome!

phiros commented 9 years ago

Even though I might repeat what other people already mentioned: I think strider should increase its modularity. What I mean by this specifically is:

Additionally:

knownasilya commented 9 years ago

I agree about everything, but not sure about ProtoBuf. I feel like that would hinder community involvement since JSON is well understood but not so much for ProtoBuf. Although I totally see the benefits of using that format.

knownasilya commented 9 years ago

The other concern is man-power. We don't have too many active contributors, but maybe a rewrite will solve that (developers love new things).

knownasilya commented 9 years ago

I would love to do a combination of json api and ProtoBuf. I think it will make it simple to have a nice convention and have the speed of ProtoBuf. In the end any kind of client could consume, since implementations are being written for many clients for JSON API.

If anyone is interesting in starting writing the API from scratch, let me know, and we can start a specific brain-storming issue (or a repo right off e.g. strider/api and discuss there).

kfatehi commented 9 years ago

I've really enjoyed reading this thread! Thanks all!

Ilya, I am very much for the idea of writing a new API from scratch.

As an aside, I currently use Buildkite for CI and I love that by design it does very little. There is an agent (open source and cross platform) that you must install on your workers, and then point them at the orchestrator (buildkite's closed-source cloud software). For example I have two types of workers... A digital ocean amd64 instance tests, builds, deploys my web code. An ARM beaglebone tests, builds, and uploads binary artifacts of my 'embedded' code.

As far as tools... I have been on a meteor stint for some time now. I have delivered several projects in meteor, and used it for a startup I'm currently working engaged in. However as my application grew more complex, the thing became slower and slower and the pub/sub stuff was getting in the way more often than helping. I scrapped it and rewrote...

I used hapi and sequelize with a postgres adapter. It has been so nice and such a breath of fresh air. I really enjoy hapi's simplicity and I missed SQL's predictability, maintainability, and enforcement of documented integrity of the underlying schema.

I have been using RabbitMQ (via CloudAMQP) and it's been just great for delivery work to downstream IoT devices. It's a good technology for delivering work. Redis and others are good too, but it was just interesting to me that I had been using Redis for pub sub so long (since I learned Rails) but that AMQP was what I was really after most of the time... I would keep that in my arsenal when designing a new CI...

As for frontend, my post-meteor development has been pretty much using AngularJS with proper usage of angular-ui ui-router and then calling back to Hapi with $http. Angular itself has been OK, but it is complicated and relatively hard to explain to people. I don't even fully understand ui-router and try to use it very minimally.

What I do agree with very much is that the core should not include UI -- if a new design can pull this off well I think it could be very special. I can imagine having a plugin that provides a base UI for those that want it. That plugin could easily be forked and have features added to it or use a totally different frontend framework or whatever.

That said, React seems quite interesting and I do believe (like we sorta have now in strider) in the idea of providing an easy way to generate plugin scaffolding. In it we can pre-bake some React boilerplate or Angular boilerplate, etc.

Also, would these plugins run on the orchestrator or on the worker? Strider currently does not have a super clear separation between orchestrator and worker and I think that was both a strength and a weakness. It was a strength because you got up and running quickly, but it was a weakness because it was difficult to figure out how to set up strider to actually deliver jobs down, to say, a beaglebone to produce ARM builds.

knownasilya commented 9 years ago

That said, React seems quite interesting and I do believe (like we sorta have now in strider) in the idea of providing an easy way to generate plugin scaffolding.

Sounds like a good plan. Write them as they come. I think that Yeoman is still a good option in this regard for writing something like that. It will allow for shared knowledge and a good platform for that exact use-case.

I used hapi and sequelize with a postgres adapter. It has been so nice and such a breath of fresh air. I really enjoy hapi's simplicity and I missed SQL's predictability, maintainability, and enforcement of documented integrity of the underlying schema.

I've yet to use Hapi. What does it bring to the table that Express doesn't? Same in regards to sequelize vs knexjs (which is what I've used in the past). I agree on postgres, and if we need json storage for plugin data, that can be done with the json column.

I know you like new things Keyvan :wink: Have you looked at rethinkdb (nosql)? It supports push notifications for an arbitrary query.

Also, would these plugins run on the orchestrator or on the worker? Strider currently does not have a super clear separation between orchestrator and worker and I think that was both a strength and a weakness. It was a strength because you got up and running quickly, but it was a weakness because it was difficult to figure out how to set up strider to actually deliver jobs down, to say, a beaglebone to produce ARM builds.

Plugins might have to be two faceted, a core portion and a drone portion. I can see some plugins implementing one or both, it would depend what the plugin needs to do I think. The drones can communicate back to core and report on what plugins they have, and those could be configured through core (or locally on the drone). Drone plugins could be simple node modules with a basic API, so that users can reuse the plethora of build/testing tools already out there.

I think we can start on a rough design on how this new system should work, and lay down the non-negotiables, and we can talk technology stack after.

kfatehi commented 9 years ago

I haven't looked at rethinkdb enough to comment on it.

The main thing that I like so far about hapi versus express has been the way you define routes. Routes are just objects that get passed into the server.route method. The method can take an object or an array of objects (routes), and the route object can contain keys that are used by other plugins or parts of hapi (e.g. using Joi to validate incoming data). This way you find yourself writing configuration objects most of the time, not so much functions that call other functions.

I also like the way 'auth strategies' work in Hapi -- it just makes sense to me. With express I vaguely remember doing some things with passportjs or something. On the other hand I might just be improving as a coder in general -- but I am pretty sure hapi has been a better toolset overall and it's not just me.

I don't know anything about knexjs, but I can say that sequelize has been well-documented and easy to use. I like how it has a tool for generating migration files and running them, I've been using it in production and doing migrations with it without any issues.

But anyway I agree with you about determining the non-negotiables, having an overall design we would like to achieve, and then looking at tools later. I'm pretty swamped with my current engagements but I will keep an eye on the new stuff :)

jpic commented 9 years ago

Multiple build environments and Simultaneous builds are a must-have for devops. If it's as easy to test that a whole infra on a superior version of as it is to test with another version of a dependency then we have a sane test system imho. Simultaneous builds is a must for devops and blue green deployment too because devop projects (ie. codebases using saltstack, ansible, chef ...) would typically take time as the test would generally be "instanciate a set of containers, provision them (which is mostly self-tested) and then run functional tests".

kfatehi commented 9 years ago

100% agree that we need true multiple build environments (this cannot be simply based on branches, it just doesn't work to tie the VCS into this).

I'd like more info about what blue green deployment is @jpic if you don't mind

jpic commented 9 years ago

Sure, is it good enough if I cite Fowler's article about BlueGreen ?

One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production. You usually need to do this quickly in order to minimize downtime. The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.

Blue-green deployment also gives you a rapid way to rollback - if anything goes wrong you switch the router back to your blue environment.

macropin commented 9 years ago

Guys, our 2c... Over at Panubo we are using Strider solely for deployments to CoreOS (Docker / Fleet) clusters. We chose Strider because it seems like a good lightweight, flexible tool that is focused on CD, rather than CI duties...

To get Strider to handle CD tasks we've needed to write a bunch of different tools and even a Docker container to host it all.

With all this we get an atomic (rolling) deployment capability. And our customers get visibility via a nice web UI.

But right now it's tricky to configure as everything must be configured manually. We are using the GitHub / BitBucket plugins (depending on customer), Custom Script runner, and Meta data plugins along with our tools above. Some of our customers have over 100 projects. So we envisage that we might need to write our own plugin to make this easier.

Given our usage, our wish list for Strider 2.0 would be:

Cheers, Andrew

mnaughto commented 9 years ago

After manually running deployment scripts for far too long, I started looking into CI applications and decided on Strider, largely because it runs on Node and seems a lot less complex to set up than Jenkins or whatever.

Since it's an easily-accessible portal to our private cloud, I've toyed with the idea of using it for more than just CI - like running database backups or running grunt tasks in arbitrary directories.

That being said, I like @bitwit's suggestion of supporting more plugin types. If things were set up correctly, that functionality could be accomplished via plugin.

I also think it makes a lot of sense to strip things down to just a REST API, and maybe have a separate repo for the "stock" UI, which, itself, could be a plugin orchestrator.

Given @macropin's comments about continual deployment, and my personal goal of using this for things other than CI, I think it would be good to get away from the idea of "builds" and think about generic, nestable "tasks". (Something like this is also mentioned in #233) So from what exists now, the "deploy" task would contain a "test" task and a final task to actually push files and such. The "test" task would contain a task to copy code, install code, and run the tests. Plugins could then publish their own tasks that could be "dragged" into other tasks to add them to the workflow. Other plugins could consume these tasks as well, paving the way for a generic task scheduler, etc. Is this idea too crazy?

knownasilya commented 9 years ago

Is this idea too crazy?

No, I think this is really the way we want to go.

Feel free to leave ideas in the core repo.

niallo commented 9 years ago

definitely. the data model is structured around jobs not builds per se. at least that was my goal :)

jobs can do more than just build and push code and can be triggered by different events - not just VCS events.

On Wednesday, August 19, 2015, Ilya Radchenko notifications@github.com wrote:

Is this idea too crazy?

No, I think this is really the way we want to go.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-132632777.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

Justinidlerz commented 5 years ago

Hello @knownasilya ,Is the 2.0 development plan still going on? I use strider inside the enterprise, it is very good, but it seems that it is very necessary to upgrade, because it frequently appears BUG after the frequency is increased. I originally planned to submit support for the corresponding Tag push event for it, but found that there were more changes. If this plan is still going on, I am willing to participate.