Details

[raniejade]

Bob is a minimal and modular CI/CD engine running entirely as an API.

This has piqued my curiosity, can you share some more details? :)

[lispyclouds]

Sure, Here goes:

I am primarily a server side dev at ThoughtWorks and deal with a lot for DevOps tools almost everyday and specially a CI server. Some of the pain points I see in systems like Jenkins and GoCD are:

• The plugin architecture. They are mainly to be extended via JARs which are loaded by the main process. This presents the following issues:
  • Bugs and vulnerabilities in the plugin. Most if not all of the plugins we use for these systems are the primary root cause for much of the CVEs. Also a badly written plugin can bring the whole thing down.
  • We absolutely have to use a JVM language to write the plugins.
  • We are limited by the API provided by the main system to do stuff like get resources and orchestrate the artifacts around.
• Monolithic, large and complex. IMO these systems have grown to a stage where they have a complex UX and requires a good amount of learning curve to use properly.
• The UI is merged with the back-end source hence pretty much providing an opinionated view of the CI infra and to control it. Building mobile interfaces or accessing on them can be tricky.
• The API is patchy. They don't really expose the entire infra via API and extending/controlling them via just their API is hard.
• Jenkins doesn't even have a proper notion of Pipeline and the flow of artifacts.
• Some good ones aren't FOSS.
• Infra is hard to version control

Hence bob is what I was thinking. This is VERY new project and haven't really thought it through. Here are the features what I think it should have:

• Use GPL and be FOSS
• Have a very small core with a limited features set. And be opinionated ONLY about them. (Inspired by GoCD)
  • Tasks(direct commands like shell)
  • Jobs(Collection of Tasks)
  • Stage(collection of jobs producing artifacts)
  • Pipeline(Collection of Stages which can consume artifacts from other pipelines)
  • Environment(Key value store associated with either Jobs and/or Stages and/or Pipelines)
  • Tags (Simple names to tag Task to make them run on specific places)
  • Resources (Things like source code or assets)
  • Workers (remote or local machines to carry out Jobs)
  • Still thinking of others... and open to suggestions
• Expose the above feature set entirely via an API, hence allow any client to configure/control it. Also not be affected by bugs in it.
• Be agnostic of an UI
• Be more extensible than great things like Concourse.ci
• Accept the above stuff via an YAML as well
• Build pretty much everything else via plugins:
  • Have a set of core plugins (like Ansible)
  • Things like getting stuff from from a VCS can be a plugin which can abstract away the type like Github or Bitbucket and do stuff like auth and finally give bob the direct resource URL.
  • Stuff like how to build an artifact like debs or RPMs can be plugins as well whose commands can be fed in as a task.
  • Plugins to automate the provisioning of a cloud instance or bare metal system to be able to execute tasks via SSH (Ansible way) and also clean up and when done.
  • Infra plugins like Terraform based stuff which can provision entire cloud infras to support bob and his agents.
• Ability to have multiple servers and share loads
• And again open to suggestions for anything else.

And this is a project born out my frustration and is VERY new. 😄 Any and every help, suggestion is most welcome!

[lispyclouds]

And probably the most important question, why Kotlin:

• BIG fan of static typed functional programming here and I just love the syntax
• Really like the simple, limited yet powerful feature set.
• Great Error handling and safety. Good for a paranoid(Rust and Haskell is what I swear by) dev like for me.
• Kotlin's DSLish syntax is great for a thing like CI.
• Kotlin's coroutines is perfect for concurrently spawning process and monitor them and the producer-consumer coroutines are perfect for implementing pipelines.
• Was originally in Clojure but the dynamic typing spooks me. Might have come Clojure components later.
• AND I HATE JAVA but like the JVM 🤣

[raniejade]

Wow, thanks for a very detailed explanation. If I understand correctly, plugins will interface against the API? So it doesn't really need to be deployed with the CI server. Essentially plugins are high level abstractions built on top of the core abstractions (tasks, stages, pipelines etc...).

[lispyclouds]

Yes, exactly. In any language and more importantly running on their own processes.

[raniejade]

That's cool. When you say running on their own processes those processes are still managed by the CI right?

[lispyclouds]

Both yes and no. Things like fetching resources and feeding it to bob might be managed by bob, but something like deployment plugin which pushes artifacts to S3 for example may run on its own and interact with bob at need. The intention is that plugins can be in different machines altogether and have bob as a synchronization point of sorts.

[lispyclouds]

Also I am envisioning scenarios where there can be multiple CI servers and can distribute and synchronize the work between them. I mean why should the server be this huge monolithic thing and have lots of agents. Have a set of Bobs and they can be a nice build cluster. Hence I'm making bob small and self contained in a single fat JAR.

[raniejade]

I'm a bit confused, say I have this deployment process for a website.

1. Run the unit tests
2. Build (if previous step pass)
3. Run automation tests using artifact from previous step (deploy to staging environment)
4. Deploy to production (if previous step pass).

How would the interaction w/ bob look like?

[raniejade]

Wait, can I build my own CI on top of bob?

[lispyclouds]

Yes. This is more like a CI platform or SDK if you will.

Say you have hosted your source in Github.

• You can define the flow in a YAML something like this (haven't decided this. some random thoughts):

  
  from_github:
  url: <some url>
  on_success: "build_it"

pipelines: [ name: "build_it" stages: [ "compile", "test", ... ] name: "deploy" ]

stage: name: "compile" jobs: [ "minify", "webpack", ... ]

job: name: "minify" tasks: [ "gulp..." artifacts: [ "./build/site_min.zip" ]

deploy_to_s3: creds: api_key: "something" artifacts: [ from: pipeline: "build_it" artifact: "..." ]

and so on. All the top level tags are plugins or some core components
Warning: this is off the top of my head. Nothing is decided yet :)

[raniejade]

The intention is that plugins can be in different machines altogether and have bob as a synchronization point of sorts.

I was confused on this part. So plugins can have its own process?

[lispyclouds]

Yes, that's one of the primary intentions. To be language agnostic and not suffer from the issues the plugin may have. Another example would be the UI plugin. That would be its own server talking to bob on its API and displaying the UI to its clients.

[lispyclouds]

So basically build your own CI setup with whatever plugins and tools using (the mostly unopinionated)bob at core to synchronize it all.

[raniejade]

It's very interesting, will closely watch this repo :+1: .

[lispyclouds]

Awesome @raniejade Any ideas, contributions and code reviews are most welcome 😄