RFC: Cicada Modules

[brendanfalk]

Modules

This is an RFC for how Cicada should integrate with 3rd-party code. First we propose what we think is best, then we ask some questions. We’d love to hear your thoughts!

Proposal

A Cicada module is a custom application for the Cicada platform that performs a complex but frequently repeated jobs/steps/tasks. Think of them just CircleCI Orbs, GitHub Actions actions, Buildkite Plugins, but for Cicada. Our modules are modelled heavily on how Deno does modules.

A module can be an abstraction over a Job, an abstraction over a Step, or a utility function that can be used inside a step.

Module Registry

A module is just deno/typescript. So theoretically, the code can be hosted anywhere and just imported into your pipeline. However, rather than having hundreds of Cicada modules strewn throughout the internet, we want to have a registry that makes it easy to find all the different Cicada modules. This is similar to other popular CI/CD providers like the GitHub Actions marketplace or CircleCI orbs. We want our registry to be just like deno.land, but only for Cicada modules.

However, the registry isn't built yet. In the meantime, we have have created github.com/cicadahq/modules. We will have a folder for every module and each module will have a mod.ts file. e.g. git/mod.ts, npm/mod.ts, github/mod.ts. Once we build the registry, we will likely split each module up into different repos.

Module Usage and Examples

Below are a few examples of how you would use a module inside a Cicada pipeline. Our aim is to make it as simple and idiomatic as possible. We'd love to hear your thoughts!

Replacing Steps

Running tests with NPM

import { Job, Pipeline, Secret } from "https://deno.land/x/cicada/lib.ts";
import * as npm from "https://deno.land/x/cicada-modules/npm/mod.ts"

const test = new Job({
    image: "ubuntu:22.04",
    steps: [
        npm.installNode({version: "16.4"}), 
        npm.installDependencies(), 
        npm.lint(), 
        npm.test(), 
    ]
})

export new Pipeline([ test ])

Publish to NPM package registry

import { Job, Pipeline, Secret } from "https://deno.land/x/cicada/lib.ts";
import * as npm from "https://deno.land/x/cicada-modules/npm/mod.ts"

const publish = new Job({
    image: "ubuntu:22.04",
    steps: [
        npm.installNode({version: "16.4"}), 
        npm.installDependencies(), 
        npm.publish(token: new Secret.value("npm-token") ), 
    ]
})

export new Pipeline([ publish ])

Deploy using terraform

import { Job, Pipeline, Secret } from "https://deno.land/x/cicada/lib.ts";
import * as terraform from "https://deno.land/x/cicada-modules/terraform/mod.ts"

const deploy = new Job({
    image: "ubuntu:22.04",
    steps: [
        terraform.install({token: new Secret.value("terraform-token")), 
        terraform.init(),
        terraform.format(),
        terraform.plan(),
        terraform.apply(),
    ]
})

export new Pipeline([ deploy ])

Run security checks using Snyk

import { Job, Pipeline, Secret } from "https://deno.land/x/cicada/lib.ts";
import * as snyk from "https://deno.land/x/cicada-modules/snyk/mod.ts"

const security = new Job({
    image: "ubuntu:22.04",
    steps: [
        snky.install({token: new Secret.value("snyk-token")), 
        snyk.codeTest(),
        snyk.iacTest(),
        snyk.containerTest(),
        snyk.publishResultsToGitHub({token: Secret.value("github-token"))
    ]
})

export new Pipeline([ security ])

Replacing Jobs

Each of the above jobs, could be replaced using a job module like this

import { Job, Pipeline, Secret } from "https://deno.land/x/cicada/lib.ts";
import * as npm from "https://deno.land/x/cicada-modules/npm/mod.ts"
import * as terraform from "https://deno.land/x/cicada-modules/terraform/mod.ts"
import * as snyk from "https://deno.land/x/cicada-modules/snyk/mod.ts"

export new Pipeline([
        new npm.TestSuite({ nodeVersion: "16.4"}),
        new npm.Publish({ token: new Secret.value("npm-token") }),
        new terraform.Deploy({ token: new Secret.value("terraform-token") }),
        new snyk.SecurityChecks({ token: new Secret.value("snyk-token") }),
    ])

Questions

1. “Module”: Do you like the name “modules”. We’ve also been considering package, plugin, extension, wing. We want it to be as easy as possible to understand. 3rd parties. We find CircleCI “orbs” fear it could be a barrier to Cicada adoption
2. Where should we start: What modules would you like to see? Rather than just saying “git” or “npm”, please give some concrete examples of workflows that you use regularly in CI/CD. e.g. “I always check out the existing git repo” or “I bump the NPM package.json version”. We would like to focus on building modules for these first!
3. Module naming convention: A module can output Jobs, Steps, or general utility functions. How should we make it clear which exports are which? As you can see above, the only difference between a step and a job is the job uses title case. However, this doesn’t account for utility functions. Some other options are:
   • Prefix each with the type: npm.Job.Test, npm.step.test, npm.fn.test
   • Only prefix jobs like npm.Job.test. Donnt
   • Title case jobs, leave the rest as camel case
   • Don’t prefix anything or have any standard! Users should just rely on deno + typescript + good docs to work out which is which

Edits

• [April 19, 12:30pm] Updated Module Registry description
• [April 19, 12:38pm] Added required "image" field for jobs

[cstrnt]

Thoughts

• I really like the granularity here. Having the option to use an (almost) zero config way (pre-built jobs) but also to have single pre-built steps is really nice. I think small to medium sized projects with pretty default setups will be able to get up and running with the pre-built jobs super quickly, while bigger teams with customised setups are still able to make it work

• I would propose to add a Job suffix to pre-built jobs. On a technical side they will probably extend the Job class (or atleast they should IMO). Here is a little comparison:

import { Job, Pipeline, Secret } from "https://deno.land/x/cicada/lib.ts";
import * as npm from "https://deno.land/x/cicada-modules/npm/mod.ts"
import * as terraform from "https://deno.land/x/cicada-modules/terraform/mod.ts"
import * as snyk from "https://deno.land/x/cicada-modules/snyk/mod.ts"

export new Pipeline([
        new npm.TestSuite({ nodeVersion: "16.4"}),
        new npm.Publish({ token: Secret.value("npm-token") }),
        new terraform.Deploy({ token: Secret.value("terraform-token") }),
        new snyk.SecurityChecks({ token: Secret.value("snyk-token") }),
    ])

vs.

import { Job, Pipeline, Secret } from "https://deno.land/x/cicada/lib.ts";
import * as npm from "https://deno.land/x/cicada-modules/npm/mod.ts"
import * as terraform from "https://deno.land/x/cicada-modules/terraform/mod.ts"
import * as snyk from "https://deno.land/x/cicada-modules/snyk/mod.ts"

export new Pipeline([
        new npm.JestJob({ nodeVersion: "16.4"}),
        new npm.PublishJob({ token: Secret.value("npm-token") }),
        new terraform.DeployJob({ token: Secret.value("terraform-token") }),
        new snyk.SecurityCheckJob({ token: Secret.value("snyk-token") }),
    ])

I think this adds some clarity and shows the difference between individual steps vs. jobs without looking into the docs

• I think the Jobs should be really smart about what they do. For example the npm job should also work with yarn and pnpm. It should auto-detect those and take about this. I think this problem has been solved and Cicada should solve this for me too

Jobs I would need

In my daily work I use Github Actions and I like to use the CI for the following: Do a sanity check that it really not only works on my machine. This includes the following:

• Install my dependencies (npm, pnpm, yarn)
• Run my tests (jest, vitest, playwright, cypress)
• Lint my project and optionally annotate the code (eslint)

On my side project I also do the following

• Compare the bundle sizes (https://github.com/pkg-size/action)
• Tag & Publish to npm (changeset)

[brendanfalk]

@cstrnt Love it. Agree with all of this, especially with making sure we work with the user's tooling (e.g. the npm job should work with pnpm/npm/yarn)

@mschrage @grant0417 what do you think about the Job suffix Tim proposed?

And @cstrnt is the Job suffix idiomatic? ie can you give some examples of other SDKs that do something like this?

[cstrnt]

And @cstrnt is the Job suffix idiomatic? ie can you give some examples of other SDKs that do something like this?

I don't think so, but when you chose the Term Job for the base class I just think it will make thinks more more readable if they just extend the Job class and also specify this in the name. I'm a huge fan of verbose naming

[mschrage]

Adding the Job suffix makes sense to me. Also, conventionally, job names should be nouns not verbs (DeploymentJob rather than DeployJob)

However, not sure if an npm.install() step should attempt to use something like yarn or pnpm under the hood.

In the case of something like new jest.TestSuiteJob() this behavior makes sense, but if a step specifies a CLI tool, it should not use a different one under the hood. I think this would be too much magic.

If we want to have this dynamic behavior at the step level, I'd propose something like node.installPackages() which could attempt to determine the correct package manager automatically.

[RichiCoder1]

If we want to have this dynamic behavior at the step level, I'd propose something like node.installPackages() which could attempt to determine the correct package manager automatically.

I was about to propose this exactly.

[ghost]

It was a bit frustrating to discover that Cicada doesn't support pnpm yet. Or does it?

[cstrnt]

It was a bit frustrating to discover that Cicada doesn't support pnpm yet. Or does it?

It works, there just isn't a zero config way of doing it (yet).

You can just add a new step with the following:

{
    name: "Install pnpm",
    run: "npm install -g pnpm",
}

And you should be able to use pnpm :)

[ghost]

Yeah, I was talking about zero config type of thing 🤓