Request: accept async function as a job step or a workflow job

[chabou]

Is there an existing issue that is already proposing this?

• [X] I have searched the existing issues

Is your feature request related to a problem? Please describe it

Given a complex configuration with some custom generated jobs with specific step like this one:

import * as CircleCI from "@circleci/circleci-config-sdk";

// Components
const config = new CircleCI.Config();
const dockerExec = new CircleCI.executors.DockerExecutor("cimg/base:stable");
const workflow = new CircleCI.Workflow("main");

// Define "reusable" job, with native JS
type customJobParameters = {
  name: string;
  cluster: string;
  container_name: string;
  task_role: string;
};
const createCustomJob = (parameters: customJobParameters) => {
  const stepsToRun = [
    new CircleCI.commands.Checkout(),
    new CircleCI.commands.Run({ command: "echo 'Hello World'" }),
  ];

  // Add deploy step if tag
  // https://circleci.com/docs/variables/
  if (process.env.CIRCLE_TAG) {
    stepsToRun.push(new CircleCI.commands.Run({ command: "echo 'Deploying'" }));
  }
  return new CircleCI.Job(parameters.name, dockerExec, stepsToRun);
};

const jobDefinitions: customJobParameters[] = [
  {
    name: "job1",
    cluster: "cluster",
    container_name: "container_name",
    task_role: "task_role",
  },
  {
    name: "job2",
    cluster: "cluster",
    container_name: "container_name",
    task_role: "task_role",
  },
];

// Add jobs to workflow

jobDefinitions.forEach((jobDefinition) => {
  workflow.addJob(createCustomJob(jobDefinition));
});

config.addWorkflow(workflow);

// Print config
console.log(config.stringify());

I want to add a Step that can only be generated asynchronously. For example I want to send a slack notification that contains the git author. Something like:

const slackCommand = async () => {
  const author = await getAuthor() // async function with some git calls
  return new CircleCI.reusable.ReusedCommand(orbSlack.commands['notify'], {
    name: 'notify-author',
    event: 'fail',
    custom: JSON.stringify({
      text: `Commit by ${author}`,
    }),
  })
} 

For now, I have to make all the function call chain, asynchronous:

import * as CircleCI from '@circleci/circleci-config-sdk'

// Components
const config = new CircleCI.Config()
const dockerExec = new CircleCI.executors.DockerExecutor('cimg/base:stable')
const workflow = new CircleCI.Workflow('main')

// Asynchronous step
const createSlackCommand = async () => {
  const author = await getAuthor() // async function with some git calls
  return new CircleCI.reusable.ReusedCommand(orbSlack.commands['notify'], {
    name: 'notify-author',
    event: 'fail',
    custom: JSON.stringify({
      text: `Commit by ${author}`,
    }),
  })
}

// Define "reusable" job, with native JS
type customJobParameters = {
  name: string
  cluster: string
  container_name: string
  task_role: string
}
const createCustomJob = async (parameters: customJobParameters) => {
  const slackCmd = await createSlackCommand()
  const stepsToRun = [
    new CircleCI.commands.Checkout(),
    new CircleCI.commands.Run({command: "echo 'Hello World'"}),
    slackCmd,
  ]

  // Add deploy step if tag
  // https://circleci.com/docs/variables/
  if (process.env.CIRCLE_TAG) {
    stepsToRun.push(new CircleCI.commands.Run({command: "echo 'Deploying'"}))
  }
  return new CircleCI.Job(parameters.name, dockerExec, stepsToRun)
}

const jobDefinitions: customJobParameters[] = [
  {
    name: 'job1',
    cluster: 'cluster',
    container_name: 'container_name',
    task_role: 'task_role',
  },
  {
    name: 'job2',
    cluster: 'cluster',
    container_name: 'container_name',
    task_role: 'task_role',
  },
]

// Add jobs to workflow
async function main() {
  jobDefinitions.forEach(async jobDefinition => {
    workflow.addJob(await createCustomJob(jobDefinition))
  })

  config.addWorkflow(workflow)

  // Print config
  console.log(config.stringify())
}

main().catch(console.error)

Describe the solution you'd like

It would be awesome if steps array in Job constructor, accepts an array of Command OR async function that returns a Command. It could lead to make this possible:

import * as CircleCI from '@circleci/circleci-config-sdk'

// Components
const config = new CircleCI.Config()
const dockerExec = new CircleCI.executors.DockerExecutor('cimg/base:stable')
const workflow = new CircleCI.Workflow('main')

// Asynchronous step
const createSlackCommand = async () => {
  const author = await getAuthor() // async function with some git calls
  return new CircleCI.reusable.ReusedCommand(orbSlack.commands['notify'], {
    name: 'notify-author',
    event: 'fail',
    custom: JSON.stringify({
      text: `Commit by ${author}`,
    }),
  })
}

// Define "reusable" job, with native JS
type customJobParameters = {
  name: string
  cluster: string
  container_name: string
  task_role: string
}
const createCustomJob = (parameters: customJobParameters) => {
  const slackCmd = createSlackCommand()
  const stepsToRun = [
    new CircleCI.commands.Checkout(),
    new CircleCI.commands.Run({command: "echo 'Hello World'"}),
    createSlackCommand,
  ]

  // Add deploy step if tag
  // https://circleci.com/docs/variables/
  if (process.env.CIRCLE_TAG) {
    stepsToRun.push(new CircleCI.commands.Run({command: "echo 'Deploying'"}))
  }
  return new CircleCI.Job(parameters.name, dockerExec, stepsToRun)
}

const jobDefinitions: customJobParameters[] = [
  {
    name: 'job1',
    cluster: 'cluster',
    container_name: 'container_name',
    task_role: 'task_role',
  },
  {
    name: 'job2',
    cluster: 'cluster',
    container_name: 'container_name',
    task_role: 'task_role',
  },
]

// Add jobs to workflow
jobDefinitions.forEach(jobDefinition => {
  workflow.addJob(createCustomJob(jobDefinition))
})

config.addWorkflow(workflow)

// Print config
console.log(config.stringify())

It would be very convenient to do the same with jobs array parameter for Workflow constructor.

Teachability, documentation, adoption, migration strategy

Typescript (and maybe example) would be enough to document it

What is the motivation / use case for changing the behavior?

With more powerful primitive for async job/command generation, it will help users to use more js mechanism instead of 2.1 yaml.

[KyleTryon]

Makes sense! And should be doable! I think it will make the final compilation done in stringify likely an async function (major/breaking change) but it makes sense.