CLI: generate workflow.json

christad92 commented 6 months ago

Overview

Developers and technical users often define their own workflows to test things out locally by writing their workflows in the JSON format that is quite fiddly and requires the developer to remember the right key names and structure (which is not always intuitive).

See the Workflows section of the readme.

Background

The OpenFn platform is an open source tool for data and workflow automation which provides users with the two different types of interface: The CLI and Lightning (a web based platform that provides a GUI/Canvas for users to design their workflows, write job expressions and run work orders.)

Requirements

We should add an interactive command to the existing CLI which makes it easy to generate a workflow. The command should ask the following questions:

What should the workflow be called
What is the name of the first step
What is the adaptor for the next step
Would you like to generate another step?
Where would you like to output the workflow? (accept a folder, not a file, default to the working dir)

Note the following:

Each step should have an id auto-generated from the name. The id should be "slugified" (spaces converted to -, special characters removed etc)
For each step, we should generate a .js file using the step id. The workflow should point to this file (expression: "./fetch-data.js", and each expression file should contain a basic expression of the form fn((state) => state)
If another step is generated, the previous step should have an edge (next) generated for it, with condition !state.error
The user can add as many steps as they want
A workflow should not be given an options object

Pretty much the minimum we expect is to generate two jobs with an edge between them. Ideally you could run this in the CLI directly without any errors (this will work if the adaptor is common).

Testing

It is too hard to unit test the final automated command, but we do expect unit tests on the individual functions used by the command.

Useful Resources

Documentation for Workflows
Documentation for OpenFn CLI

smitjiwani commented 5 months ago

Hey @christad92 , I would like to work on this project as part of DMP 24.

glitcher007 commented 4 months ago

Hi @christad92

I'm proposing enhancements to the OpenFn CLI to streamline workflow creation:

Workflow template generation command. Interactive workflow sketching command. We need expertise in:

CLI development Workflow automation JSON manipulation Environment variable handling Let's discuss further.

alishad846 commented 4 months ago

name: CLI Workflow Tooling about: OpenFn's submission for the Code for GovTech program title: CLI Workflow Tooling labels: OpenFn, Lightning, History assignees: ''

Description

The OpenFn platform is an open source tool for data and workflow automation which provides users with the two different types of interface: The CLI and Lightning -- a web based platform that provides a GUI/Canvas for users to design their workflows, write job expressions and run work orders.

Developers and technical users often define their own workflows to test things out locally by writing their workflows in the JSON format that is quite fiddly and requires the developer to remember the right key names and structure (which is not always intuitive).

There are a few ways the CLI might help us out:

A command to generate a workflow template (with two jobs and a start that can be edited)

An interactive command which lets you sketch out a workflow upfront, asking you for each job, what adaptor and credential it wants

Useful Resources :

Documentation for Workflows

Documentation for OpenFn CLI

Side note:

It might be nice to have some way to load env vars into JSON. This is easy if we can load workflows as JS files, but maybe we can use some kind of convention like "user": "$OPENFN_SALESFORCE_USERNAME", which will load $names from the env.

Goals

[ ] Make it easier for developers to design workflows

[ ] Reduce the time it takes to write workflows

Expected Outcome

Improved developer experience for engineers who use the CLI

Acceptance Criteria

[ ] Generated workflow.json can be run from the CLI without syntax errors.

Product Name

Product Name: OpenFn

Project Name

Project Name: CLI Workflow Tooling

Organization Name:

Open Function Group

Domain

[Others]

Tech Skills Needed:

Javascript

Mentor(s)

Complexity

[Medium]

Category

Pick one or more of [Performance Improvement], [Feature], [PoC]

Sub Category

Pick one or more of [API], [Backend], [Configuration].

Hey @christad92 , I would like to work on this project as part of DMP 24

taylordowns2000 commented 4 months ago

Hi all, apologies for the confusion here. This particular issue is not being sponsored as part of DMP 2024. You can find a list of all DMP 2024 issues here: https://github.com/orgs/OpenFn/projects/3/views/3?filterQuery=is%3Aissue+label%3A%22DMP+2024%22

ShreyanshGautam commented 4 months ago

name: CLI Workflow Tooling about: OpenFn's submission for the Code for GovTech program title: CLI Workflow Tooling labels: OpenFn, Lightning, History assignees: ''

Description

The OpenFn platform is an open source tool for data and workflow automation which provides users with the two different types of interface: The CLI and Lightning -- a web based platform that provides a GUI/Canvas for users to design their workflows, write job expressions and run work orders.

Developers and technical users often define their own workflows to test things out locally by writing their workflows in the JSON format that is quite fiddly and requires the developer to remember the right key names and structure (which is not always intuitive).

There are a few ways the CLI might help us out:

A command to generate a workflow template (with two jobs and a start that can be edited)

An interactive command which lets you sketch out a workflow upfront, asking you for each job, what adaptor and credential it wants

Useful Resources :

Documentation for Workflows

Documentation for OpenFn CLI

Side note:

It might be nice to have some way to load env vars into JSON. This is easy if we can load workflows as JS files, but maybe we can use some kind of convention like "user": "$OPENFN_SALESFORCE_USERNAME", which will load $names from the env.

Goals

[ ] Make it easier for developers to design workflows

[ ] Reduce the time it takes to write workflows

Expected Outcome

Improved developer experience for engineers who use the CLI

Acceptance Criteria

[ ] Generated workflow.json can be run from the CLI without syntax errors.

Product Name

Product Name: OpenFn

Project Name

Project Name: CLI Workflow Tooling

Organization Name:

Open Function Group

Domain

[Others]

Tech Skills Needed:

Javascript

Mentor(s)

Complexity

[Medium]

Category

Pick one or more of [Performance Improvement], [Feature], [PoC]

Sub Category

Pick one or more of [API], [Backend], [Configuration].

Hello sir @christad92 i would like to contibute my skill to this project as a part of DMP24.

josephjclark commented 4 months ago

Interactively generating a workflow.json would be a good first issue. But after some comments from Mtuchi this morning I want to be absolutely sure that the generated JSON is lightning compatible

OpenFn / kit