Workflows overhaul

[MS-Painter]

Currently main idea is to apply one handler to a directory.

Handlers aren't idempotent or atomic currently.

New overhaul describes a way to apply multiple handlers of different order types that could fit in a structured workflow.

What if you could move to a folder then zip the resulting folder? And what if you wanted after the zip to notify a remote target of this change via a db? Or multiple dbs?

This leads us to the this workflow structure idea:

In the above Example Arrived relates to data (file/folder) which was just added which file watcher noticed. In the above Example Results relates to data results (file/folder) from previous action. In the above Example Filepath relates to file path of the file (results or arrived).

Above all this is the configuration to save tracing which is relevant depending on what you want for the workflow. So these tracings could not be stored at all or stored only locally or log rotated locally or provided as the data for tracing exported to a target of your choice from message queues to databases.

These workflows have the constraints of being single directional and cannot go backwards to potentially loop parts of the workflow.

[MS-Painter]

Node - Read file data

[MS-Painter]

Visual graph for input / output?

[MS-Painter]

Petgraph?

DAGs?

Need to coerse a very strict structure for the flow and what's allowed.

[MS-Painter]

CWL based workflow parsing.

There are many ways it could be done but the main issue is these workflows are static in nature. This could be circumvented if we generate just the nodes based on described flow functionality from command line Ex. -

folden generate [EVENTTYPE] [ACTION[]]

EVENTTYPE - Description:
Type of event on file/s to apply workflow on.
Acceptable values: [create|delete|modify]
(?) Flags, can be supplied multiple to work against using "|" seperator
ACTION -  Description:
Multiple can be supplied, 
(?) Order matters

[MS-Painter]

Non visual approach - Zero depth config files:

Configuration generated in a toml file with the same CLI of above comment - Each event|action is stored one after the other.

Then using keywords default data can be replaced to suggest the usage of values from other toml fields.

Example: folden generate creation moveToDir publishToMongo Generates the following file -

[event]
[event.creation]
naming_regex_match = "*"
from_date_created = "08-03-2021 08:50:49"

[action]
[action.moveToDir]
directory_name = ""
requires_exist = ""

[action.publishToMongo]
db_uri="mongo://"
collection_name=""

From the generated workflow we can insert keywords to tell the workflow to use specific inputs from other fields -

event]
[event.creation]
naming_regex_match = "*"
from_date_created = "08-03-2021 08:50:49"

[action]
[action.moveToDir]
Input_path="event.creation.file_path"
directory_name = ""
requires_exist = ""

[action.publishToMongo]
Input_text="event.creation.file_content"
db_uri="mongo://"
collection_name=""

When going over the workflow the file_content or file_path key from the event will be used for for provided logic - Input_... fields will specifically bind the provided key to existing parts in the workflow rather then using the string as is.

Workflow config file parsed as regular toml file rather than structured object.