Provide a Common Execution Context Shared Among Tasks

CurroRodriguez commented 7 years ago

Story

As a task implementer, I want to share state between two separate tasks, so I can implement dependencies and decisions and still maintain independent tasks decoupled.

Description

The best way to describe this user story is to give some practical examples of some of the problem I've run into while using Bolt with other systems and some of the benefits the feature will provide.

One common problem I run into while using Bolt within a CI/CD system is that I need to make some decisions externally to Bolt to control which tasks should be executed; or alternatively, create a more complex task that makes those decisions and performs all the intended operations. As an example, this is a situation I run into with some of the projects where I use Bolt:

Let's say during the execution of a project build process I want to generate a wheel of my Python library or application and submit it to PyPi (some CI/CD systems provide this functionality out-of-the-box, but bare with me for a moment). Ideally, I will only generate the wheel if the build process is happening in the master branch and the current version I am building does not already exist in PyPi because I don't want to override an already posted wheel and I don't want the build to fail because I forgot to update the version. This is how some pseudo-code for that functionality looks like:

if branch=="master" and not version_exists(version):
    build_wheel()
    post_wheel()

My options right now are to create a single task that performs both operations and checks both conditions. In other words, the logic has to happen within the same task because I cannot share states among tasks. This creates a humongous task that has a lot of knowledge about my CI/CD system. For example in Jenkins, I will have to know that the system sets an environment variable BRANCH_NAME to determine my current branch. I will also need to know how to check PyPi for existing versions, and if the conditions are met, then I have to build the wheel and post it in the same task.

Another example is when a task will always depend on a set of pre-conditions in order to be able to execute successfully. In a recent project I ran into a situation where I want to automate triggering the load testing through bolt. For that to be accomplished successfully, I need to:

Start the load testing server.
Wait for the SUT (system under test) to be available.
Trigger execution of the tests.

Turns out that in this situation, I cannot execute step 3 unless 1 has been executed. If I miss to add step 1, step 3 cannot execute successfully, and since this will execute unattended, it may be hard to figure out what's going on.

The Solution

I've been pondering about the best way to implement this sharing of state without creating something that complicates the implementation of tasks or that complicates the nice straight execution flow of Bolt, and I had many solutions in mind, but the most flexible solution is to provide an execution context object to each of the executed tasks, which they can use or ignore, but that will allow them to store some state that then can be retrieved by a subsequent task.

Currently, task are any callable that gets registered under a "name" or "id", and it is recommended that the callable's prototype is defined as callable(**kwargs) to allow further extension of the parameters passed to a task. This is one of those cases where this will come handy because we can pass a context object to any existing task without affecting behavior since those existing tasks will just ignore the parameter, and new tasks that want to leverage the functionality can still access to it.

Right now, tasks are invoked as:

task_callback(config=config)

And within the task you can do:

def my_task(**kwargs):
    config = kwargs.get('config')
    # Get configuration values from config

The idea is to invoke the task with a context object that will be shared by all tasks, so you can store state and access it from a subsequent task. Therefore, the invocation will be:

task_callback(config=config, context=context)

And now tasks can do something like:

def task_1(**kwargs):
    context = kwargs.get('context')
    # do the work
    context.task_1_state = True

def task_2(**kwargs):
    context = kwargs.get('context')
    if context.task_1_state:
        # do something
    else:
        # do something else

This is an un-intrusive way of sharing state among tasks without coupling the implementation of both tasks or creating a humongous task to accomplish the work.

CurroRodriguez commented 7 years ago

Issue #75: Provide a Common Execution Context Shared Among Tasks

CurroRodriguez commented 7 years ago

This is implemented in PR #84.

abantos / bolt