feat: refactor call exec

[thantos]

Work in progress, major refactor of calls and events within eventual.

Goals:

• Reduce work to add new resources and calls/operations
• Reduce duplicate logic between the workflow and workers
• Remove all logic from the CORE and reduce coupling/dependencies within core
• Remove the confusing "hooks" and "register" methods
• Create a single set of all known Calls required to fully support the system.
• Improve the ability to extend eventual from outside in the future (no more global if/else or register calls) blocks.
• Improve the ability to support smart tree shaking in the future.
• Reduce or at least improve visibility of feature parity between workers and workflows (ex: workflow supports ChildWorklow, but not StartWorkflow, worker supports ExecutionHandle).

Out of Scope (but wanted)

• Replace the Per-Call Success and Failed events with generic ones (ex: EntityRequestSucceeded and EntityRequestFailed with CallSucceeded and CallFailed).

Overview

• Call - a named operation or group of operations that are defined within Eventual (ex: TaskCall - myTask()). They may have a payload and may contain sub-operations within them for simplicity (ex: EntityRequest - myEntity.put/get).
• Property - a new concept representing synchronous value lookup. Like a Call, it may have a payload, but it must be synchronous from end to end. Examples are retrieving the BucketPhysicalName from a bucket, retrieve the open search client, or just get the service name in a uniform way.
• Client/Store - an interface or class that defines how to invoke real resources like AWS. EntityClient.put("myEntityName", ...params)
• EventualHook - A set of methods that accept Calls or Properties and returns promises or values. Registered by the system, for example the workflow executor or the entity worker.
• Worker and Orchestrator/Workflow - A general term for a unique wrapper around user logic. For example, the Entity Stream Worker executes user logic when an Entity Stream event is triggered. It must bring with it real clients and deliver call to the correct client. The workflow Orchestrator is like a worker, but more special, it captures all Calls while executing the workflow, turns them into events, compares them to past events, and then only calls clients when necessary.
• Executor an executor is a discrete piece of logic or logics that handles a Call. In come cases it maps the call to it's Client or it may map a Call to events/tasks for a workflow. Each worker must provide an Executor for each call, even if it is to say the call is unsupported. Executors can be composed with one another or wrap other executors. They are typesafe.
• EventualHook an interface accessible to user space code (core) that allows executing calls and retrieving properties. This interface is replaced in core-runtime as needed, generally with Executors and PropertyRetrievers.

graph TD
    UserCode --> Resource
    subgraph Core
        Resource --> Call
        Resource --> Property
    end
    subgraph Runtime
        Property --> PropertyRetrievers
        subgraph Worker
        Call --> WorkerExecutor
        end
        subgraph Workflow
        Call --> EventualFactory -->|Aka a Promise| Eventual
        Call -->|Transform| Event --> History
        Call -->|When the Call is New| WorkflowExecutor
        end
    end
    Workflow -->|Save/Load| Clients
    WorkerExecutor -->|Call| Clients
    WorkflowExecutor -->|Call and Add to Queue| Clients
    subgraph Environement
       Clients --> Cloud
       Clients --> Local
    end

Task List

• [x] - remove hooks in favor of a single EventualHook.
• [x] - clear relation between core and runtime using Calls (existing) to Executors (new)
• [x] - Remove or reduce duplicate mapping between calls and events and client calls
• [x] - Workers
  • [x] remove "register" calls in favor of the enterEventualHook scope with locally defined executors.
  • [x] Organization: rename and refactor executors to decouple from clients (from this refactor's early stages...)
• [x] - Workflow
  • [x] Replace Workflow-Executor logic with shared executors from the workers
  • [x] Add wrapper to turn any executor into an event to the execution task queue (ex: bucket, entity)
  • [x] Replace the per History State Events with a single CallEvent (still has a sub-type)
  • [x] CallEvents
    • [x] are no longer generated when the call is invoked (workflow-call-executor), they are generated within the workflow-executor when a call is made by the user.
    • [x] events are compared to the past events (and not calls anymore) to determine deterministic logic
  • [x] Replace the eventual-factory with special executors that can be co-located with the WorkflowCallExecutors (WorkflowCallToEventualDefinitionExecutors).
  • [x] ExecutionHandle returned by StartWorkflow SHOULD work within a workflow
  • TODO later: Support StartWorkflow, GetExecutionStatus, TaskRequest (succeed, fail, heartbeat) call within a workflow
• [x] - Transactions: Update to use executors where possible
  • Note - the handling of calls is pretty custom within the transaction executor, so I left this mostly as is.
• [x] - Remove unessecary hooks and or roll them into properties/calls
  • [x] - ServiceType
  • [x] - TaskRuntimeContext
• [x] - Organization: the names of things need another pass, the meaning has changed during the refactor
• [x] - Tests Passing

Result

• The Core user land code only uses EventualHook (make calls, get proeprties) and RegisterResource (register a resource)
• There is an executor and property retriever for each Call
• There is a workflow executor and workflow eventual factory for each Call (or use common helpers)
• ServiceType, TaskRuntimeContext (token), EventualManifest, ServiceClient, and OpenSearchClient are rolled into the Properties concept
• Workers can override executors and proeprty providers to customize (or just create their own EventualHook(s)), for example, the TaskWorker provides a function to provide the TaskToken on request.
• Was able to remove a lot of content from the core/internal package. Some of it was moved to core-runtime (ex: Result) and some of it was just removed (ex: some hooks).
• The workflow now compares historical events to the entire event generated during the current execution (no more isCorresponding). Events MUST be serializable.
• Calls SHOULD be serializable, through I don't that is currently true.
• BUG Fix: Bucket put would fail with a stream when not in a workflow
• BUG Fix: open search workflow support was broken for many reasons

[thantos]

smart tree shaking in the future

To expand on this... if we know what resources or at least the imports of a single worker function, we know which clients/executors to include or exclude. Maybe pre-mature optimization, but just getting the groundwork for more dynamic dependency inclusion.

[thantos]

This was a good exercise, I found a bug that existed in workflow and not workers (open search) and a bug that existed in workers and not the workflow (bucket put with stream). The code is more shared now, should be an issue less often.

[sam-goodwin]

;)