[Workflows] Create a custom code executor

[martmull]

Scope & Context

We are working on enabling users to run custom code in their workspace. This ticket is about creating a way to execute safely custom code on our server

What

• in settings, create a new section call Functions
• those functions will contain one folder containing files or folders with at least an index.ts file that contains a export const handler = async (event, context) => {} that will be executed by a twenty-code-executor
• each function folder will be stored in the storage defined by STORAGE_TYPE env variable (s3 or local)
• each function will be executed by a code-executor using either aws lambda or local node execution context depending on a env variable FUNCTION_EXECUTION_CONTEXT (lambda or local)
• functions will be stored in a <workspace_schema>.function table that will persist the location of the function
• when creating/updating a function with lambda FUNCTION_EXECUTION_CONTEXT, we will create the lambda on aws by:
  • compiling the typescript handler function into javascript function
  • creating the lambda function using the compiled javascript files. Add the node_modules in an aws layer
  • store the lambda id in <workspace_schema>.function
  • add an info saying that function is ready to be executed in <workspace_schema>.function (eg: sync_status=ready)
• when using local FUNCTION_EXECUTION_CONTEXT, we:
  • compile the typescript handler function into javascript function
  • store compiled files in the file system
  • store the compiled file location in <workspace_schema>.function
  • add an info saying that function is ready to be executed in <workspace_schema>.function (eg: sync_status=ready)
• execute the function using event provided to the code-executor:
  • with lambda FUNCTION_EXECUTION_CONTEXT use invoke
  • with local FUNCTION_EXECUTION_CONTEXT use eval (same node context)
• code-executor will then return the result of the function

[greptile-apps[bot]]

Implementation Steps

1. Create Functions Section in Settings

   • Add a new section in the settings UI for Functions.

2. Define Function Storage

   • Store function folders in the storage defined by STORAGE_TYPE env variable (s3 or local).

3. Create Function Table

   • Define a new table <workspace_schema>.function to store function metadata.

4. Implement Code Executor

   • Create a code-executor service to handle function execution.
   • For lambda context:
     • Compile TypeScript handler to JavaScript.
     • Create AWS Lambda function and store Lambda ID in <workspace_schema>.function.
     • Update sync_status to ready.
     • Execute Lambda with provided event.
   • For local context:
     • Copy TypeScript handler and execute using eval or child_process.exec.

5. Update WorkspacePreQueryHookService

   • Reference workspace-pre-query-hook.service.ts for executing hooks.

6. Handle Jobs

   • Reference call-webhook-jobs.job.ts for job handling logic.

// Example: code-executor.service.ts
import { Injectable } from '@nestjs/common';
import { exec } from 'child_process';
import * as AWS from 'aws-sdk';

@Injectable()
export class CodeExecutorService {
  async executeFunction(event: any, context: any, functionId: string) {
    const functionData = await this.getFunctionData(functionId);
    if (process.env.FUNCTION_EXECUTION_CONTEXT === 'lambda') {
      return this.executeLambdaFunction(functionData, event);
    } else {
      return this.executeLocalFunction(functionData, event);
    }
  }

  private async executeLambdaFunction(functionData: any, event: any) {
    const lambda = new AWS.Lambda();
    const params = {
      FunctionName: functionData.lambdaId,
      Payload: JSON.stringify(event)
    };
    return lambda.invoke(params).promise();
  }

  private async executeLocalFunction(functionData: any, event: any) {
    return new Promise((resolve, reject) => {
      exec(`node ${functionData.filePath}`, (error, stdout, stderr) => {
        if (error) {
          reject(error);
        } else {
          resolve(stdout);
        }
      });
    });
  }

  private async getFunctionData(functionId: string) {
    // Fetch function data from the database
  }
}

References

/packages/twenty-server/src/workspace/workspace-query-runner/workspace-pre-query-hook/interfaces/workspace-pre-query-hook.interface.ts /packages/twenty-server/src/workspace/workspace-query-runner/workspace-pre-query-hook/workspace-pre-query-hook.service.ts /packages/twenty-server/src/workspace/workspace-query-runner/workspace-pre-query-hook/workspace-pre-query-hook.module.ts /packages/twenty-server/src/workspace/workspace-query-runner/jobs/call-webhook-jobs.job.ts /packages/twenty-server/src/workspace/workspace-query-runner/workspace-pre-query-hook/workspace-pre-query-hook.config.ts /packages/twenty-server/src/workspace/workspace-query-runner/workspace-pre-query-hook/types/workspace-query-hook.type.ts /packages/twenty-server/src/workspace/workspace-query-runner /packages/twenty-server/src/workspace/workspace-resolver-builder /packages/twenty-server/src/integrations/exception-handler/hooks

#### About Greptile

This response provides a starting point for your research, not a precise solution. Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. [Ask Greptile](https://app.greptile.com/chat/github/twentyhq/twenty/main) · [Edit Issue Bot Settings](https://app.greptile.com/apps/github)