Parallelize using thread pool

[WilliamRagstad]

Dispatch a new job on the first available thread from a pool in: https://github.com/webx-net/webx/blob/62e405e8e75440d4ff7214d1252eeb59ff219798/src/engine/runtime.rs#L746-L752

Each thread running handle_request should be concurrent to optimize IO utility time further:

• 1

Thus, the threading model should look like this:

graph TD;

RT["Runtime thread\n(run function performing main event loop)"] -->|"listener.accept()"| TP;
TP["ThreadPool Handler"] -->|assign| T1["Thread 1"] --- T1A["..."];
TP -->|assign| T2["Thread 2"] --- T2A["..."];
TP -->|assign| TX["..."];
TP -->|assign| TN["Thread n"];
TN --- A1["Handle Client 1"] ---|"Client 1 awaits"| A2["Handle Client 2"] ---|"Client 2 is done, resume Client 1"| A12["Handle Client 1"];
TP -.-> A2;

[WilliamRagstad]

This diagram represents the following thread structure and interactions:

1. Main Thread: The application's entry point is responsible for initializing and starting the thread pool.
2. Thread Pool: Manages the worker threads, distributing tasks among them.
3. Worker Threads (1, 2, ..., N): Individual threads for executing tasks. These threads access the JS Runtime Manager.
4. JS Runtime Manager (Singleton): A singleton instance accessible by all worker threads, managing the JavaScript runtime.

The flow starts with the main thread, which initializes the thread pool. The thread pool then manages multiple worker threads, each capable of processing tasks independently. All worker threads access the singleton JS Runtime Manager to execute JavaScript or TypeScript code, ensuring a shared and persistent runtime state across the application.

graph TB
    A["Main (Thread)"]
    Dev["Dev Mode Features (Thread)"]
    subgraph Tokio["Tokio Runtime (Thread Pool)"]
        W["Web Server"]
        C["Worker 1"]
        D["Worker 2"]
        E["Worker N"]
    end
    F["JS Runtime Manager (Thread + Singleton)"]

    A -->|"Starts"| F
    A -->|"Starts"| W
    A -.->|"Starts"| Dev
    W -->|"Starts"| C
    W -->|"Starts"| D
    W -->|"Starts"| E
    C -->|"Accesses"| F
    D -->|"Accesses"| F
    E -->|"Accesses"| F
    Dev -.->|"Accesses"| F

    style Tokio fill:#803037

[WilliamRagstad]

A tip is to start with a more straightforward implementation and not go into parallelization with worker threads or optimizing the workers with concurrency. Thus, first, make sure this works correctly:

graph TB
    A["Main (Thread)"]
    W["Web Server (Thread)"]
    Dev["Dev Mode Features (Thread)"]
    F["JS Runtime Manager (Thread + Singleton)"]

    A -->|"Starts"| F
    A -->|"Starts"| W
    A -.->|"Starts"| Dev
    W -->|"Accesses"| F
    Dev -.->|"Accesses"| F

All accesses should be performed via thread-safe mpsc channels and message enums. Come up with a standard communication method.

[WilliamRagstad]

Essentially we are constructing an actor model using channel messages to share the mutable JsRuntime between multiple worker threads. All access arrows are representing bidirectional actor communication for request-based RPC over channels.

There is a great talk by Alice on this: https://youtu.be/fTXuGRP1ee4?si=Him09QiiZ2HE00o0

A unique solution for "async" responses would be to use the backchannel pattern to send a Sender<ResponseT> in the request using channel.send(Msg(data, response_tx)) followed by a blocking response_rx.recv() which hangs the worker thread momentarily.