RFC: Workflow of Arc programs

This issue is about improving the workflow of writing Arc programs.

Current solution

Currently, writing an Arc program first requires creating a Rust-crate. The Rust-crate contains an Arc-file, containing the application logic, that translates into a Rust-file. The crate can then import the generated source into a Rust module.

Pros:

Rust allows circular dependencies within crates. Therefore, this solution is flexible since code inside Arc can depend on code inside Rust and vice-versa.

Cons:

User needs to have basic knowledge of Rust to use Arc.
Since the generated code is monomorphic and name mangled, it can be difficult to use it from the Rust side.

Alternative solution 1

This solution takes inspiration from DDlog (Differential Datalog). DDlog resembles Arc in that programs compile to Rust. If you write a program such as:

// example.dl
input relation Parent(a: string, b: string)

output relation Ancestor(a: string, b: string)

Ancestor(x, y) :- Parent(x, y).
Ancestor(x, z) :- Parent(x, y), Ancestor(y, z).

You can compile it into a crate using the ddlog compiler:

ddlog -i example.dl

Then you need to build the crate with:

(cd example_ddlog/ && cargo build)

The result is a command-line interface that can be executed using:

./example_ddlog/target/debug/example_cli

Inside the CLI, you can write commands to ingest data into the program.

>> start;
>> insert Parent("A", "B"),
>> insert Parent("B", "C"),
>> insert Parent("C", "D").
>> commit;
>> dump Ancestor;

In Arc, we could also generate a crate that includes a CLI for feeding data into pipelines defined inside the script. This should not be too difficult if we use a clap parser generator.

Pros:

Data processing is decoupled from data generation. One Arc program can be used to process many kinds of inputs.
The CLI can integrate with the state API of Arcon.

Cons:

It becomes more difficult to integrate external user-code from Rust into Arc. This functionality is only meant for power users. DDlog solves it by allowing users to declare a custom example.toml file which contains dependencies to custom Cargo crates.

Alternative solution 2

In the previous solution, an individual CLI is generated for each program. A different approach could be to have a CLI for all programs. Each program thus translates into a process that communicates with the CLI using sockets.

cda-group / arc