What if ... we create a language designed to compile to Rust - for fun!

[bvssvni]

I think working on language design is fun, and don't think that just because people can use Rust, there should be no new innovation in this design space.

Why are we making all these assumptions about "correct" language design, when in practice people find it useful to have features that make their everyday work easier?

This thread is about language design ideas.

[bvssvni]

cc @dobkeratops

[bvssvni]

Some ideas:

• Take a bunch of lessons from Dyon
  • 4D vectors
  • link-syntax
  • current objects
  • optional namespaces
• Direct control over dependencies (making is easier to swap backends without use of generics)
• Parallel processing of higher dimensional arrays (Dyon-like loops, but parallel and more efficient)
• Optional directional arbitrary generic inference (some idea from Jai)
• Compile time evaluation (grab at top level)
• [] instead of <> for generics
• Optional grab expressions for closure capture (control over partial evaluation)
• Automatic proof of trait equivalence (making it possible to duplicate interfaces)

[bvssvni]

I think Dyon made a lot of good design choices, however, the most annoying bits are:

• It is a scripting language (sometimes you want better performance)
• It does not have struct or enum
• No match expressions
• It has a safe, but limited memory model
• Types are not guaranteed (you can convert to any and then back)
• You can't load Rust libraries (everything needs to be put in a Dyon wrapper)

Basically, if Dyon was more like Rust, then it would be nice, but Rust does not have all the convenience that Dyon has.

Also, Dyon's focus on safety makes it less powerful:

• You can use grab at top level, but this is only for static data for use in hot loops
• Sometimes you need low level stuff, u8, u16 etc. and pointers

[bvssvni]

The biggest pain point of Rust is how it handles dependencies:

• Interfaces (traits) need to be put in their own crates with version numbers
• Module hierarchy form a tree, with no external control of swapping nodes
• Rebuilding all dependencies for every project, with little lack of control

I think the point of interfaces in language design is that they are interfaces. If you wrap them in libraries with version numbers, then the whole point of interfaces goes away. I wonder how many hours of ecosystem maintenance I could have avoided just by having smarter interfaces in Rust. Think meta, don't think that interfaces is code you depend on. Interfaces are mathematical objects with mathematical properties. It should be possible to prove traits to be equivalent and use them interchangeably (like Homotopy Type Theory treats types as equivalent).

The today's need for shipping to various platforms requires a language suitable to swapping code without needing to define an explicit interface with generics. Generic programming is just pain.

You should be able to control how a project builds, not just rely on crates.io. A project should try to reuse existing binaries to avoid geometric increasing building times with project complexity.

[bvssvni]

Nicknaming this language "Drun" for now.

Like Dyon, modules are loaded externally in Drun, but one compiles instead:

fn main() -> res {
    math := load(rust: "math.rs", namespace: "math")? // Load Rust module
    program := load(drun: "main.drun", imports: [math])?
    exe := compile(program, "bin/main")? // Generate executable
    run(exe)
}

One can write modules in Rust and integrate them into Drun projects. Drun files expand to Rust code and are compiled as static libraries using the Rust compiler. When the source code changes, the loader scripts detects this automatically and rebuilds only libraries that have changed. All libraries are stored in a common folder with a signature file telling what the interface is and the source tree. There are no version numbers, because the "version" is the whole source tree requested from a loader script for a library. The signature files predicts whether compilation will succeed.

The syntax of Drun is close enough to Dyon such that most loader scripts can be run in a Dyon wrapper. This also means that some Dyon modules might be compiled to Rust, or at least a subset. Explicit conversion from Dyon to Drun might also be possible.

[dobkeratops]

heh interesting thread.

I've had my own foray into writing a language and probably wont do it again, but I remain interested in ideas.

my findings with rust main ambiguous and it's a shame because 95% of it is really good.. it would only take simple tweaks to dramatically improve it.

At the same time I do persevere on and off with Rust since part of my goal is exposure to ideas outside of my existing interests, and there is that 95% that I like :)

So your idea of making something that compiles to it... might not be so insane (as it would be easy to leverage the underlying features.

I have always also wondered if it would be practical to maintain a fork with those simple tweaks

lately I've put a bit more time into improvising tools (because I realise many of my frustrations aren't with Rust itself, but rather with losing C++ IDE's )

I have a huge number of distractions as it is but maybe i'll take another look at what you tried in Dyon. .I seem to remember "dynamic scope" which you called something else with ~syntax ('current object variables or something?)

at the same time sometimes find cases where it really takes me time to get out of my preconceptions , e.g using enum/match more can cut back on the amount of parameter passing) the pain point in 'pass-from-above' which itself i'm generally in favour of as a means of achieving threadsafety and stability. )

[dobkeratops]

(incidentally the next time I do use some embedded scripting, i'm probably going to head toward a lisp dialect but keep it simple - having done a bit more improvising in elisp/emacs to improve my rust experience, lisp has grown on me quite a bit. i just remembered the dynamic scope/dyon again, which is one of many things people rediscover which various LISPs have had for a long time)

[dobkeratops]

regarding rapid iteration i have another strange idea to try ... automatic A/B testing. Imagine if you could use macros to mark things you want to toggle in experiments, and just have multiple machines setup to simultaneously build several permutations. as it happens I have some raspberry pis lying around, and a current+spare desktop machine aswell as a laptop. I'm sure many people do cross-platform builds simultaneously.

[dobkeratops]

oh and yet more thoughts on this subject

regarding Dyon, and scripting languages generally, note the amazing versatility of modern compute shaders. its strange that we talk about performance vs scripting languages, but then the main compute engine of a modern machne is actually useable with on-the-fly re-loadable code modules... There does seem to be a lot of scope for using custom languages etc to drive the GPU. so.. language design remains an open area..

[bvssvni]

In Dyon you can write:

for i, j, k { arr[i][j][k] = random() }

However, this runs sequential, which is slow. Also, the length for the inner loops depends on the arrays of the outer loops:

for i len(arr), j len(arr[i]), k len(arr[i][j]) { arr[i][j][k] = random() }

In Drun I want to write this:

for i, j, k { arr[i, j, k] = random() }

This means that the lengths are the same (multi-dimensional) and arr can be flattened to a single dimension, which is detected by the compiler as "this is done for every element". The compiler injects parallel chunk processing with Rayon under the hood.

[bvssvni]

You can't do parallel chunk processing in Dyon that efficiently, because the run-time needs to exist in order to interpret the AST. Parallel chunk processing would require customized run-time environment with a local scope.

However, when compiling to Rust, you are no longer limited to such design choices. You can leverage existing libraries.

[bvssvni]

Also, once you have parallel processing semantics, it is easier to generate shaders from Drun. There could even be a hot-swap feature, where you detect file changes and re-load the shader at the run-time, as long as it confirms to a type signature.

[HeroicKatora]

Going in a different direction: One particular pain point of mine is the ability to hot-swap modules. This work flawlessly in elm, partially in Javascript, or is simply a fast compiler for Go, or is as bad as custom-built, operating system and architecture specific JIT-compiler-patch-injection monsters in C. Ideally, it would be possible to transparently switch components into an interpreted mode where hot-swapping is as simple as writing new source code to a file.

[bvssvni]

Drun could have a byte-code interpreter in addition to compilation to Rust. This might even be JIT-ed using various Rust crates. The cool thing about generating Rust code is that you have meta-control over how the byte-code interpreter is running, since Drun itself is written in Rust!

[bvssvni]

Nothing stops Drun from using itself as a library. This means that the language can depend on features where Drun uses itself as library, e.g. for compile-time evaluation. Just compile the program for the expression, run it and inject the result back into the AST as serialized Drun-code.

[dobkeratops]

One particular pain point of mine is the ability to hot-swap modules.

right I touched on this above: I've done hot-swapping shaders in the past, and note that today we have extremely versatile compute shaders.

Isn't it ironic that hot swapping of massively parallel compute intensive work is actually within reach :) - and it's not hard to imagine controlling this from a scripting language.

I'm unlikely to actually try to use Dyon but given all these possibilities it's a general area and discusion that interests me. When I was building my own language I did hope I could try some of this but in the end i figured (after 2months getting the itch out of my system) that whilst it's quick to get a language going, there is so much around it that it's hard to maintain. I know I should focus on mastering existing languages.

I had another foray into Haskell recently , have you seen LambdaCube? this is extremely impressive i.e. a DSL from within Haskell lets a single program embed shader code to get something onscreen. Now imagine extending that to compute-shaders..

http://lambdacube3d.com/editor.html

I'd like to emphasise that streamlining CPU<->GPU interaction could be more important than fully utilising the CPU.. given that these days the GPU really is the main computational engine ( although that means you also seriously dont want the CPU bottlenecking it.. and there is certainly still a lot that the GPU is unsuitable for)

[bvssvni]

I picture Drun starting out as a Dyon-wrapper:

1. Leverage the Rust compiler
2. Leverage shader languages for hot-swapping
3. Leverage JIT-crates in the Rust ecosystem
4. Leverage parallel processing-crates in the Rust ecosystem
5. Leverage Dyon-integration by generating Rust code
6. Start copying ideas from Dyon into its own language

[bvssvni]

There is no problem using Piston-Meta to extract type information from shaders. One does not need to write the Rust code for conversion more than once, just use different Piston-Meta grammars for different shader languages.

This can be converted to a common type interface that Drun uses internally. This type interface can have many mathematical properties such as whether a type inferface is subset of another etc.

[bvssvni]

Drun can also extract types from Rust code, then call into shaders from Rust that are hot-swapped. You can call these shaders as normal Rust functions, because Drun generates the dependencies-wrapper which it feeds the Rust compiler. So, from a Rust perspective, shaders just become an extension of the language.

The whole trick is taking control over the building process. Drun defines how this building process takes place, so you never write the "application top level"-code. Instead, you just write a script for the Dyon wrapper for Drun and call drun <app.drun> on the command line. It outputs a standalone executable which can be run without the script, and it also runs the executable directly if Drun detects that no files have changed.

[bvssvni]

The semantics that makes it possible to make Rust call into Dyon code or shaders, is simply swapping the dependencies. This is the big lesson from Dyon.

For example:

fn main() -> res {
    math := load(dyon: "math.dyon")?
    app := compile(load(rust: "app.rs", imports: [math])?)
    run(app)
}

However, the same code might run with GLSL:

fn main() -> res {
    math := load(glsl: "math.glsl")?
    app := compile(load(rust: "app.rs", imports: [math])?)?
    run(app)
}

You don't have to change the Rust code, if you have control over the building process.

[dobkeratops]

tangentially , i've written some Rust Macros to streamline shader parameter binding a little (eg a vertex type that can generate a helper struct for setting up gl layout/attribs, and somethign for getting back uniforms)

I would be curious to know how far you could get with "procedural macros" (i haven't touched them),

and perhaps as a middle ground between 'new langauge' and Rust.. what about generating source files (which are still readable and searchable with rust tools)

[bvssvni]

The whole point of Drun is to generate Rust source files. In that sense, it's not just a language, but a way of configuring the build process by deciding which dependencies you use in the project.

[dobkeratops]

Generic programming is just pain.

it's got up and downsides; i'm used to relying on it from C++ alot.

What I appreciate in Rust is you can use it more extensively with the better error messages (thanks to traits),

but what I miss from C++ is nested classes which let you share type-parameters between many related items, e.g you can do this sort of thing..

    template<typename NodeData,typename EdgeData>
    class Graph {
        class Edge {
            EdgeData data; ..link stuff ..
        }
        class Node {
             NodeData data; .. link stuff..
        }
        // stuff in the 'graph' dealing with nodes and edges, 
        // accesors for NodeData/Edge data for the users
        void create_edge(NodeData& a, NodeData& b, EdgeData& d) { .. }
        void iterate_edges_of(Node& n, function<void(NodeData& d, EdgeData& e, NodeData& other)> f) {...}
         etc etc..
    }

translate that into Rust .. you'd have a mod graph and graph::Edge, graph::Node(so far so good) but then the impl's will be heavily peppered with repetition of the NodeData/EdgeData type parameters everywhere (thats where it gets ugly and bloated fast)

If they added module level type parameters that would be amazing (better than C++), but I dont have syntax suggestions for that. (this honestly would be one of my bigger rust language requests)

an embedded module would be obvious - mod foo<X> { ... } - but what would be a way of specfying file level params? It would seem weird to declare them in main/mod.rs e.g. main.rs ."mod foo" (then 'T' is available in foo.rs) . I suppose with a code-generator you can do anything type-params could have done..

[bvssvni]

Also, since Drun is using Rust's type system, one can just another Piston-Meta grammar for a subset of Rust, such that Rust modules can be hot-swapped too.

[HeroicKatora]

A reason why hotswapping is generally avoided, is to work around ABI compatibility concerns. However, in a language that embeds these abi-declarations into its object code one could check this compatibility dynamically in the trampoline that makes calling hot-swapped code possible. Just make sure that the caller assertions on the abi format such as object layout, ..., are visible to the module or the other way around. Depending on complexity, it could be possible to correct minor differences through a polyfill-call-convention.

This is strongly related to shaders, by the way they are ensuring correctness in Vulkan. A declaration in glsl such as uniform sampler2D creates a abstract interface to several possible different samplers. Those declarations are/can be checked on pipeline creation, which are yet another object explicitely denoting their input and output interface. Then the pipeline I/O interface is consumed by a draw call, which had it own interface in the form of render passes. The only chink in this tiered system is that there are some complications and some checks may pass mistakenly but that should not be the most pressing issue when designing such an explicit interface module system from the ground up.

The definitional issue can also be solved without respect to program execution, i.e. how your code behaves when an interface is violated.

[bvssvni]

For example, this is how I picture control over hot-swapping in Rust:

fn main() -> res {
    foo := load_hot(rust: "foo.rs")? // Loads a hot-module
    bar := load(rust: "bar.rs", imports: [foo])? // Make `bar` depend on `foo`
    ...
}

When "foo.rs" changes, it gets hot-swapped by the application.

You can mix hot-modules with compiled modules by injecting wrappers in their inteface. With the piston-current crate it is no problem sharing a run-time at thread level.

[bvssvni]

By using Drun, when writing Rust, you can refer to modules which never are written in Rust directly:

mod shader_toy; // A module written in Drun/GLSL

// Must be `pub`.
pub fn main() {
    shader_toy::run();
}

This is because Drun copies the Rust source file to a new directory and injects new dependencies, generating Rust code, that corresponds to Rust's standard source tree. In order to set this up properly with hot-swapping, run-times and current objects, a new main function is generated that calls the Rust main function.

From Rust perspective, these injected modules behaves just like ordinary Rust code, because Drun is controlling the build process. Again, controlling the build process is the key to making seamless integration possible.

[bvssvni]

Btw, since Drun generates Rust code, it can just wrap all current objects in Drun code in new-types and use piston-current. Then, it generates a prefix for functions using the ~ syntax that brings these objects into scope safely.

[bvssvni]

Current objects look like this:

fn main() {
    ~ foo := 3
    bar()
}

fn bar() ~ foo: f64 {
    println(foo) // Prints `3` when called from `main`
}

[bvssvni]

Drun will probably use f64 by default for numbers, like Dyon.

foo := 3 // `f64`
foo : f32 = 3 // `f32`
foo := bar() // inferred to be `f32`

fn bar() -> f32 { ... }

Conversion to usize will be implicit, all other conversions will be explicit, except conversion from f64 to a component of a 4D vector.

4D vectors will be [f32; 4] by default.

[bvssvni]

Drun will probably use call-by-reference everywhere, that is immutable by default and set to mutable with the mut keyword.

Lifetimes in Drun will be similar to Dyon by default, such that instead of comparing lifetimes individually, they are interpreted as "proof that the lifetime plus all lifetimes in X live longer than lifetime of Y":

// Lifetime of `a` plus all lifetimes in `a` outlives `b`.
fn foo[T](a: 'b T, mut b: [T]) { b[0] = a }

This means that b can store a. I think this solves 90% of all use cases.

The corresponding Rust code is:

fn foo<'c, T: 'c>(a: &'c T, b: &'c mut [T]) { b[0] = a.clone() }

.clone() is automatically inserted when assigning from &T to T.

Another example:

fn foo[T](a: 'b T, mut b: [&T]) { b[0] = a }

Here, the Rust code generated is:

fn foo<'c, T: 'c>(a: &'c T, b: &'c mut [&'c T]) { b[0] = a }

[bvssvni]

I wonder whether the Drun compiler can become smart enough to treat general pointers like this:

// Infer that `a: 'b` and insert `clone` at call site.
fn foo[T](a: *T, b: mut [*T]) { b[0] = a }

foo(a, b) => foo(clone(a), b)

Then, Drun can reduce clone calls that are not necessary.

[bvssvni]

@dobkeratops I have an idea for module-level type parameters:

mod foo {
    type Bar;

    ...
}

use foo::{Bar = f32};

[dobkeratops]

IMO invoking them would probably be more intuitive if they looked exactly like other type parameters i.e. use foo<f32>; i did think it might be like 'over-riding type's' but you'll need to declare them with type-bounds like other rust generics.. there should ideally be no difference between a struct,trait,or fn type parameter and a module type parameter (it's as if you just cut-paste the parameter into each item .. within the module)

mod foo { type Bar; ...} -> maybe better to write mod foo<B:Whatever> { ... }

but then problem then is how to do it for files.. something like this?

foo.rs

self<B:Whatever>;   // B is now a type parameter for this module

or mod<B:Whatever>; or even just <B:Whatever> as the first line.. ugly. it's like the file is an implicit {...} and type-parameters really belong outside .. (the idea of mod or self is to apply something to 'the whole unit' ..

maybe ..

     // in foo.rs
     mod self<F:Float>; // declaration of current module (foo) with added details..

it's not at all obvious to me what would be comfortable

[bvssvni]

Hmm...

mod<Bar> foo {

}

use foo::{Bar = f32};

[dobkeratops]

thats probably better , but i still think the use should be <>, also IMO the Bar would go after the foo for symmetry with other items, eg:-

struct Vec3<T>{ 
    x:T,
    y:Z,
    z:T
}
fn bar<T>(x:&T){
   ...
}
mod foo<T> {
    struct Point3{ xT:,y:T,z:T}
}

it's always keyword item_name <type params> {...definition...}

.. but that still leaves the big problem of how to declare them at the file level, which is the really useful case (like a whole maths library being generic over FLOAT=F32 or F64, without having to write it with an additional nesting level )

now i've written that i even wonder if you would want a shortcut for making it like inserting the type-params on the individual items.. consider Vec3<f32> vs foo<f32>::Point3 above .. is the latter really a Point3 ?

[bvssvni]

I like the type Foo; syntax. It makes it very close to existing Rust code, and I would also be able to override e.g. use math::{Scalar = f32}; on existing type aliases. However, one would have to check compilation without specifying a default type.

Also, why not override traits and structs using the same technique?

mod foo {
    pub trait Foo {}
}

pub trait Bar {}

use foo::{Foo = Bar};

[bvssvni]

This mechanism might be used in Drun to prove two interfaces equivalent. All libraries that builds on some interface can be used with a new library that prove new structs/enums/traits for the interface. These libraries will then be interpreted using the new type interface in the scope.

[bvssvni]

The proof is pretty simple: If overriding types for a module passes type check, then all modules depending on it will type check when substituting for the same types. Therefore, no extra type checking is needed for the modules depending on the interface.

[bvssvni]

The type interface of a modules can be abstracted to a tree indexed over equivalence classes of types, such that every substitution detects a type error if the same equivalence class is assigned two different types in a scope.

[dobkeratops]

ahh you're talking about what to do in DRun ok. (I was thinking of proposals for rust)

[bvssvni]

I want a general mechanism for proving two interface to be equivalent, so they can be swapped with each other. This is incredibly useful since you can write e.g. a graphics API that extends an existing one with new features, then reuse all the existing code for the old API.

Instead of direct inheritance, you prove an API to be equivalent for a scope, then Drun compiles the existing code using the new substitution.

Inheritance is just a short version for "implement all existing features except for those I overload".

[bvssvni]

Consider the following:

mod my {
   fn foo<T: Foo>(a: T) -> T { ... }
}

Now, I have a trait Bar that inherits Foo:

pub trait Bar : my::Foo {}

I want to use the function foo, but I want the output type to inherit Bar instead of Foo.

So, I substitute Bar for all occurrences of Foo in the generic parameters of my::foo:

use my::foo<Foo = Bar>;

foo(a) : Bar; // Now the output type of `foo` inherits `Bar`

[dobkeratops]

right a general purpose substitution mechanism in a code generator has a lot more options and scope. (proposals for Rust mod type-params are a subject for elsewhere)

[bvssvni]

The point is that even though I write Rust code, I can use Drun to substitute Rust types in existing codebases, not needing to rewrite the whole thing to add the features I wanted.

Also, you can use the same mechanism in backends:

use opengl_graphics::GlGraphics;
use graphics::{Graphics = ScreenshotGraphics}; // Done!

pub trait ScreenshotGraphics: Graphics {
    fn screenshot(&self);
}

impl ScreenshotGraphics for GlGraphics { ... }

// `Graphics` is now equivalent to `ScreenshotGraphics`.
pub fn draw_something<G: Graphics>(c: &Context, g: &mut G) {
    clear([1.0; 4]; &mut g);
    ...
    g.screenshot(); // New method available.
}

Formally, Graphics ~> ScreenshotGraphics, meaning that whenever you refer to Graphics in the scope, you mean ScreenshotGraphics. However, ScreenshotGraphics ~> Graphics is not true, such that impl ScreenshotGraphics carries the same meaning as in normal Rust.

The docs will show draw_something<G: ScreenshotGraphics>.

[bvssvni]

When a trait inherits the one that it substitutes, then the proof is trivial that substitution is possible:

// `B ~> A` is possible, `A ~> B` is only possible if `A` adds no new stuff
pub trait A: B {...}

However, one can also use this when copying the interface:

// `A ~> B` is possible
pub trait A {fn foo();}
// `B ~> A` is possible
pub trait B {fn foo();}

This is useful when you don't want a library to depend on others unnecessary.

You can also declare new traits and implement them for existing types, then substitute old traits with news traits to use the existing types in places where they could not be used before.

[bvssvni]

Now, consider two modules:

mod foo;
mod bar;

use self::{foo = bar};

foo ~> bar holds only if every type in foo have the same name and kind in bar.

If the modules contain types that differ, one can provide the proof using the scope:

mod foo {pub trait Foo {}}
mod bar {pub trait Bar {}}

use foo::{Foo = Bar};
use self::{foo = bar}; // Now `foo ~> bar` is possible

[bvssvni]

Because of pub use, one can reexport proofs of equivalence, such that a glob import of a crate that uses proof of substitution automatically applies it to the scope.

[bvssvni]

@dobkeratops Yeah, I think of Drun as a powerful code generator language first and foremost. It needs to understand Rust code and types pretty well in order to do the transformations. However, this is much less complex than e.g. the type checking in the Rust compiler.

[dobkeratops]

I just encountered some other person's custom language (from the free node rust irc channel) ; https://www.youtube.com/watch?v=lUQpaa3bd54 these are his screencasts and this one in particular details generating shader code from his language. I think that's pretty interesting. (I hadn't asked if you do tackle and shader generation in dyon ) . just another example of how things could work. (this particular language seems to be 'lisp inspired but statically (inferred) types' .. does actually look quite nice. This is his project page (bitbucket) https://bitbucket.org/duangle/scopes/wiki/Home