Unify languages

[jacereda]

I was wondering if it would be possible to replace the dynamic language with plain Carp.

AFAICS we could have something like a compilation cache to reduce the amount of compilation for dynamic functions and an execution cache to reduce the amount of "shelling out". With that in place, I guess it could be speedy enough. What do you think?

[jacereda]

This isn't a novel idea, zig does use a compilation cache and I'm not sure, but probably does use an execution cache as well.

[eriksvedang]

Interesting idea. How would the (binary) functions modify the REPL environment?

[jacereda]

Via primitives?

[jacereda]

Let's take https://github.com/carp-lang/Carp/blob/134d9a5b02ce91c53ad61262d3758d85443239a4/core/Dynamic.carp#L56 as an example.

relative-to could be instead a defncomptime. current-file and add-c could remain dynamic functions.

The point here is we have lots of useful stuff in carp that we can't use from dynamic functions and have to end up having duplicate functionality in the dynamic world.

At a later stage, and if it proves useful, we could consider having a protocol to communicate with the comptime executables (maybe some sort of Interpreter monad) and get rid of dynamic functions completely.

[jacereda]

And looking at the issue title, it wasn't very fortunate, since what I'm proposing now is adding another mode instead of removing one :)

[hellerve]

I don‘t think I understand the proposition completely just yet: would all code then be essentially static, in the sense of a JIT-like environment rather than an interpreter (but leveraging static compilers rather than a JIT framework)?

[jacereda]

I've been thinking a bit more about this.

Let's start with the mechanism for evaluating static functions at the REPL. Let's say we want to evaluate:

(Int.+ 1 2)

I would propose separating the function from its arguments. The function Int.+ would then be compiled into an executable in $HOME/.cache/carp/fn/. The hash can be computed via Data.Hashable easily.

Now, the generated executable would need to read the arguments passed via the command line, apply the function and repr the result (I think we really need a repr, looks like the current approach is to use println*).

If we now evaluate at the REPL:

(Int.+ 10 20)

It will find the previously compiled executable and call it with the new arguments.

With that mechanism in place, I think Eval.hs could use it in most places where we error out with HasStaticCall.

Does that make sense?

[eriksvedang]

The first version of Carp (see the c branch) actually did something like this! It got very convoluted when we tried to add support for Windows (using dll:s) but I don't remember the details.

I think it sounds like an interesting idea, but I want to know more about what the primary goal is... Is it to get a faster dynamic runtime (for macros and such) or to make calling of compiler code more robust? Or neither/both..?

I'm not familiar with the term repr but I completely sympathize with the need for a better way to get results back from compiled code in the repl.

[jacereda]

I think the current mechanism doesn't make any guarantee that a value turned into a string can be converted back to the original value. That's what I meant with repr.

As for the goal, I just want to have most functions in the static side available in the dynamic side. @TimDeve was talking about adding some math operations to the dynamic side. It wouldn't be needed if we had something like this.

[eriksvedang]

Ok,that is a great reason!

And yes – there's currently no guarantee we can convert back so that would be very nice too.

[scolsen]

I think the current mechanism doesn't make any guarantee that a value turned into a string can be converted back to the original value. That's what I meant with repr.

As for the goal, I just want to have most functions in the static side available in the dynamic side. @TimDeve was talking about adding some math operations to the dynamic side. It wouldn't be needed if we had something like this.

That would be great! We'd be able to get rid of a lot of code. --Am I right in thinking we'd still be keeping the dynamic language dynamic, though? That is, even though we'd make Int.+ available in dynamic code, the call wouldn't be type checked--so the difference between the contexts really is the presence or absence of typechecking?

[hellerve]

I think the current mechanism doesn't make any guarantee that a value turned into a string can be converted back to the original value. That's what I meant with repr. As for the goal, I just want to have most functions in the static side available in the dynamic side. @TimDeve was talking about adding some math operations to the dynamic side. It wouldn't be needed if we had something like this.

That would be great! We'd be able to get rid of a lot of code. --Am I right in thinking we'd still be keeping the dynamic language dynamic, though? That is, even though we'd make Int.+ available in dynamic code, the call wouldn't be type checked--so the difference between the contexts really is the presence or absence of typechecking?

I don’t think that both taking the static language for dynamics and not having types would work together. Neither would, as of now, metaprogramming patterns, since that does use a type (S expressions, akak lists) that aren’t present in runtime Carp.

[scolsen]

I think the current mechanism doesn't make any guarantee that a value turned into a string can be converted back to the original value. That's what I meant with repr. As for the goal, I just want to have most functions in the static side available in the dynamic side. @TimDeve was talking about adding some math operations to the dynamic side. It wouldn't be needed if we had something like this.

That would be great! We'd be able to get rid of a lot of code. --Am I right in thinking we'd still be keeping the dynamic language dynamic, though? That is, even though we'd make Int.+ available in dynamic code, the call wouldn't be type checked--so the difference between the contexts really is the presence or absence of typechecking?

I don’t think that both taking the static language for dynamics and not having types would work together. Neither would, as of now, metaprogramming patterns, since that does use a type (S expressions, akak lists) that aren’t present in runtime Carp.

Right, that's a good point. I guess what I meant was we wouldn't call the static typechecker--I'm not proposing we get rid of list? or array? or anything like that. Not sure if it's feasible for us to call a static fn like Int.+ without typechecking depending on the context. But I may also just be misunderstanding the proposal here.

Related to your point though, how exactly would evaluating Int.+ in dynamic code work? I suppose that's where repr is key?

[eriksvedang]

As I understand the idea (correct me if I'm wrong @jacereda) this would mean that you could do:

(defndynamic f [x y]
  (Int.+ x y))

and thus reuse the Int.+ implementation within a dynamic function. f would still be dynamically typed, to the extent that any dynamic numbers would work as its arguments (since we would pull out the contents of the XObj Num and pass them to the external Int.+ function.)

[jacereda]

That's the idea, yes.

Another approach with minor modifications to the haskell side, could be to just have a compile-time macro

(def sin-tab (compile-time generate-sin-tab 1024))

The macro would need to build an executable for the generate-sin-tab function, save it with the hash of that function as part of the file name so it doesn't need to be rebuilt next time (#1069), shell out via run-with-arguments passing the argument and read back the result.

The generated main for the executable can parse the argument via Int.from-string (should be possible to generate that, since generate-sin-tab is a static function whose type is known).

Sounds feasible?