Birectional conversion bewteen Julia Expr and sexpr representation in this package?

[thautwarm]

This is really beneficial because currently Julia Expr is very difficult to analyze when we want to do advanced codegen.

For instance, @tkf have to use JuliaVariables.jl to achieve the scope analysis in FLoops.jl, which certainly causes high latency and I'm very sorry for that.

I also have use cases for having a simpler program representation. I'm trying to optimize pattern maching code generation in MLStyle.jl, and I need to identify the outer referenced symbols in a Julia Expr, which is now extremely disturbing and difficult if I don't introduce JuliaVariables.jl in the project.

As you're already starting this sort of awesome works, if possible I feel like to give my concerns and hope it can be considered in the roadmap https://github.com/TotalVerb/SExpressions.jl/issues/3 .

IMO supporting whole racket syntax and features wouldn't help programming in Julia, maybe you can export a lot of racket libraries in Julia but the work is kind of "fancy".

[tkf]

This is very off-topic, and you might have already noticed, but, FYI, there is a series of work towards re-writing julia front-end in Julia by @c42f. Like CSTParser.jl can now be hooked into julia parser https://github.com/JuliaLang/julia/pull/35243, https://github.com/JuliaLang/FancyDiagnostics.jl/pull/4, I wonder some of the machinery in MLStyle.jl ecosystem can be used to implement the lowering. For example, I'd imagine you'd need something like JuliaVariables.jl for implementing the lowering of closures. Also there is a similar discussion on s-exp vs Expr in https://github.com/JuliaLang/julia/pull/32201 (esp https://github.com/JuliaLang/julia/pull/32201#issuecomment-500068466).

BTW, I've never noticed any overhead in FLoops.jl :) though maybe I haven't payed enough attention to it.

[thautwarm]

Yes, you're right, it's quite off-topic. Sorry for this.

I'm actually thinking about a simpler program representation for easier analysis, instead of only a front-end problem or implementing closures. However this is quite a restricted field, and I shall not ask others to focus on this..

Like CSTParser.jl can now be hooked into julia parser JuliaLang/julia#35243, JuliaLang/FancyDiagnostics.jl#4, I wonder some of the machinery in MLStyle.jl ecosystem can be used to implement the lowering. For example, I'd imagine you'd need something like JuliaVariables.jl for implementing the lowering of closures.

I don't see what's concerning lowering in JuliaLang/FancyDiagnostics.jl .. In MLstyle my use case is simpler: I'm not implementing closures, but identifying outer symbols to decide which symbol to mangle.

BTW, I've never noticed any overhead in FLoops.jl :) though maybe I haven't payed enough attention to it.

That's not the execution overhead but startup latency.. I mean the first time to load programs, but if you're using Revise this will not become a problem.

[tkf]

Oh, sorry, I was meant to say "My comment would be quite off-topic." It was not my intention to suggest closing this issue. :sweat_smile:

lowering in JuliaLang/FancyDiagnostics.jl

I don't think FancyDiagnostics.jl would need lowering (at least for now?). I was meant to say that there is a big project for re-writing parsing and maybe also lowering in Julia (FancyDiagnostics.jl is just a part of it). If that happens, I thought something like JuliaVariables.jl would play the role of CSTParser.jl (but in lowering, not parsing).

[thautwarm]

@tkf Sorry, I'm very sensitive in communition things and I'm afaird of annoying people.. That's why I closed this issue after I misunderstood your reply..

I was meant to say that there is a big project for re-writing parsing and maybe also lowering in Julia (FancyDiagnostics.jl is just a part of it). If that happens, I thought something like JuliaVariables.jl would play the role of CSTParser.jl (but in lowering, not parsing).

Oh... I totally misunderstood your words, I read your links for quite a long time and didn't find any scoping analysis/general program analysis stuffs. I felt kind of upset for this, and I just thought, it's my lack of abilities and knowledges to catch up the discussions about Julia internals..

Thanks for explaining this to me, just now I feel quite sad but suddenly turn pleased!

[c42f]

Yes, we'll eventually need some julia implementation of lowering. I did a lot of the "simplify expressions" / "desugaring" part a while ago in https://github.com/JuliaLang/julia/pull/32201 but there is a lot of lowering which wasn't implemented there (including scope analysis).

Alas, I'm quite sad to say that my project to rewrite the frontend was cancelled (it was a small part of a much larger project outside our control.) I plan to release a modernized version of FancyDiagnostics though and hopefully have time to maintain it. I hope to be able to come back the larger project later.

[thautwarm]

@c42f Thanks for your works on this field. When you plan to come back to the larger project, if the human resources are limited please contact me. I'm very addicted to this sort of works and, I think I have some essential experience with lowering stuffs or general compiler stuffs.