Add `define` macro with first argument treated as `this` to enable chaining

[dadhi]

I want to be able chain the output produced by define macro, e.g.

define AppendCsv($strBuilder, $list, $itemStr) {
     for (var i = 0; i < $list.Count, ++i)
         (i == 0 ? strBuilder : strBuilder.Append(", "))
             .Append($itemStr($list[i]));
     return $strBuilder;
}

string Curly(string x) => "{"+x+"}";
s.Append("(").AppendCsv(args, Curly).Append(")");

[qwertie]

This, in fact, would require changes to LeMP's engine, not just the define macro.

The problem is that a.b(c) essentially means (a.b)(c) and although macros can register themselves with the . operator to trigger on a.b, they cannot trigger on the parent expression a.b(c).

Making this more difficult, you want to be able to use a for loop inside an expression. Luckily I have already implemented limited support for this via the #ecs macro in combination with #runSequence... but it doesn't work in all cases (plain-old C# doesn't support it, and LeMP doesn't have access to type information, occasionally leading to awkward situations where statements-in-expressions can't be converted to C#.)

[dadhi]

@qwertie Thanks for the answer.

Do you have the docs, or samples, or tests for all macros available.. or maybe you can suggest the way how to search for them?

[qwertie]

The docs are here, each macro documented with a simple example. Hmm, I wonder if I forgot to link to any important macros from that main page... in any case, at the bottom you'll see 5 pages of documentation, and every macro should be mentioned on one of those 5 pages. Also, you can ask LeMP itself what macros it knows about (with brief documentation) by typing #printKnownMacros;.

[qwertie]

Note that Loyc APIs such as LNode are documented separately here. Here is a guide to using Loyc trees.

[dadhi]

@qwertie Really appreciate the docs. Personally, I always struggle to write them.

printKnownMacros;

Cool!

[dadhi]

@qwertie

I found the tests for #runSequence here https://github.com/qwertie/ecsharp/blob/1743316eeb4ad57daba57fdc1e8dd5d2a47ebe98/Main/LeMP/Tests/TestSequenceExpressionMacro.cs

but still failing to understand when is fine to apply it... maybe missing something crucial here.

Meanwhile, I am experimenting with this kind of define operator where $s is a StringBuilder:

    define operator<<($s, $x) => $s.Append($x);

    define operator>>($s, ($items, $itemStr))
    {
        for (var i = 0; i < $items.Count; ++i)
            (i == 0 ? $s : $s << ", ") << $itemStr(its[i]);
        $s;
    }

Calling it like so

s << '(' >> (n.Arguments, ExpressionToCSharp) << ')';

produces kind of cryptic error:

CompileTimeDI.ecs(66,1,173,2): error : LeMP.StandardMacros.compileTime: (36,5): error CS0103: The name '_aposx2E' does not exist in the current context - in <    @_aposx2E(_numfor(_num(i), i < args.Count, _num(++i), > [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]

Btw, do you support define overloading/matching based on the input pattern, I want both to be <<. Tried, but got and Ambiguous macros error.

[qwertie]

Um... I guess you're going for a C++-style cout << foo thing here, but in that case why mix << and >>? And what is its intended to mean?

_aposx2E represents '. and refers to the dot operator (I'll change it to output _apos_period in the next version so it's easier to see the connection with '.), while _numfor refers to a for loop. It looks like in the macro expansion, the left-hand side of the dot operator is a for loop, which can't be translated sensibly to C#, but the printer is trying its best to do a conversion anyway.

[dadhi]

but in that case why mix << and >>

I am using it just to distinguish from << to prevent ambiguous macros error. I would love to use << for the loop operation as well.

_aposx2E represents '. and refers to the dot operator (I'll change it to output _apos_period in the next version so it's easier to see the connection with '.)

Thanks.

My question is specifically about the define below - ignore the actual operator name selected

    define operator>>($s, ($items, $itemStr))
    {
        for (var i = 0; i < $items.Count; ++i)
            (i == 0 ? $s : $s << ", ") << $itemStr(its[i]);
        $s; // How to return a thing from macro?
    }

How can I combine for loop and "returning" something from define. I have looked at #runSequence tests as examples but I am not sure how to apply it in this case.

[qwertie]

define defines a macro to replaces one syntactic pattern with another, so if you write define foo($x) { while($x<10) $x++; } and then result = foo(xyz);, you'll get a nonsense result conceptually like result = (while ($x < 10) x++;), but it's actually printed as result = _numwhile(xyz < 10, xyz++);.

If you intend to use an operator in a C# expression, it must itself be an expression, but since the output of define is defined with a braced block, the expression must be written in statement notation (with a semicolon!):

define MulDiv($a, $b, $c) { ($a * $b) + $c; }
result = MulDiv(x, y, z);

// output
result = (x * y) + z;

The #ecs and #runSequence macros can often be used to work around this. I'm actually surprised that #runSequence {...} doesn't work, I think it's an oversight on my part.

There's a workaround as shown here:

#ecs;

define runSequence({ $(..args); }) {
    #runSequence($args);
}
define MakeItBig($x) {
    runSequence {
        while ($x < 1_000_000)
            $x *= 2;
        $x;
    }
}
void fn() {
    int x = 5;
    Console.WriteLine(MakeItBig(x));
}

// Generated from Untitled.ecs by LeMP 2.8.2.0.
void fn() {
    int x = 5;
    while (x < 1000000)
        x *= 2;
    Console.WriteLine(x);
}

Now, the behavior of #runSequence depends on context - if you write double x = #runSequence(Console.WriteLine(), Math.PI) at class scope, it employs a different transformation than if you use it at function scope, or rather, that's what it's supposed to do. Shockingly, #runSequence isn't processing double x = #runSequence(Console.WriteLine(), Math.PI); at all inside a function right now. That's a bug.

Unfortunately, if you are not inside a function nor a class, #runSequence doesn't work at all and ignores your request. Notably, compileTime and precompute are not contexts recognized by #runSequence at the present time.

So my advice is not to use #runSequence right now - If you are tempted to run statements inside define(), I recommend defining an ordinary function instead. Of course, you can also define an operator transformation that calls the ordinary function.

// MakeItBig as an ordinary function - based on my Loyc.Math generic numerics library
class Methods {
    public static T MakeItBig<T>(ref T x) {
        var m = Maths<T>.Math;
        while (m.IsLess(x < m.From(1000000)))
            x = m.Mul(x, m.From(2));
        return x;
    }
}

Finally, if you need to give one macro priority over another, you can:

define operator<<($a, $b) {
    $a.Append($b);
}
[PriorityOverride, Passive]
define operator<<($a, ($x, $y)) {
    $a.Append($x + $y); // or whatever
}

sb << foo << (a, b);

Alternately, use [PriorityFallback] to give one definition lower priority than another. However I would personally avoid defining that first macro because it completely replaces the original << operator, which can no longer be used.

[dadhi]

@qwertie Thanks for the detailed answer. Will wait for the fixes.

Quick question, can I use the define macro of the two arguments (patterns) as a binary operator? Asking because it would be great for solving the fluent chain of opetations without overriding the "standard" operators. Also I saw in the code somewhere the postfix `fooBar` notation.

[dadhi]

@qwertie Hi again,

Following the recent release of Source Generators I see them now as a niche solution with they own problems (including the maturity one).. So that returns me back to LeMP :)

Did you have a chance to look at this thing

Shockingly, #runSequence isn't processing double x = #runSequence(Console.WriteLine(), Math.PI); at all inside a function right now. That's a bug.

And sorry for bothering.

[qwertie]

Thanks for the anti-procrastination message. I fixed the bug, which turned out to be specific to statements of the form var x = #runSequence(...). The existence of this bug suggests that there are other rough edges waiting to be found, so I encourage you to go looking for them. Are you using the LeMP-tool package? Because I've just published LeMP to NuGet as v28.2.1 v28.3, but I haven't done to the extra step of publishing a new version of the Visual Studio extension.

Edit: I changed the version to 28.3, as there were several changes, not just bugfixes, since the last release. I will prepare a GitHub release of the VS extension.

[dadhi]

@qwertie, I am happy to help you :) Again I think, that you've develeped an unique product and the approach with the Ecs and LeMP in the .NET landscape - the real Alt .NET, it is by itself has its value. Hope it will boost your morale bit.

I am using the cli tool package, but extension is relevant later for someone else who may consume my code from the VS.

I will check and play with the fixes soon.

Btw, I wonder how LeMP will take on the really big source file, like the 10k loc. What is your take on the tool performance in general?

[qwertie]

I've made reasonable efforts to optimize LeMP - premature optimization, I suppose - but no large codebases exist on which to verify its speed. I expect the main C# compilation step to take longer than LeMP, at least if you're not using compileTime (which of course has the overhead of compiling and running C#) or LLLPG (whose performance is not so great because LL(k) is difficult to do efficiently).

[dadhi]

@qwertie Hello,

I have updated the tool to 28.3.0 and uncommented this line:

`// s << '(' << (n.Arguments, ExpressionToCSharp) << ')';

see https://github.com/dadhi/LempTest/blob/8a4c943ceae43efddc5b1ff4fe099e26dcacd409/CompileTimeDI/CompileTimeDI.ecs#L122)

which uses the defines like these:

    define operator<<($s, $x) => $s.Append($x);

    define runSequence({ $(..args); }) {
        #runSequence($args);
    }

    // todo: @wip does not work until esharp #121 is fixed
    [PriorityOverride, Passive]
    define operator<<($s, ($items, $itemStr)) {
        runSequence {
            for (var i = 0; i < $items::it.Count; ++i)
                (i > 0 ? $s << ", " : $s) << $itemStr(it[i]);
            $s;
        }
    }

I am getting this error:

I've played with the different variations of define operator<<($s, ($items, $itemStr)) but no luck.

[qwertie]

Actually, after commenting out some lines from "ServiceRegistrations.ecs.include", your code is processed without error on my machine (I'm using LeMPDemo.exe to test). I am having a hard time understanding the error messages you got; I am guessing that MSBuild is adding the suffix [in square brackets] mentioning the .csproj file, but I can't think of a reason why it says "EXEC" at the beginning. You cut off at least one error message and I'm curious what it says.

[dadhi]

@qwertie Hello. I am checking this again and still have the errors.

Again the macro looks like this:

    [PriorityOverride, Passive]
    define operator<<($s, ($items, $itemStr)) {
        runSequence {
            for (var i = 0; i < $items::it.Count; ++i)
                (i > 0 ? $s << ", " : $s) << $itemStr(it[i]);
            $s;
        }
    }

Case 1

When invoking the macros like this:

s << '(' << (n.Arguments, ExpressionToCSharp) << ')';

the error:

 LeMP macro compiler (2.8.3.0)
CompileTimeDI.ecs(11,5,11,29): warning : LeMP.StandardMacros.compileTimeAndRuntime: The C# scripting engine does not support namespaces. They will be ignored when running at compile time. [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]
EXEC(1,24): error : LeMP.StandardMacros.compileTime: Reached end-of-file before '(' was closed [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]
EXEC(1,14): error : LeMP.StandardMacros.compileTime: Reached end-of-file before '(' was closed [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]
EXEC(1,6): error : LeMP.StandardMacros.compileTime: Reached end-of-file before '(' was closed [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]
  (1,6): Critical: LeMP.StandardMacros.compileTime: Bug: unhandled exception in parser - Input string was not in a correct format. (FormatException)
CompileTimeDI.ecs(67,1,182,2): error : LeMP.StandardMacros.compileTime: ParseSingle: result was empty. (InvalidOperationException) [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]
  CompileTimeDI.ecs(67,1,182,2): LeMP.StandardMacros.compileTime: Stack trace:    at Loyc.Syntax.ParsingService.Single(IListSource`1 e) in C:\projects\ecsharp\Core\Loyc.Syntax\IParsingService.cs:line 280
     at Submission#5.Parse(String expr) in :line 71
     at Submission#5.<<Initialize>>b__0_0(KeyValuePair`2 r, Int32 i) in :line 11
     at System.Linq.Enumerable.SelectIterator[TSource,TResult](IEnumerable`1 source, Func`3 selector)+MoveNext()
     at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
     at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
     at Submission#5.<<Initialize>>d__0.MoveNext() in :line 10
  --- End of stack trace from previous location where exception was thrown ---
     at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)
     at Microsoft.CodeAnalysis.Scripting.Script`1.RunSubmissionsAsync(ScriptExecutionState executionState, ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, Func`2 catchExceptionOpt, CancellationToken cancellationToken)

Case 2

When invoking like this (removing the s << '(' and s << ')' from the equitation):

s << (n.Arguments, ExpressionToCSharp);

the error is different:

 LeMP macro compiler (2.8.3.0)
CompileTimeDI.ecs(11,5,11,29): warning : LeMP.StandardMacros.compileTimeAndRuntime: The C# scripting engine does not support namespaces. They will be ignored when running at compile time. [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]
CompileTimeDI.ecs(67,1,184,2): error : LeMP.StandardMacros.compileTime: (44,5): error CS0201: Only assignment, call, increment, decrement, await, and new object expressions can be used as a statement - in <    s;> [C:\Code\LempTemplatingTest\CompileTimeDI\CompileTimeDI.csproj]

[qwertie]

I'm still confused. Normally if an error occurs during a Roslyn scripting invocation, LeMP will print a warning message that refers to a temporary file in which you can view the code that was passed to the scripting engine. Are you not seeing a warning like that?

[dadhi]

@qwertie

Hi again. I have extracted the command to run the LeMP into the separate lemp_test.bat and running it in my project CompileTimeDI. The bat contains a command: lemp CompileTimeDI.ecs --outext=.Generated.cs --o-indent-spaces=4 --o-omit-comments=false --set:ProjectDir=@C:/Code/LempTemplatingTest/CompileTimeDI

It runs fine and produces the correct output in the CompileTimeDI.Generated.cs until I uncomment the line

s << '(' << (n.Arguments, ExpressionToCSharp) << ')';

and comment the line with the method call

// AppendCsv(s << '(', n.Arguments, ExpressionToCSharp) << ')';

The error:

C:\Code\LempTemplatingTest\CompileTimeDI>lemp CompileTimeDI.ecs --outext=.Generated.cs --o-indent-spaces=4 --o-omit-comments=false --set:ProjectDir=@C:/Code/LempTemplatingTest/CompileTimeDI
LeMP macro compiler (2.8.3.0)
CompileTimeDI.ecs(11,5,11,29): Warning: LeMP.StandardMacros.compileTimeAndRuntime: The C# scripting engine does not support namespaces. They will be ignored when running at compile time.
(1,24): Error: LeMP.StandardMacros.compileTime: Reached end-of-file before '(' was closed
(1,14): Error: LeMP.StandardMacros.compileTime: Reached end-of-file before '(' was closed
(1,6): Error: LeMP.StandardMacros.compileTime: Reached end-of-file before '(' was closed
(1,6): Critical: LeMP.StandardMacros.compileTime: Bug: unhandled exception in parser - Input string was not in a correct format. (FormatException)
CompileTimeDI.ecs(67,1,179,2): Error: LeMP.StandardMacros.compileTime: ParseSingle: result was empty. (InvalidOperationException)
CompileTimeDI.ecs(67,1,179,2): LeMP.StandardMacros.compileTime: Stack trace:    at Loyc.Syntax.ParsingService.Single(IListSource`1 e) in C:\projects\ecsharp\Core\Loyc.Syntax\IParsingService.cs:line 280
   at Submission#5.Parse(String expr) in :line 68
   at Submission#5.<<Initialize>>b__0_0(KeyValuePair`2 r, Int32 i) in :line 11
   at System.Linq.Enumerable.SelectIterator[TSource,TResult](IEnumerable`1 source, Func`3 selector)+MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Submission#5.<<Initialize>>d__0.MoveNext() in :line 10
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Scripting.Script`1.RunSubmissionsAsync(ScriptExecutionState executionState, ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, Func`2 catchExceptionOpt, CancellationToken cancellationToken)
Session log at C:\Users\Maxim Volkov\AppData\Local\Temp\tmpE88C.tmp contains emitted code.

[qwertie]

Could you modify your "ServiceRegistrations.ecs.include" file, as I did, and check whether the errors go away?

compileTime 
{
    // NOTE: This is how you can reference the assembly with your types
    //##reference(precompute(System.IO.Path.Combine(#get(ProjectDir), @"..\AnotherLib\bin\Debug\netstandard2.0\AnotherLib.dll")));

    // Variables available via `#get(varName)`:
    // - #get(ProjectDir))
    // - #get(OutDir))
    // - #get(OutDirFullPath))

    // NOTE: Add the usings for your services here
    //using AnotherLib;
    //using AnotherLib.Experimental;

    // NOTE: Put your registrations here
    public static DI ConfigureDI() 
    {
        var di = new DI();

        // NOTE: Example resgitrations:
        //di.Register<A>(r => new A());
        //di.Register<B>(r => new B());
        //di.Register<X>(r => new X(new A(), new B(), r.Resolve<Y>()));

        return di;
    }
}

Edit: pls ignore my previous edit if you saw it

[dadhi]

@qwertie Yes, it works but it does not call the macro with the #runSequence - that's why. The call to operator<<($s, ($items, $itemStr)) is what causing the problem.

[dadhi]

@qwertie Ok, for the test purposes I have added the Test.ecs (https://github.com/dadhi/LempTest/blob/master/CompileTimeDI/Test.ecs)

The first time LeMP complained to me to add the #useSequenceExpressions; and then it is worked out!

I've also added it into the CompileTimeDI.ecs but without success :( The error did not change. So it seems like the combination of things - though the main problem is that error is unhelpful.

[dadhi]

@qwertie Added example into Test.ecs of trying to use macro inside the compileTime and it does not work:

compileTime
{
    #ecs;
    using System (.Text, );
    using Loyc (.Syntax, .Ecs, );

    define operator<<($s, $x) => $s.Append($x);

    define runSequence({ $(..args); }) {
        #runSequence($args);
    }

    // todo: @wip does not work until esharp #121 is fixed
    [PriorityOverride, Passive]
    define operator<<($s, ($items, $itemStr)) {
        runSequence {
            for (var i = 0; i < $items::it.Count; ++i)
                (i > 0 ? $s << ", " : $s) << $itemStr(it[i]);
            $s;
        }
    }

    string Ack(string s) => s + "!";
    var names = new[] { "Holly", "Molly" };
    var b = new StringBuilder();
    // b << "Hello"  // This works!
    b << "Hello " << (names, Ack) << " here"; // todo: #121- does not work
    var expectedHello = quote { string hello = precompute(b.ToString()); }; 
}

#useSequenceExpressions; // note: required and need to be put exactly here

using System.Text;

static class X
{
    precompute(expectedHello);

    public static string Ack(string s) => s + "!";
    public static steing SayHello(StringBuilder b)
    {
        var names = new[] { "Holly", "Molly" };
        b << "Hello " << (names, Ack) << " here";
        return b.ToString();
    }
}

[qwertie]

In this new example, the problem is that #runSequence only works inside methods. I have been meaning to change this, especially since C# 9 now allows top-level statements. So I will make it work in the next release.

The reason for the original behavior is that #runSequence must behave differently at class scope in order to correctly translate a statement like T x = #runSequence(...), and since classes can't appear inside methods the easiest thing to do was not pay attention to classes/structs but rather "assume we're in a class until we see a method".

Edit: also it seems reasonable to support #runSequence {...} so you won't need that extra macro.

[qwertie]

Er... I can see that your macro isn't going to do what you want.

define operator<<($s, ($items, $itemStr)) {
    #runSequence {
        for (int i = 0; i < $items::it.Count; ++i)
            (i > 0 ? $s << "", "" : $s) << $itemStr(it[i]);
        $s;
    }
}

in the context of sb << "..." << (..., ...), $s means b.Append("...") so you can only run it once, but the for-loop runs it multiple times. On top of that there's a bug, it seems :: isn't working in the context of the second or third clause of a for-loop. Edit: oops, it's not a bug and it is not related to for-loops. The problem is that :: is the scope resolution operator in standard C# (see extern alias) and it doubles as the quick-binding operator. Since LeMP acts on the syntax tree and is unaware of semantics, if you write foo::bar it can't tell whether there is a extern alias that refers to foo or not. Instead it assumes that lowercase names like foo are extern aliases while uppercase names and complex expressions like foo() are not. To avoid this ambiguity you can use the =: operator instead of ::. =: is the mirror image of := and both of them create variables:

var1 := "hello";
"hello"=:var2; // =: has high precedence, like ::

Edit: aside from that, if you write sb << a << (b, c) << (d, e), two variables both named it will be defined, so that won't work either.

[dadhi]

@qwertie Thanks for looking, will try it out.

[qwertie]

Okay, I made your original request possible and I think the prerelease can do it:

define ($obj.AppendCsv($list, $itemToStr)) {
    DoSomethingWith($obj, $list, $itemToStr);
}
string Curly(string x) => "{"+x+"}";
s.Append("(").AppendCsv(args, Curly).Append(")");

// Generated from Untitled.ecs by LeMP 2.9.0.1.
string Curly(string x) => "{" + x + "}";
DoSomethingWith(s.Append("("), args, Curly).Append(")");

In order for your original idea to work, you need a uniquely-named temporary variable, though - which is supported in a post-prerelease commit:

#ecs;

define ($obj.AppendCsv($list, $itemToStr)) {
    #runSequence {
        var temp# = $obj;
        for (var i = 0; i < $list.Count; ++i)
            (i == 0 ? temp# : temp#.Append(", "))
                .Append($itemToStr($list[i]));
        temp#;
    }
}

string Curly(string x) => "{"+x+"}";
s.Append("(").AppendCsv(args, Curly).Append(")");

// Generated from Untitled.ecs by LeMP 2.9.0.1.

string Curly(string x) => "{" + x + "}";
var temp10 = s.Append("(");
for (var i = 0; i < args.Count; ++i)
    (i == 0 ? temp10 : temp10.Append(", "))
    .Append(Curly(args[i]));
temp10.Append(")");

[dadhi]

In order for your original idea to work, you need a uniquely-named temporary variable, though - which is supported in a post-prerelease commit

Great, it is better and better.

[dadhi]

I can confirm that the fix is working. Now I can have an operator like this:

    [PriorityOverride, Passive]
    define operator<<($s, ($items, $itemStr)) {
        #runSequence {
            var temp# = $s;
            for (var i = 0; i < $items.Count; ++i)
                (i == 0 ? temp# : temp#.Append(", "))
                    .Append($itemStr($items[i]));
            temp#;
        }
    }