X# preprocessor doesn't understand nested #translate

[DenGhostYY]

Describe the bug X# preprocessor doesn't understand nested #translate

To Reproduce

#xtranslate __xlangext_SuppressNoInit(<h> [, <t>]) => (<h> := <h>[, <t> := <t>])
#xtranslate __xlangext_Apply(<f>, <h> [, <t>]) => <f><h>[, <f><t>]
#xtranslate __xlangext_Last(<h>) => <h>
#xtranslate __xlangext_Last(<h>, <t,...>) => __xlangext_Last(<t>)
//
#xtranslate local {<args,...>} := <expr>;
    =>;
    local <args> := ({<args>} := <expr>, __xlangext_Last(<args>))

#xtranslate {<args,...>} := <expr>;
    =>;
    (__xlangext_SuppressNoInit(<args>),;
    xlangext_AssignTuple(<expr>, __xlangext_Apply(@, <args>)))

local {a, b, c} := _Get()
{a, b, c} := _Get()

Expected behavior (xBase ppo)

local a,b,c := (((a := a,  b :=  b,  c :=  c),xlangext_AssignTuple(_Get(), @a,  @ b,  @ c)), c)
((a := a,  b :=  b,  c :=  c),xlangext_AssignTuple(_Get(), @a,  @ b,  @ c))

Actual behavior (X# ppo)

local a, b, c := ((__xlangext_SuppressNoInit(a, b, c  ), xlangext_AssignTuple(_Get() , __xlangext_Apply(@, a, b, c  ))) , c)
(__xlangext_SuppressNoInit(a, b, c ), xlangext_AssignTuple(_Get() , __xlangext_Apply(@, a, b, c )))

Error message error XS9111: The error is most likely related to the token 'local' that was used at this location. error XS9002: Parser: unexpected input ','

Additional context X# Compiler version 2.14.0.4 (release)

[DenGhostYY]

What is the difference between _GETMPARAM() and _GETFPARAM()?https://www.xsharp.eu/help/pseudo-functions.html Is there an analogue of pseudo-function PValue(<nPosition>, [xValue])? Example:

procedure xlangext_AssignTuple(src)
    local i
    local len

    len := min(len(src), pCount() - 1)
    for i := 1 to len
        PValue(i + 1, src[i])
    next
return

[RobertvanderHulst]

What is the difference between _GETMPARAM() and _GETFPARAM()?https://www.xsharp.eu/help/pseudo-functions.html Is there an analogue of pseudo-function PValue(<nPosition>, [xValue])? Example:

procedure xlangext_AssignTuple(src)
    local i
    local len

    len := min(len(src), pCount() - 1)
    for i := 1 to len
        PValue(i + 1, src[i])
    next
return

There is no difference. In Visual Objects there was (_GetMParam() was for methods, _GetFParam() was for functions). The reason for that was that these functions were accessing the parameters on the stack and _GetMParam() "knows" that there is a SELF value on the stack for a method call. Inside X# the parameters for Clipper Calling Convention are passed as an array of usual values. Both function access the elements of this array. And PCount() is mapped to the length of that array.

[RobertvanderHulst]

W.r.t. the #xtranslate bug: We will have a look at this. Are you trying to implement tuple support? We are planning to add something like that in the cause of this year. We will probably not use curly braces in our syntax but parentheses, just like C. Something like this:

local (a, b, c) := _Get()
(a, b, c) := _Get()

FUNCTION _Get as (INT,INT,INT)
   RETURN (28,02,2023)

[DenGhostYY]

Yes, we actively use tuples in XBase++ sources. It would be nice to move them to the X#.

What about the pseudo-function PValue(<nPosition>, [xValue])?

[RobertvanderHulst]

PValue(<nPosition>)is a synonym for _GetMParam() and _GetFParam(). We did not realize that XBase++ has that function. We can add that as pseudo function to the compiler (for the Xbase++ dialect at least) fairly easy. Implementing the second parameter for existing parameters is a bit more complex. We have implemented clipper calling convention by passing a (Param) array of usuals. In the compiler generated startup code we create a local variable for each of the declared parameters and assign either the value of the argument that was passed to the local, or NIL. If parameters are updated in the code then we write these values back to the array at the end of the method or function. To do that we keep track of which of the parameters was updated in the code. Implementing the second parameter for PValue() would mean that we update:

• the array element in the params array
• but also the local variable that was declared for the parameter

That will be more work. How important is that to you?

And what should the compiler do when the second parameter is passed and nPosition is greater than the # of values passed to the function?

[DenGhostYY]

I consulted with colleagues, and we decided not to waste your time on this ticket. We will rewrite PValue() ourselves, including using _ARGS(). But it would still be nice to do recursive rule processing.

[RobertvanderHulst]

I analyzed your UDC and found a problem in the UDC that causes this to fail. The first translate rule is #xtranslate __xlangext_SuppressNoInit(<h> [, <t>]) => (<h> := <h>[, <t> := <t>]) But it gets matched with __xlangext_SuppressNoInit(a,b,c) That fails because the rule only covers 2 tokens If you write a rule like this #xtranslate __xlangext_SuppressNoInit(<h1> [, <hn>]) => (<h1> := <h1>[, <hn> := <hn>]) Then the preprocessor "knows" that the optional token can match more than one token. The same is true for the second rule #xtranslate __xlangext_Apply(<f>, <h> [, <t>]) => <f><h>[, <f><t>] This should also be written as #xtranslate __xlangext_Apply(<f>, <h1> [, <hn>]) => <f><h1>[, <f><hn>]

After changing this, the preprocessor (at least the one in the current development build translates local {a, b, c} := _Get() to local a, b, c := (( (a := a, b := b, c := c), xlangext_AssignTuple(_Get(), @a, @b, @c)), c) and {a, b, c} := _Get() to ( (a := a, b := b, c := c), xlangext_AssignTuple(_Get(), @a, @b, @c))

I am not sure why the XBase++ preprocessor handles this differently than the Clipper and Harbour preprocessor, but it also accepts the repeated tokens that I suggest that you use.

[cpyrgas]

After making the changes in the directives described in https://github.com/X-Sharp/XSharpPublic/issues/1213#issuecomment-1719106370, compiling the new code results to a compiler crash:

Unhandled Exception: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: token index -1 out of range 0..281
   at LanguageService.SyntaxTree.BufferedTokenStream.Get(Int32 i)
   at LanguageService.CodeAnalysis.XSharp.XSharpSyntaxTree.GetXNodeSpan(TextSpan span)
   at LanguageService.CodeAnalysis.XSharp.XSharpSyntaxTree.GetLineSpan(TextSpan span, CancellationToken cancellationToken)
   at LanguageService.CodeAnalysis.SourceLocation.GetLineSpan()
   at LanguageService.CodeAnalysis.DiagnosticFormatter.Format(Diagnostic diagnostic, IFormatProvider formatter)
   at LanguageService.CodeAnalysis.CommonCompiler.PrintError(Diagnostic diagnostic, TextWriter consoleOutput)
   at LanguageService.CodeAnalysis.CommonCompiler.<ReportDiagnostics>g__reportDiagnostic|53_0(Diagnostic diag, SuppressionInfo suppressionInfo, <>c__DisplayClass53_0& )
   at LanguageService.CodeAnalysis.CommonCompiler.ReportDiagnostics(IEnumerable`1 diagnostics, TextWriter consoleOutput, ErrorLogger errorLoggerOpt, Compilation compilation)
   at LanguageService.CodeAnalysis.CommonCompiler.RunCore(TextWriter consoleOutput, ErrorLogger errorLogger, CancellationToken cancellationToken)
   at LanguageService.CodeAnalysis.CommonCompiler.Run(TextWriter consoleOutput, CancellationToken cancellationToken)
   at LanguageService.CodeAnalysis.XSharp.CommandLine.Xsc.<>c__DisplayClass1_0.<Run>b__0(TextWriter tw) in C:\XSharp\Dev\src\Compiler\src\Compiler\xsc\Xsc.cs:line 49
   at LanguageService.CodeAnalysis.CommandLine.ConsoleUtil.RunWithUtf8Output[T](Func`2 func) in C:\XSharp\Dev\src\Roslyn\Src\Compilers\Core\CommandLine\ConsoleUtil.cs:line 26
   at LanguageService.CodeAnalysis.XSharp.CommandLine.Xsc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader) in C:\XSharp\Dev\src\Compiler\src\Compiler\xsc\Xsc.cs:line 51
   at LanguageService.CodeAnalysis.CommandLine.BuildClient.RunLocalCompilation(String[] arguments, BuildPaths buildPaths, TextWriter textWriter) in C:\XSharp\Dev\src\Roslyn\Src\Compilers\Shared\BuildClient.cs:line 213
   at LanguageService.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter, String pipeName) in C:\XSharp\Dev\src\Roslyn\Src\Compilers\Shared\BuildClient.cs:line 154
   at LanguageService.CodeAnalysis.CommandLine.BuildClient.Run(IEnumerable`1 arguments, RequestLanguage language, CompileFunc compileFunc, ICompilerServerLogger logger) in C:\XSharp\Dev\src\Roslyn\Src\Compilers\Shared\BuildClient.cs:line 98
   at LanguageService.CodeAnalysis.XSharp.CommandLine.Program.MainCore(String[] args) in C:\XSharp\Dev\src\Compiler\src\Compiler\xsc\Program.cs:line 37
   at LanguageService.CodeAnalysis.XSharp.CommandLine.Program.Main(String[] args) in C:\XSharp\Dev\src\Compiler\src\Compiler\xsc\Program.cs:line 19

#xtranslate __xlangext_SuppressNoInit(<h1> [, <hn>]) => (<h1> := <h1>[, <hn> := <hn>])
#xtranslate __xlangext_Apply(<f>, <h1> [, <hn>]) => <f><h1>[, <f><hn>]
#xtranslate __xlangext_Last(<h>) => <h>
#xtranslate __xlangext_Last(<h>, <t,...>) => __xlangext_Last(<t>)
//
#xtranslate LOCAL {<args,...>} := <expr>;
    =>;
    LOCAL <args> := ({<args>} := <expr>, __xlangext_Last(<args>))

#xtranslate {<args,...>} := <expr>;
    =>;
    (__xlangext_SuppressNoInit(<args>),;
    xlangext_AssignTuple(<expr>, __xlangext_Apply(@, <args>)))

FUNCTION Start() AS VOID
LOCAL {a, b, c} := _Get()
{a, b, c} := _Get()

[RobertvanderHulst]

The crash has nothing to do with the UDC. If you use the compiler option /nostddefs then the error is gone. It has to do with the parenthesized expressions that are part of this code on several places. Like here:

( (a := a, b := b, c := c), xlangext_AssignTuple(_Get(),  @a, @b, @c))

There is no easy way to emulate expression lists which return the value of the last expression in C#. So I am generating a local function that takes care of this and returns the value of the last expression (which is what Xbase does).

[CompilerGenerated]
internal static __Usual <Start>g__Start$ParenExpr$1|0_0(ref <>c__DisplayClass0_0 P_0)
{
    P_0.a = P_0.a;
    P_0.b = P_0.b;
    return P_0.c = P_0.c;
}

P_0 is a structure that is shared between the main function and the local function. It contains the fields.

Because of the assignment a := a there is a compiler warning: warning XS1717: Assignment made to same variable; did you mean to assign something else?

When looking up the source code line for this warning the error happens. When you change the code to assign NIL to the various variables

#xtranslate __xlangext_SuppressNoInit(<h1> [, <hn>]) => (<h1> := NIL[, <hn> := NIL])

then these warnings disappear.

The following example shows that it works. I have rewritten xlangext_AssignTuple to assign the values to the parameter array because of the absense of the PValue() pseudo function in X#:

#pragma options("vo7", on) // allow @ for REF
#xtranslate __xlangext_SuppressNoInit(<h1> [, <hn>]) => (<h1> := NIL[, <hn> := NIL ])
#xtranslate __xlangext_Apply(<f>, <h1> [, <hn>]) => <f><h1>[, <f><hn>]
#xtranslate __xlangext_Last(<h>) => <h>
#xtranslate __xlangext_Last(<h>, <t,...>) => __xlangext_Last(<t>)
//
#xtranslate LOCAL {<args,...>} := <expr>;
    =>;
    LOCAL <args> := ({<args>} := <expr>, __xlangext_Last(<args>))

#xtranslate {<args,...>} := <expr>;
    =>;
    (__xlangext_SuppressNoInit(<args>),;
    xlangext_AssignTuple(<expr>, __xlangext_Apply(@, <args>)))

FUNCTION Start() AS VOID
LOCAL {a, b, c} := _GetTuple()
? a,b,c
xAssert(a == 1)
xAssert(b == 2)
xAssert(c == 3)
{a, b, c} := _GetTuple()
? a,b,c
xAssert(a == 1)
xAssert(b == 2)
xAssert(c == 3)
LOCAL {d, e} := _GetTuple()
? d,e
xAssert(d == 1)
xAssert(e == 2)
LOCAL {f, g,h,i} := _GetTuple()
? f, g,h,i
xAssert(f == 1)
xAssert(g == 2)
xAssert(h == 3)
xAssert(i == NIL)

function xlangext_AssignTuple(src) as usual
    local i
    local len
    local args := _ARGS()   AS USUAL[]
    // src contains the array
    // the other parameters are passed by reference in the Xs$Args array from position 2
    // the Xs$Args array is assigned to args
    // position 2 on. They have no declared parameter
    len := min(len(src), pCount() - 1)
    for i := 1 to len
        args[i+1] := src[i]
    next
return NIL

function _GetTuple() AS USUAL
    return {1,2,3}

PROC xAssert(l AS LOGIC)
IF l
    ? "Assertion passed"
ELSE
    THROW Exception{"Incorrect result"}
END IF

[cpyrgas]

Confirmed fixed