AOT/Trimming

[lilith]

Does this support trimming and AOT?

[b3b00]

hello @lilith , the short answer is I don't know . I 've juste done a quick try with a real simple expression parser as a console app (code attached, simply run dotnet run and it should display "parse OK 4") It works OK without AOT + trim. It seems not to work with AOT + Trimming, giving AOT warning on publish. And then running it produces a stack :

❯ dotnet publish .\SimpleExpressionParser.csproj
Version MSBuild 17.8.5+b5265ef37 pour .NET
  Determining projects to restore...
  Restauration effectuée de C:\Users\olduh\dev\generated\SimpleExpressionParser.csproj (en 271 ms).
  SimpleExpressionParser -> C:\Users\olduh\dev\generated\bin\Release\net8.0\win-x64\SimpleExpressionParser.dll
  Generating native code
C:\Users\olduh\.nuget\packages\sly\3.2.7\lib\net7.0\sly.dll : warning IL2104: Assembly 'sly' produced trim warnings. Fo
r more information see https://aka.ms/dotnet-illink/libraries [C:\Users\olduh\dev\generated\SimpleExpressionParser.cspr
oj]
C:\Users\olduh\.nuget\packages\sly\3.2.7\lib\net7.0\sly.dll : warning IL3053: Assembly 'sly' produced AOT analysis warn
ings. [C:\Users\olduh\dev\generated\SimpleExpressionParser.csproj]
  SimpleExpressionParser -> C:\Users\olduh\dev\generated\bin\Release\net8.0\win-x64\publish\
      generated                                                                                      09:03:54
❯ .\bin\Release\net8.0\win-x64\publish\SimpleExpressionParser.exe
Unhandled Exception: System.Collections.Generic.KeyNotFoundException: The given key 'root' was not present in the dictionary.
   at System.ThrowHelper.ThrowKeyNotFoundException[T](T) + 0x14
   at System.Collections.Generic.Dictionary`2.get_Item(TKey) + 0x28
   at sly.parser.llparser.RecursiveDescentSyntaxParser`2.SafeParse(IList`1, SyntaxParsingContext`1, String) + 0x5d
   at sly.parser.Parser`2.ParseWithContext(IList`1, Object, String) + 0x60
   at ns.Program.Main(String[] args) + 0x8c
   at SimpleExpressionParser!<BaseAddress>+0x218d60

I don't know much about AOT and Trimming. Maybe you could look at it and share ?

generated.zip

[b3b00]

My guess is that trimming will trim out the visitor methods as they are not called directly but through reflection , the trimmer thinks it could safely remove them.

Maybe some mechanism exists to avoid this ?

I've not try AOT alone.

[b3b00]

hello @lilith ,

After playing with aot and trimming (see repo : https://github.com/b3b00/csly-aot) I've found :

• trimming alone seems OK
• AOT alone is KO
• AOT and trimming : kO as well

I've tried to cheat the trimmer calling directly the visitor's methods but it did not work. I don't know exactly what's going wrong but I suspect that the use of attributes to build the lexer and parser is related to it. I need some reading about .net AOT compilation to better understand what is possible.

Do you have any idea that could help ? documentation ?

thanks for replying . I 'd really like to make CSLY AOT-compliant but I need help for this

[lilith]

Run-time reflection is the problem; have you tried source generation? It can access attributes too.

On Sun, Jul 21, 2024, 1:04 AM Olivier Duhart @.***> wrote:

hello @lilith https://github.com/lilith ,

After playing with aot and trimming (see repo : https://github.com/b3b00/csly-aot) I've found :

• trimming alone seems OK
• AOT alone is KO
• AOT and trimming : kO as well

I've tried to cheat the trimmer calling directly the visitor's methods but it did not work. I don't know exactly what's going wrong but I suspect that the use of attributes to build the lexer and parser is related to it. I need some reading about .net AOT compilation to better understand what is possible.

Do you have any idea that could help ? documentation ?

thanks for replying . I 'd really like to make CSLY AOT-compliant but I need help for this

— Reply to this email directly, view it on GitHub https://github.com/b3b00/csly/issues/466#issuecomment-2241503202, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2LH63LXQ4CG62NEXCNOLZNNMPXAVCNFSM6AAAAABLEI4TNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBRGUYDGMRQGI . You are receiving this because you were mentioned.Message ID: @.***>

[b3b00]

Thanks for your feedback @lilith. I've never played with source generators. I'll take a look at then and evaluate how WE Can use them for CSLY.

[b3b00]

@lilith , I am working on AOT/Trimming issue using source generators. It works pretty well (still alpha). You can have a look at it with CSLY 3.5.0 alpha-6. I will merge this work in dev branch and publish a version as soon as I will be sufficitiently confident.

There is no documentation for now so here is a dead simple use :

// define a lexer as usual
public enum Lexer
{
    [AlphaId]
    ID,

    [Double]
    DOUBLE,

    [Keyword("YOLO")]
    YOLO
}

// define a parser as usual
[ParserRoot("root")]
public class Parser
{
    [Production("root : ID '@'[d] DOUBLE yolos")]
    public string Root(Token<Lexer> id, Token<Lexer> version, string yolos)
    {
        return $"{id.Value}@{version.DoubleValue:F} [{yolos}]";
    }

    [Production("yolos : YOLO*")]
    public string Yolos(List<Token<Lexer>> yos)
    {
        return string.Join(", ",yos.Select(x => x.Value));
    }
}

// define a parser generator that will generate the parser using generators : 

[ParserGenerator] // tagged as parser generator
// MUST be partial (as the generator will use partial to add methods
// MUST also inherit from AbstractParserGenerator, type parameters are 
//   - Lexer type
//   - Parser type
//   - parser output type
public partial class Generator : AbstractParserGenerator<Lexer, Parser, string>
{

}

// use the parser : 
public void UseTheParser() {
Generator generator = new();

        // get the parser build result
        var build = generator.GetParser();
        if (build.IsOk)
        {
            // call Parse....
            var parsed = build.Result.Parse("aot@1.0 YOLO YOLO YOLO");
            if (parsed.IsOk)
            {
                Console.WriteLine($"parse OK : {parsed.Result}");
            }
            else
            {
                foreach (var error in parsed.Errors)
                {
                    Console.WriteLine(error.ErrorMessage);
                }
            }
        }
        else
        {
            foreach (var error in build.Errors)
            {
                Console.WriteLine(error.Message);
            }
        }
}

[lilith]

Wow, thank you! Does the final code need a nuget dependency or is it all bundled into generated source?

[b3b00]

@lilith , all you need is to add a dependency to csly nuget. the generated source relies on the csly runtime so you still need a dependency to csly. Is that what you asked ?

[lilith]

Yes, that answers my question. How difficult would it be to support a zero-runtume-dependency source gen mode where the needed files are injected as private/internal? My nuget packages tend to be pretty deep in other people's dependency trees, and I wouldn't want to conflict with another lib requiring a different or future version of csly. I have a few million downloads, so it happens more often than you would think.

[b3b00]

It will be really difficult. Even antlr needs a runtime nugget. I guess that it is even not really sensible, we would need to generate a lot of code and a great part of it would finally be .... a runtime.

[lilith]

If you do every consider it, you could probably pull from PolySharp since it embeds runtime into projects.

[b3b00]

You 've decided me to start looking at generating runtime less, parser. For now I am simply trying to handwrite a parser (I 've start with GenericSimpleExpressionParser as it is quite simple. This will help me identify patterns to replicate.

As a side benefit parsers generated this way may be quite faster then those using the CSLY runtime

[lilith]

It's definitely a competitive advantage since no other C# parser generators have taken advantage of source generation yet.

On Fri, Sep 20, 2024, 5:42 AM Olivier Duhart @.***> wrote:

You 've decided me to start looking at generating runtime less, parser. For now I am simply trying to handwrite a parser (I 've start with GenericSimpleExpressionParser https://github.com/b3b00/csly/blob/dev/src/samples/SimpleExpressionParser/GenericSimpleExpressionParser.cs as it is quite simple. This will help me identify patterns to replicate.

— Reply to this email directly, view it on GitHub https://github.com/b3b00/csly/issues/466#issuecomment-2363534261, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2LH73XRRF3ISKSKORZLDZXQC33AVCNFSM6AAAAABLEI4TNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRTGUZTIMRWGE . You are receiving this because you were mentioned.Message ID: @.***>