Assign location to tokens

LanguageDev / Yoakke

A collection of libraries for implementing compilers in .NET.

Apache License 2.0

141 stars 8 forks source link

Assign location to tokens #150

Closed kant2002 closed 6 months ago

kant2002 commented 1 year ago

That allow implement parsers which indirectly span across multiple files, like C/C++

kant2002 commented 1 year ago

This PR can be considered draft in a sense that it has flaws and require discussion. Also tired.

SourceFile as abstraction too much tied to TextReader. Lexers need direct access to TextReader, but SourceFile diagnostics machinery require buffered access where precalculated lines. Possble highligt issue in SourceFile implementation
I have to reimplement Equals and GetHashCode for Token. That also hints on too much coupling between concepts.

Would like to start talking about directions.

kant2002 commented 1 year ago

Also I did not attempt to make API without breaking changes. Have to know opinion on the matter, and is breaking changes possible

kant2002 commented 1 year ago

I come to conclusion that features of SourceFile which trigger inconsistent behaviour is overkill. Would be good if SourceFile would no longer be derived from TextWriter, and for backward compatibility there either other class with similar functionality, or make breaking change.

kant2002 commented 1 year ago

Will try to summon @LPeter1997 to discuss things

LPeter1997 commented 1 year ago

Sorry for the super-delayed reply. I'll try to get to this in a week or so. I'm sadly working on other things and I have many-many things left here to tidy up.

kant2002 commented 6 months ago

@fornever I manage to change PR to the state where it does not have source breaking changes, and only binary one. What do you think, should I go further and make it binary compatible (which ideally proper way)

ForNeVeR commented 6 months ago

I'd say that Yoakke shouldn't care about binary compatibility for now.