What is meant by "tokens"?

[FismanMaxim]

I find it weird that the term "token", being the core of the whole challenge, was almost not explained in the description or the video. It is not a sort of a widely-accepted term, is it? What does it include except for the names of variables/functions/classes?

• colons and semicolons
• commas and points
• brackets and braces
• mathematical operators (+, -, , /, =, +=, -=, =, /=, etc)
• bitwise operators (&, |, >>, <<, etc.)
• logical operators (&&, ||, !)
• declarations of types that include several "names" such as: List, int[], List< List < float > >, etc.
• keywords (for, foreach, if-else if-else, return, private/public, static, class/struct etc.)
• and so on...

I think it would be correct to state explicitly what is considered a "token", or at least provide a link to the definition that is used.

[lucasmeneghin]

Take a look at TokenCounter class

Basically everything is a token, except for:

PrivateKeyword PublicKeyword SemicolonToken CommaToken CloseBraceToken -> open and close brackets should be 1 token, not 2 CloseBracketToken CloseParenToken

String literals count for as many chars as are in the string

[CREAsTIVE]

He use Microsoft.CodeAnalysis.CSharp.CSharpSyntaxTree to parse youre code

peculiarities (comments from code): String literals count for as many chars as are in the string Regular tokens count as just one token

[lucasmeneghin]

Although i think the token limit could be higher or tokensToIgnore should have more entries, i feel like the bots are going to be very dumb

[davejrodriguez]

View how they're counted/excluded here: https://github.com/SebLague/Chess-Challenge/blob/main/Chess-Challenge/src/Framework/Application/Helpers/Token%20Counter/TokenCounter.cs

Cross reference the exclusions with: https://learn.microsoft.com/en-us/dotnet/api/microsoft.codeanalysis.csharp.syntaxkind?view=roslyn-dotnet-4.6.0

String literals are very expensive.

[FismanMaxim]

@lucasmeneghin Yeah, having a look at this class was the first thing I did! And that's the point of this issue. I think one should get this information from the rules, not fish it out from the project's internal classes.

I feel like the concept of tokens, which, being the only restriction, is basically the concept of the whole challenge, must be explicitly explained in the rules.

[lucasmeneghin]

I get your point, but it doesnt seem to big of a deal This is a coding challenge, the source code is there, and its indicated in the UI how many tokens you're using. Its pretty intuitive and easy to test/figure it out

[lucasmeneghin]

But yeah wouldnt hurt to have it better detailed in the description

[FismanMaxim]

@lucasmeneghin Neither do I consider it a big deal, I'd say a middle-size deal :)

[Vanny-Chuck]

@lucasmeneghin Yeah, having a look at this class was the first thing I did! And that's the point of this issue. I think one should get this information from the rules, not fish it out from the project's internal classes.

I feel like the concept of tokens, which, being the only restriction, is basically the concept of the whole challenge, must be explicitly explained in the rules.

He put up a youtube video, he explains all of this in it.

[cdrch]

For those wondering where this concept of a token comes from in the first place, I see it most often when discussing creating lexers and parsers, and other things related to the creation of a programming language, or the translation of one into another. See also the Wikipedia page on Lexical Analysis and this Stack Overflow question.

(Related, I'm actually quite excited by this choice of limitation. It avoids the craziest of the code golfing nonsense while keeping the same spirit, and it also means it's possible to create an interesting new chess engine concept that doesn't require insane amounts of dedication to pushing state of the art tech currently dominated by big corporations. A random programmer on their own could come up with something very interesting here!)