[ELF] A new Lexer for Linker Script

[hongyu-dev]

• [x] delete peek2 https://github.com/llvm/llvm-project/pull/99790
• [x] delete consumeLabel https://github.com/llvm/llvm-project/pull/99567
• [ ] #100462
• [x] #101035
• [ ] Improve https://github.com/yugier/llvm-project/commit/4ad3deeefc6c7cd2b1248d6b06c6888618210093
• [ ] update symbol assignment
• [ ] MEMORY, merge memory.s and memory-err.s
• [ ] improve testing for `DATA_SEGMENT_RELRO_END
• [x] introduce state to remove some unquote and https://github.com/llvm/llvm-project/pull/84130 hack https://github.com/llvm/llvm-project/commit/30ec2bf58d1559b161254118e39271b829fd0867
• [x] fully stateful lexer; replace eager tokenize with on-demand lexing;with on-demand lexing, replace getColumnNumber (from Token) with Lexer::getColumnNumber https://github.com/llvm/llvm-project/pull/100493
• [x] replace mbs with an include stack (INCLUDE); replace getCurrentMB() https://github.com/llvm/llvm-project/issues/93947
• [ ] add a consume overload to accept Token. The existing consume(StringRef) is be a special case that expects identifier/quoted_string.
• [ ] #99921 https://github.com/llvm/llvm-project/pull/100180

We want to replace current lexer with a stateful one to report more errors and get rid of many hacks in current lexer such as (ScriptLexer::maybeSplitExpr is removed in https://github.com/llvm/llvm-project/issues/93947). The list of items is the plan to update lexer gradually.

[llvmbot]

@llvm/issue-subscribers-lld-elf

Author: Hongyu Chen (yugier)

- [x] ~~delete peek2~~ https://github.com/llvm/llvm-project/pull/99790 - [ ] delete consumeLabel - [ ] add `struct Token {StringRef str; TokenKind kind;}; vector<StringRef> tokens;` => `vector<Token> token;` change next/peek to return Token - [ ] add a consume overload to accept Token. The existing consume(StringRef) is be a special case that expects identifier/quoted_string. - [ ] introduce state to remove some unquote and https://github.com/llvm/llvm-project/pull/84130 hack - [ ] introduce state to replace inExpr - [ ] fully stateful lexer; replace eager tokenize with on-demand lexing;with on-demand lexing, replace getColumnNumber (from Token) with Lexer::getColumnNumber - [ ] replace mbs with an include stack (INCLUDE); replace getCurrentMB() We want to replace current lexer with a stateful one to report more errors and get rid of many hacks in current lexer such as `ScriptLexer::maybeSplitExpr`. The list of items is the plan to update lexer gradually.