pygments / pygments

Pygments is a generic syntax highlighter written in Python
http://pygments.org/
BSD 2-Clause "Simplified" License
1.8k stars 661 forks source link

PostgresLexer's String.Name is basically unused #2780

Open PIG208 opened 3 weeks ago

PIG208 commented 3 weeks ago

PostgresLexer (added in https://github.com/pygments/pygments/commit/4e878a0d8bd39450028938295d52436c630fef4d) can generate String.Name, which is not included in pygments.token.STANDARD_TYPES.

Not sure if this is intended, but for this input

"

it produces something like this:

<span class="s s-Name">"</span>

This token is only used in PostgresLexer and nowhere else:

$ rg String.Name
pygments/lexers/sql.py
182:            (r'((?:U&)?)(")', bygroups(String.Affix, String.Name), 'quoted-ident'),
203:            (r'[^"]+', String.Name),
204:            (r'""', String.Name),
205:            (r'"', String.Name, '#pop'),

tests/examplefiles/postgresql/postgresql_test.txt.output
292:'"'           Literal.String.Name
293:'\\0441\\043B\\043E\\043D' Literal.String.Name
294:'"'           Literal.String.Name

tests/examplefiles/psql/psql_session.txt.output
236:'"'           Literal.String.Name
237:'\n'          Literal.String.Name
240:' foo'        Literal.String.Name
241:'"'           Literal.String.Name
309:'"'           Literal.String.Name
310:'hex'         Literal.String.Name
311:'"'           Literal.String.Name
322:'"'           Literal.String.Name
323:'dec'         Literal.String.Name
324:'"'           Literal.String.Name

It appears to me that it is pretty uncommon for any style to use tokens that are not yet a part of the STANDARD_TYPES.

We could either use sn as the shortened value, or just switch to an existing standard token like String (which is how String.Name is effectively styled now).

birkenfeld commented 2 weeks ago

Yeah, if it's some form of quoting a name it should be Name.something not String.something.

MySQL has Name.Quoted (which also appears only there)...

PIG208 commented 2 weeks ago

Looks like a motivation for adding these non-standard token types are for user customization (#1555). They typically use the fallback of the parent token type for most styles. However, perhaps it will be better to use a fitting standard token type (or add a new one) for better discoverability.

In the case of String.Name, maybe it can be refactored to use Name.Quoted as well, making the latter a standard token type too.