search

andialbrecht / sqlparse

A non-validating SQL parser module for Python
BSD 3-Clause "New" or "Revised" License
3.71k stars 693 forks source link

Lexer default instance is set in class before keywords are added #730

Closed gshank closed 6 months ago

gshank commented 1 year ago

In https://github.com/dbt-labs/dbt-core we use sqlparse to merge ctes at a particular place in our code. Multiple threads are started at the same time, and because the Lexer default instance is set before the keywords are added we were getting intermittent failures because we were parsing with a Lexer with no keywords.

In order to avoid this problem we had to initialize the Lexer earlier in the process: https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/compilation.py#L527

The keywords should be added to the Lexer instance before it is is stored in the class.

andialbrecht commented 1 year ago

Agreed. The Lexer shouldn't change iff it's a singleton and it should be thread-safe to use.