reata / sqllineage

SQL Lineage Analysis Tool powered by Python
MIT License
1.19k stars 215 forks source link

SQLfluff Not read nested configs. #628

Open lkurgan55 opened 1 week ago

lkurgan55 commented 1 week ago

Describe the bug

SQLfluff can read nested configs and override values from them. But sqllineage use wrong read config function here.

FluffConfig.from_root can read only local config from cwd. Intead of that, I think, we shoud use FluffConfig.from_path and throw the path of validating file. In that case, sqlfluff will read all configs from cwd to file location.

To Reproduce

Create some directories like folder1/folder2/folder3 In folder1 and folder3 create .sqlfluff configs. Add foo.sql to folder3

Use sql query with mocks like SELECT * FROM {{ source_table }} Add source_table value to /folder1/folder2/folder3/.sqlfluff and leave /folder1/.sqlfluff empty.

Run sqllineage -d athena -f /folder1/folder2/folder3/foo.sql

As result we will get Error Line 1, Position 10: Found unparsable section: 'FROM'

Expected behavior

Sqllineage uses sqlfluff correctly to read neasted configs.

Additional context

To fix that issue we have to change this function to FluffConfig.from_path(path=file_path, overrides={"dialect": self._dialect}) and throw validating file path.

KulykDmytro commented 6 days ago

Just to clarify, jinja templating is used here