SQLfluff can read nested configs and override values from them. But sqllineage use wrong read config function here.
FluffConfig.from_root can read only local config from cwd. Intead of that, I think, we shoud use FluffConfig.from_path and throw the path of validating file. In that case, sqlfluff will read all configs from cwd to file location.
To Reproduce
Create some directories like folder1/folder2/folder3
In folder1 and folder3 create .sqlfluff configs.
Add foo.sql to folder3
Use sql query with mocks like SELECT * FROM {{ source_table }}
Add source_table value to /folder1/folder2/folder3/.sqlfluff and leave /folder1/.sqlfluff empty.
Run
sqllineage -d athena -f /folder1/folder2/folder3/foo.sql
As result we will get Error Line 1, Position 10: Found unparsable section: 'FROM'
Expected behavior
Sqllineage uses sqlfluff correctly to read neasted configs.
Additional context
To fix that issue we have to change this function to FluffConfig.from_path(path=file_path, overrides={"dialect": self._dialect}) and throw validating file path.
Describe the bug
SQLfluff can read nested configs and override values from them. But sqllineage use wrong read config function here.
FluffConfig.from_root
can read only local config from cwd. Intead of that, I think, we shoud useFluffConfig.from_path
and throw the path of validating file. In that case, sqlfluff will read all configs from cwd to file location.To Reproduce
Create some directories like
folder1/folder2/folder3
Infolder1
andfolder3
create.sqlfluff
configs. Addfoo.sql
tofolder3
Use sql query with mocks like
SELECT * FROM {{ source_table }}
Addsource_table
value to/folder1/folder2/folder3/.sqlfluff
and leave/folder1/.sqlfluff
empty.Run
sqllineage -d athena -f /folder1/folder2/folder3/foo.sql
As result we will get Error
Line 1, Position 10: Found unparsable section: 'FROM'
Expected behavior
Sqllineage
usessqlfluff
correctly to read neasted configs.Additional context
To fix that issue we have to change this function to
FluffConfig.from_path(path=file_path, overrides={"dialect": self._dialect})
and throw validating file path.