Open dschneiderch opened 2 years ago
RobertOstermann
commented
2 years ago @dschneiderch Do you have any extension specific settings? I'm not quite sure why it wouldn't be picking up your config file, but sometimes the easiest solution is just to manually set the config setting specifically for your workspace.
dschneiderch
commented
2 years ago These are my extension specific settings
"sqlfluff.executablePath": "/usr/local/bin/sqlfluff",
"sqlfluff.ignoreParsing": true,
"sqlfluff.linter.run": "onSave",
"sqlfluff.excludeRules": [
"L034",
"L032"
],
"[sql]": {
"editor.defaultFormatter": "dorzey.vscode-sqlfluff"
},i added these and now it gets the config file at least. can there be multiple config locations? i'm wondering how to handle nested sql files
"sqlfluff.config": "${fileDirname}/.sqlfluff",
"sqlfluff.ignoreLocalConfig": false,I still see this error in the log but it happens with i used the integrated terminal too. i guess this might be a sqlfluff issue?:
Format triggered for /Users/dschneider/bombora/repos/topic-interest-score-af2-dags/bombora/bairflow/topic_interest_score/include/sql/import_event_profile_tables.sql
Reading from stdin, not file, input may be dirty/partial
--------------------Executing Command--------------------
/usr/local/bin/sqlfluff fix --force --config /Users/dschneider/bombora/repos/topic-interest-score-af2-dags/bombora/bairflow/topic_interest_score/include/sql/.sqlfluff --exclude-rules L034,L032 --ignore parsing -
------------------------------------------------------------
Received close event, code 1 signal null
Raw stdout output:
------------------------------------------------------------
------------------------------------------------------------
Raw stderr output:
------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/bin/sqlfluff", line 33, in <module>
sys.exit(load_entry_point('sqlfluff==1.3.1', 'console_scripts', 'sqlfluff')())
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/cli/commands.py", line 781, in fix
result = lnt.lint_string_wrapped(stdin, fname="stdin", fix=True)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 1075, in lint_string_wrapped
linted_path.add(self.lint_string(string, fname=fname, fix=fix))
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 920, in lint_string
parsed = self.parse_string(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 852, in parse_string
rendered = self.render_string(in_str, fname, config, encoding)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 801, in render_string
templated_file, templater_violations = self.templater.process(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/base.py", line 53, in _wrapped
return func(self, in_str=in_str, fname=fname, config=config, **kwargs)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 376, in process
env, live_context, make_template = self.template_builder(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 331, in template_builder
live_context = self.get_context(fname=fname, config=config, env=env)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 311, in get_context
self._extract_macros_from_path(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 106, in _extract_macros_from_path
cls._extract_macros_from_path(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 91, in _extract_macros_from_path
cls._extract_macros_from_template(template, env=env, ctx=ctx)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 62, in _extract_macros_from_template
for k in macro_template.module.__dict__:
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1478, in module
return self._get_default_module()
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1446, in _get_default_module
self._module = self.make_module()
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1405, in make_module
return TemplateModule(self, ctx)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1535, in __init__
body_stream = list(template.root_render_func(context)) # type: ignore
File "<template>", line 12, in root
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1010, in get_template
return self._load_template(name, globals)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 969, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/loaders.py", line 126, in load
source, filename, uptodate = self.get_source(environment, name)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/loaders.py", line 218, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: sql/weekly__calc_tis__1_filter_features_for_bots_and_history.sql
------------------------------------------------------------
Reading from file, not stdin
--------------------Executing Command--------------------
/usr/local/bin/sqlfluff lint --format json --config /Users/dschneider/bombora/repos/topic-interest-score-af2-dags/bombora/bairflow/topic_interest_score/include/sql/.sqlfluff --exclude-rules L034,L032 --ignore parsing bombora/bairflow/topic_interest_score/include/sql/import_event_profile_tables.sql
------------------------------------------------------------
Received close event, code 1 signal null
Raw stdout output:
------------------------------------------------------------
------------------------------------------------------------
Raw stderr output:
------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/bin/sqlfluff", line 33, in <module>
sys.exit(load_entry_point('sqlfluff==1.3.1', 'console_scripts', 'sqlfluff')())
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/cli/commands.py", line 579, in lint
result = lnt.lint_paths(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 1173, in lint_paths
self.lint_path(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 1125, in lint_path
for i, linted_file in enumerate(runner.run(fnames, fix), start=1):
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/runner.py", line 106, in run
for fname, partial in self.iter_partials(fnames, fix=fix):
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/runner.py", line 59, in iter_partials
for fname, rendered in self.iter_rendered(fnames):
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/runner.py", line 46, in iter_rendered
yield fname, self.linter.render_file(fname, self.config)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 830, in render_file
return self.render_string(raw_file, fname, config, encoding)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/linter/linter.py", line 801, in render_string
templated_file, templater_violations = self.templater.process(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/base.py", line 53, in _wrapped
return func(self, in_str=in_str, fname=fname, config=config, **kwargs)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 376, in process
env, live_context, make_template = self.template_builder(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 331, in template_builder
live_context = self.get_context(fname=fname, config=config, env=env)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 311, in get_context
self._extract_macros_from_path(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 106, in _extract_macros_from_path
cls._extract_macros_from_path(
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 91, in _extract_macros_from_path
cls._extract_macros_from_template(template, env=env, ctx=ctx)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/sqlfluff/core/templaters/jinja.py", line 62, in _extract_macros_from_template
for k in macro_template.module.__dict__:
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1478, in module
return self._get_default_module()
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1446, in _get_default_module
self._module = self.make_module()
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1405, in make_module
return TemplateModule(self, ctx)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1535, in __init__
body_stream = list(template.root_render_func(context)) # type: ignore
File "<template>", line 12, in root
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 1010, in get_template
return self._load_template(name, globals)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/environment.py", line 969, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/loaders.py", line 126, in load
source, filename, uptodate = self.get_source(environment, name)
File "/usr/local/Cellar/sqlfluff/1.3.1/libexec/lib/python3.10/site-packages/jinja2/loaders.py", line 218, in get_source
raise TemplateNotFound(template)
jinja2.exceptions
.TemplateNotFound: sql/weekly__calc_tis__1_filter_features_for_bots_and_history.sql
------------------------------------------------------------```Looking at that last error, that is coming from a different file that i'm not trying to lint. And that file used to work fine once I added
load_macros_from_path = ../sql to my .sqlfluff
RobertOstermann
commented
2 years ago Using the sqlfluff.config does not allow for multiple configuration files to be used at once. That simply allows you to choose a single configuration file. If you only need 1 config it might be better to use the workspace folder and point the path directly to that .sqlfluff path instead of using the file directory which changes when the files are nested.
"sqlfluff.config": "${workspaceFolder}/path/to/sqlfluff"But if you want to use multiple configuration files you could try something like this, which is basically telling the extension to run the commands from the file directory. Basically what would happen if you cd into that directory and run the sqlfluff command. If you try this you should comment out or remove the config setting.
"sqlfluff.workingDirectory": "${fileDirname}"As for that last error, it looks like a sqlfluff or setup issue so not something I could fix with this extension.
dschneiderch
commented
2 years ago when i added the workingDirectory entry and removed the config entry it tells me it can't find the sqlfluff executable (even though it's explicitly set)
Format triggered for /Users/dschneider/bombora/repos/topic-interest-score-
af2-dags/bombora/bairflow/topic_interest_score/include/sql/
import_event_profile_tables.sql
Reading from stdin, not file, input may be dirty/partial
--------------------Executing Command--------------------
/usr/local/bin/sqlfluff fix --force --exclude-rules L034,L032 -
------------------------------------------------------------
Child process threw error
Error: spawn /usr/local/bin/sqlfluff ENOENT
Reading from file, not stdin
--------------------Executing Command--------------------
/usr/local/bin/sqlfluff lint --format json --exclude-rules L034,L032
../Users/dschneider/bombora/repos/topic-interest-score-af2
-dags/bombora/bairflow/topic_interest_score/include/sql/
import_event_profile_tables.sql
------------------------------------------------------------
Child process threw error
Error: spawn /usr/local/bin/sqlfluff ENOENT
Linting command failed to executeHm ok, probably best not to use that setting for now then. Are you able to get everything working just using the config setting? Or do you need to use multiple .sqlfluff configuration files at once?
dschneiderch
commented
2 years ago In that repo I was testing on its just a flat hierarchy (for now). In newer repos we've organized into a nested structure though. And in one folder export_surge_for_shipper the dialect would actually be mysql instead of bigquery so I was hoping I could override the config for that one directory with a local config file.
~/b/r/s/b/b/s/i/sql ❯❯❯ tree -L 2 -a
.
├── .sqlfluff
├── __init__.py
├── __pycache__
│ └── __init__.cpython-310.pyc
├── decorate
│ ├── __init__.py
│ ├── gen_decorated_tables.sql
│ ├── gen_decorated_tables__cte__merge_dt_dtc_dtcs_dimensions.sql
│ ├── gen_decorated_tables__cte__merge_dt_dtc_dtm_dimensions.sql
│ ├── gen_decorated_tables__cte__merge_full_dt_dtc_dtm_dimensions.sql
│ ├── gen_flat_company_data.sql
│ ├── map_top_n_countries_per_domain.sql
│ └── map_top_n_zipcodes_per_metro.sql
├── export_surge_for_freemium
│ ├── __init__.py
│ ├── freemium.sql
│ ├── freemium_arctic.sql
│ ├── freemium_waist_high.sql
│ └── freemium_waist_high_count.sql
├── export_surge_for_sales
│ ├── __init__.py
│ └── surge_for_sales.sql
├── export_surge_for_shipper
│ ├── .sqlfluff
│ ├── __init__.py
│ ├── surge_export_config__shipper_sb_s3_destination.sql
│ ├── surge_export_config__shipper_sb_sftp_destination.sql
│ └── surge_export_config__surge_export_output.sql
└── sqlfluff_libs
├── DT.py
├── __init__.py
├── __pycache__
├── params.py
└── posixpath.py
I don't think its recognizing my .sqlfluff config file since its complaining about the dialect but then linting shows up anyway. the config is at /Users/dschneider/bombora/repos/topic-interest-score-af2-dags/bombora/bairflow/topic_interest_score/include/sql my config:
the error: