sqlfluff and PyCharm keep creating hundreds of __sqlfluff_tmp_ files

[nenkie76]

I have sqlfluff plugin installed:

• PyCharm 2022.2.4 (Professional Edition)
• Python 3.10
• MacOS 13.6 (22G120)

sqlfluff is using Global Recommended configuration with following arguments:

--dialect snowflake --config /Users/xxx/git/xxx/.sqlfluff_dbt

Whenever I open any sql file, for example, macros, it keeps creating a lot of temporary files in the same folder. 1 temp file named __sqlfluff_tmp_ for each time you open sql file in PyCharm. For example, here I have opened and closed create_stage.sql file 5 times:

The problem is that you can't run dbt models locally, because the content is duplicated.

How can we avoid creation of these files?

[dalgarins]

Hi @nenkie76 the plugin deleted the file after use it with sqlfluff, creating this temp file was the only way I found to execute sqlfluff when you change your script, other option could be change the plugin to execute the linter after save the file, and in that case I don't need to create the temp file.

[dalgarins]

maybe if I add a setting option to allow users decided when they want to execute sqlfluff could and option, execute after save or execute when you modified the original file.

[nenkie76]

@dalgarins , thanks for quick response.

The fact, that plugin keeps creating temp files is fine, but creating files in the same folder, as appeared, creates conflicts. Like in my example, it is not possible to use plugin, when you're editing dbt models.

Is it not possible to add an option to specify a custom path for storing temp files?

[jmskarda]

Another possibility would be to write the temp files to the OS designated temp folder: %TMP% or /tmp. However, that would affect SQLFluff's config file search process. You'll be okay if your SQLFluff config files or only in your user home. However, if you rely on an SQL config file that is within your project folder, then SQLFluff will ignore that config file when it is linting a file in an OS temp folder.

The same would be true for a user-specified temp folder, but at least the user could copy the repo's SQLFluff config file to that folder and not have it be deleted automatically by the OS.

In my case, this plugin's temp files were being synchronized with OneDrive. I just got a warning email from OneDrive mentioning that I "deleted a large number of files". All of them were temp files from this plugin. I can avoid that by not working on any files located in OneDrive paths. However, its nice to have a scratch folder that is back-up. (These files don't really belong in a repo.)

I would prefer the approach that would allow me to specify a folder for the temp files.

[imonteroq]

I am also facing this issue. For now I have added a .dbtignore file at the repo root with the following `/__sqlfluff*.sql` in it. It's doing the job.

[dalgarins]

@imonteroq , @jmskarda , @nenkie76 please update to latest plugin versión and let me know if your issues were fixed.

[dalgarins]

@nenkie76, @jmskarda, @imonteroq any updates, it's the issue fixed to you?

[nenkie76]

@dalgarins , sorry, I can't check it since we stopped using sqlfluff for dbt in favour to sqlfmt

[joshuataylor]

I'm still having this issue, not as many but still there.

It'll probably be worse when you latency, eg Snowflake etc.

Could you make it an action instead, so we could trigger with a keymap etc?

[dalgarins]

@joshuataylor which plugin version are you using?