format and add test for KPDB sql files

NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team

14 stars 0 forks source link

format and add test for KPDB sql files #175

Closed damonmcc closed 9 months ago

damonmcc commented 9 months ago

changes

add a pyproject.toml file to configure sqlfuff (rather than a .sqlfluff file. we get better list value formatting and will allow for one file for future configurations like pytest)
- excluded rules were chosen to avoid substantial untestable changes to queries in this PR / first pass
call sqlfluff in PR tests on KPDB files only
format KPDB sql files using sqlfluff fix and manual edits

alexrichey commented 9 months ago

@damonmcc Nice... I'd actually added a pyproject file locally, just to configure my python linter. Mind adding the following?

[tool.pydocstyle]
ignore = ["D100", "D101", "D102", "D107", "D104", "D213", "D407", "D413"]

[tool.black]

damonmcc commented 9 months ago

@alexrichey what effect does the empty [tool.black] section have?

alexrichey commented 9 months ago

@alexrichey what affect does the empty [tool.black] section have?

@damonmcc It's usually just to inform an editor to use black. Mine requires it

damonmcc commented 9 months ago

@alexrichey added to pyproject.toml

fvankrieken commented 9 months ago

Thanks for putting the time into this, very happy with it

damonmcc commented 9 months ago

@alexrichey

looks good... curious if you were able to run these just to check that the formatting isn't invaliding the files somehow?

to follow up: it isn't a test of all the queries because the build is "broken", but it is currently failing at the same spot as it was before the reformatting