search

TobikoData / sqlmesh

Efficient data transformation and modeling framework that is backwards compatible with dbt.
https://sqlmesh.com
Apache License 2.0
1.61k stars 141 forks source link

Pretty SQL before sending to execution engine #2935

Open MikeWallis42 opened 1 month ago

MikeWallis42 commented 1 month ago

A small request I hope. Would it be possible to use SQLGlot's pretty functionality before executing SQL statements? It would be much easier to find syntax issues if it wasn't one long line.

patricksurry commented 1 month ago

had a similar comment for audits - it's great to be able to cut & paste sql for a failing audit, but would be even better if I could see pretty sql without an external tool. either by default or with sqlmesh audit --pretty or something.

MikeWallis42 commented 4 weeks ago

@izeigerman, please correct me if I'm wrong but would this be as simple to implement as changing this line from False to True?

https://github.com/TobikoData/sqlmesh/blob/v0.117.0/sqlmesh/core/engine_adapter/base.py#L2037

If so then I'd be happy to contribute this.

izeigerman commented 4 weeks ago

@MikeWallis42 is the purpose of this to make the queries in logs prettier? Or some other place?

izeigerman commented 4 weeks ago

I'm concerned that this will significantly increase the size of the payload and the IO overhead as a result. I'm open to making this configurable with this behavior being disabled by default.

MikeWallis42 commented 4 weeks ago

If I were to be really specific then this would be to better enable debugging and would typically be used during development more than in a production setting. As SQLGlot cannot and should not be validating the syntax of the SQL (apart from really basic things), it's often the case that bad SQL is written and run whilst developing. Finding the location of the bad SQL (from the execution engine) is typically easier when we can scan down and then across. At the moment all the issues are on line 1 and, sometimes, several thousand characters in.

It might be that this would be configured against a specific gateway in the context of SQLMesh, at least in the way that I'm using it currently.

tobymao commented 3 weeks ago

@MikeWallis42 are you planning to make a PR?

MikeWallis42 commented 2 weeks ago

I would love to have a go! I've made a start today

tobymao commented 2 weeks ago

Great let us know if you need any help, you can ping us on slack.

sungchun12 commented 1 week ago

@MikeWallis42 looks like you're not the only one that cares about this: https://github.com/TobikoData/sqlmesh/issues/3086#issuecomment-2329678997

But looks like you don't want to see the pretty SQL rendered in the terminal, only in the query history for the engine executing the SQL. Is that correct?