Implementing a Before_run_SQL

[luis-indicium]

Implement a Before_run_SQL for SQL commands before sending the data. This would make Drop_Table commands easier for projects that need it. Today doing this is not possible using Meltano.

[pnadolny13]

@luis-indicium this is an interesting idea!

How were you imagining this to be implemented? As a config value in your meltano.yml or maybe a path to a sql script?

If you provide a SQL script:

• Should the target run it for every stream it sees?
• Or just once?
• If for every stream, will the target inject the table name into the sql? Like should the sql be templated in some way?

Without templating it might be difficult to use environments because table names will be hardcoded so you'll need different scripts for each environment.

[luis-indicium]

Hi @pnadolny13

I think of the implementation in meltano.yml as a target config. We could pass a .sql file or a .sql description in the config itself.

Passing in the config, couldn't we define the stream that will be used through Meltano's pipelines?

Something similar to this:

https://hub.meltano.com/loaders/target-postgres--datamill-co/#before_run_sql-setting