confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
102 stars 1.04k forks source link

Support DEFINE and UNDEFINE in headless mode #7594

Open dstelljes opened 3 years ago

dstelljes commented 3 years ago

Is your feature request related to a problem? Please describe. When starting ksqlDB in headless mode with a script that contains DEFINE or UNDEFINE, the server fails to start with the following exception:

[2021-05-26 16:29:09,207] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:66)
io.confluent.ksql.util.KsqlStatementException: Unsupported statement. Only the following statements are supporting in standalone mode:
CREAETE STREAM AS SELECT
CREATE STREAM
CREATE TABLE
CREATE TABLE AS SELECT
INSERT INTO
REGISTER TYPE
SET
UNSET
Statement: DEFINE test = 'value';
    at io.confluent.ksql.rest.server.StandaloneExecutor$StatementExecutor.execute(StandaloneExecutor.java:310)
    at io.confluent.ksql.rest.server.StandaloneExecutor.executeStatements(StandaloneExecutor.java:218)
    at io.confluent.ksql.rest.server.StandaloneExecutor.validateStatements(StandaloneExecutor.java:202)
    at io.confluent.ksql.rest.server.StandaloneExecutor.processesQueryFile(StandaloneExecutor.java:180)
    at io.confluent.ksql.rest.server.StandaloneExecutor.startAsync(StandaloneExecutor.java:122)
    at io.confluent.ksql.rest.server.KsqlServerMain.tryStartApp(KsqlServerMain.java:89)
    at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:64)

Describe the solution you'd like KLIP-38 suggests that variable substitution should work in headless mode. It looks like this is a matter of adding DEFINE and UNDEFINE to the supported handlers in StandaloneExecutor. Happy to submit a PR; just wanted to confirm that this is a valid request.

Describe alternatives you've considered Deploying in interactive mode is an option, but headless works better with our CD workflow.

Additional context Steps to reproduce:

$ docker run -it -u root --rm --entrypoint bash confluentinc/ksqldb-server:0.17.0

# cat > /define.sql <<EOF
DEFINE test = 'value';
EOF
# export KSQL_BOOTSTRAP_SERVERS=localbroker:9092
# export KSQL_KSQL_QUERIES_FILE=/define.sql
# /usr/bin/docker/run
TheDarkFlame commented 1 year ago

@colinhicks Wanted to confirm the status of issue? Including this would considerably simplify CICD which includes KSQLDB as a component, and I'm still bumping into it as an issue.

dstelljes commented 1 year ago

For whatever this is worth, as a workaround, we deploy a patched internal image that runs envsubst(1) on the queries file at startup. This works well enough for our purposes, and the syntax is similar.