teragrep set|get config does not allow setting configuration before streaming

teragrep / pth_10

Data Processing Language (DPL) translator for Apache Spark

GNU Affero General Public License v3.0

0 stars 6 forks source link

teragrep set|get config does not allow setting configuration before streaming #431

Open kortemik opened 4 days ago

kortemik commented 4 days ago

Describe the bug

%dpl

| teragrep set config dpl.pth_06.kafka.enabled false

[WRITE_STREAM_NOT_ALLOWED] `writeStream` can be called only on streaming Dataset/DataFrame.

Expected behavior should work

How to reproduce run the query

Screenshots

Software version pth_07 8.2.0

Desktop (please complete the following information if relevant):

OS:
Browser:
Version:

Additional context

kortemik commented 4 days ago

introduced in #394

eemhu commented 4 days ago

I think it might be related to EmptyDataframeStep generating a non-streaming dataset in cases where there is no logicalStatement. Explains why it only occurred now as most commands generate a streaming dataset or use the data provided by archive. set config only passes the pre-existing dataset through with no changes.