search

teragrep / pth_10

Data Processing Language (DPL) translator for Apache Spark
GNU Affero General Public License v3.0
0 stars 6 forks source link

#384: Add `| teragrep exec foreachbatch` command that sets processing to batch mode #390

Closed eemhu closed 9 hours ago

eemhu commented 6 days ago

Adds | teragrep exec foreachbatch command, which turns off batch collect and sets processing to batch mode via step properties.

See issue

eemhu commented 6 days ago

I guess this has to be tested in QA mostly as the unit tests don't support batches of data?

Yes, with the current testing setup it's not really possible. As-is the test only really tests that the command produces expected results data-wise, and with only one batch.