dedup command does not work

teragrep / pth_10

Data Processing Language (DPL) translator for Apache Spark

GNU Affero General Public License v3.0

0 stars 6 forks source link

dedup command does not work #198

Closed eemhu closed 9 months ago

eemhu commented 9 months ago

Describe the bug

%dpl
index=alert_examples earliest="01/01/2021:02:34:19"
| rex4j field=_raw "message=(?<message>\"[a-zA-Z\s]+\"[^0-9])"
| rex4j field=message "(?<message>[^\"]+)"
| dedup message

results in one batch of data after which each batch is completely empty.

Expected behavior Deduplicate the message column

How to reproduce Run dpl command above

Software version pth-10 4.16.0-18-g11cc4941

eemhu commented 9 months ago

internal pull request submitted

eemhu commented 9 months ago

Merged to internal dpl2 branch