🌊 Streams: Processing for classic streams

flash1293 commented 2 weeks ago

This PR allows to add processing to classic streams.

When defining processing for a classic stream, it will add a reference to the <stream name>@stream.processing pipeline to the default pipeline of the stream (or a @custom sub pipeline if it exists). Then, it will write processing rules like for wired streams.

On the UI, the processing tab is not implemented yet, but this PR also prepares that there are two tabs (overview and enrich) shown for classic streams, not just an overview page.

flash1293 commented 4 days ago

Thanks @simianhacker - added that:

I also extended the logic a bit to clean up the processor in the user-managed pipeline if the stream gets deleted so the user has a safe way to get rid of it.

Should be good for another look.

I will add some integration tests for this on a follow-up PR.

elasticmachine commented 2 days ago

:yellow_heart: Build succeeded, but was flaky

Buildkite Build
Commit: fff329b7ff8b26c22679c3102fc860417291216f

Failed CI Steps

Jest Tests #9

Test Failures

[job] [logs] Jest Tests #9 / useUpdateTags should call onSuccess when bulk update tags succeeds
[job] [logs] Jest Tests #9 / useUpdateTags should call show error toast when bulk update tags fails
[job] [logs] Jest Tests #9 / useUpdateTags should call show error toast when update tags fails

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`streamsApp`	149	152	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`streamsApp`	95.5KB	96.1KB	+587.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`streams`	4	3	-1

History

:yellow_heart: Build #258697 was flaky 2862da41d6083637b97dd0a6ce192671d67b87b5
:green_heart: Build #258406 succeeded 5ceb58955048aedf11252c610f9ff6cdb18c3dfc
:green_heart: Build #258036 succeeded 31e171edc9aed816fcc2b76d455ec66239a50ce8
:green_heart: Build #257417 succeeded 6083f3242b958d135a68d425bdcd3fa99a925526
:green_heart: Build #257329 succeeded 105b0d2502d7297a6d9b634b5d9ea487df7a81f4
:yellow_heart: Build #257095 was flaky ef110d65593f3cd0dbfe4fcebf543e28d051bc08

kibanamachine commented 2 days ago

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12275394441

kibanamachine commented 2 days ago

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

elastic / kibana