Pipeline simulator is too technical (wrong level of abstraction)

[joschi]

The pipeline simulator is too technical for most users and requires intimate knowledge of Graylog's internals to use.

A normal user without deep knowledge of Graylog's internals such as the separation of transports and codecs, which never shown anywhere in the Graylog web interface or in the documentation, will not succeed in using the pipeline simulator at all.

Furthermore, it's not possible to simulate the handling of messages received via a binary protocol such as NetFlow or Beats without using some tricks (which won't be evident without deep knowledge of Graylog's internals).

For example, in order to simulate a rule on a structured message, i. e. not just a "raw" message with the "Raw string" codec, users have to craft a valid GELF message which then can be run through the simulator. Unfortunately users won't know that because the "Raw message" text fields lacks a description.

[joschi]

Possible solutions:

• Add a message loader to the pipeline simulator, similar to the extractors page, to load an existing message and simulate its run through the pipelines.
• Store input and codec which was being used to receive a message, as well as its raw, unparsed form, allow selecting messages via some sort of message loader, and run its pristine form through the codec to parse it and then the simulator.

[gimmic]

I do like the idea of a message loader which is similar to other locations.

[danielo515]

This is the reason why I keep using extractors. They are so easy to setup and test. Till now, I have never been able to test a pipeline