elastic / support-diagnostics

Support diagnostics utility for elasticsearch and logstash
Other
289 stars 150 forks source link

Add `GET _node/pipelines?graph=true` for Logstash 7.6.0+ #678

Open sakurai-youhei opened 7 months ago

sakurai-youhei commented 7 months ago

This PR adds GET _node/pipelines?graph=true for Logstash 7.6.0+.

Closes #513

Run logstash-api against Logstash 7.6.0:

$ ./diagnostics.sh --type logstash-api --host 7df8989a7113 --port 9600 -o . --bypassDiagVerify
Using /bin/java as Java Runtime
Using -Xms256m -Xmx2000m  for options.
ERROR StatusConsoleListener Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...

Creating temp directory: ./logstash-api-diagnostics
Configuring log file.
Diagnostic logger reconfigured for inclusion into archive
Getting Logstash Version.                                                            <- THIS IS NEW.
Results written to: ./logstash-api-diagnostics/logstash_version.json
Results written to: ./logstash-api-diagnostics/logstash_node.json
Results written to: ./logstash-api-diagnostics/logstash_node_stats.json
Results written to: ./logstash-api-diagnostics/logstash_node_pipelines_graph.json    <- THIS IS ONLY FOR 7.6.0+.
Results written to: ./logstash-api-diagnostics/logstash_nodes_hot_threads.json
Results written to: ./logstash-api-diagnostics/logstash_nodes_hot_threads_human.txt
Results written to: ./logstash-api-diagnostics/logstash_plugins.json
Writing legacy [manifest.json].
Writing [diagnostic_manifest.json].
Identified Docker installations - bypassed log collection and some system calls.
Closing loggers.
Archiving diagnostic results.
Archive: ./logstash-api-diagnostics-20231115-085834.zip was created
Deleted directory: ./logstash-api-diagnostics.

Run logstash-api against Logstash 7.5.2:

$ ./diagnostics.sh --type logstash-api --host 84d4c8cbab5c --port 9600 -o . --bypassDiagVerify
Using /bin/java as Java Runtime
Using -Xms256m -Xmx2000m  for options.
ERROR StatusConsoleListener Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...

Creating temp directory: ./logstash-api-diagnostics
Configuring log file.
Diagnostic logger reconfigured for inclusion into archive
Getting Logstash Version.
Results written to: ./logstash-api-diagnostics/logstash_version.json
Results written to: ./logstash-api-diagnostics/logstash_node.json
Results written to: ./logstash-api-diagnostics/logstash_node_stats.json
Results written to: ./logstash-api-diagnostics/logstash_nodes_hot_threads.json
Results written to: ./logstash-api-diagnostics/logstash_nodes_hot_threads_human.txt
Results written to: ./logstash-api-diagnostics/logstash_plugins.json
Writing legacy [manifest.json].
Writing [diagnostic_manifest.json].
Identified Docker installations - bypassed log collection and some system calls.
Closing loggers.
Archiving diagnostic results.
Archive: ./logstash-api-diagnostics-20231115-090151.zip was created
Deleted directory: ./logstash-api-diagnostics.

Checklist

Not sure if this is sensitive data but logstash_node_pipelines_graph.json contains full path of pipeline config files.

Example of logstash_node_pipelines_graph.json ```json { "host": "7df8989a7113", "version": "7.6.0", "http_address": "0.0.0.0:9600", "id": "d0e77f42-aee4-4436-a32f-5dd2a903b134", "name": "7df8989a7113", "ephemeral_id": "0f1cedd2-4386-4252-95c4-2ce0fb77933f", "status": "green", "snapshot": false, "pipeline": { "workers": 16, "batch_size": 125, "batch_delay": 50 }, "pipelines": { "main": { "ephemeral_id": "7a145d9b-2637-4c0d-9757-bd7ed3c19afc", "hash": "1e3da6bfb66bd25bbb98769a5dcdc16c06f98a333675d3b71cccad117e103b99", "workers": 16, "batch_size": 125, "batch_delay": 50, "config_reload_automatic": false, "config_reload_interval": 3000000000, "dead_letter_queue_enabled": false, "graph": { "hash": "1e3da6bfb66bd25bbb98769a5dcdc16c06f98a333675d3b71cccad117e103b99", "type": "lir", "version": "0.0.0", "graph": { "vertices": [ { "config_name": "beats", "plugin_type": "input", "meta": { "source": { "protocol": "file", "id": "/usr/share/logstash/pipeline/logstash.conf", "line": 2, "column": 3 } }, "id": "0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa", "explicit_id": false, "type": "plugin" }, { "meta": null, "id": "__QUEUE__", "explicit_id": false, "type": "queue" }, { "config_name": "stdout", "plugin_type": "output", "meta": { "source": { "protocol": "file", "id": "/usr/share/logstash/pipeline/logstash.conf", "line": 8, "column": 3 } }, "id": "f4944472678ac54e7343c1a49748c402b0bafd76ebab7fe2f3930269e0e5097b", "explicit_id": false, "type": "plugin" } ], "edges": [ { "from": "0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa", "to": "__QUEUE__", "id": "141b82e2394ae228527d441ecf49aae13e2a4158e5801e46043ecbd3b6ddf5ec", "type": "plain" }, { "from": "__QUEUE__", "to": "f4944472678ac54e7343c1a49748c402b0bafd76ebab7fe2f3930269e0e5097b", "id": "f4b0392cfc0f7a8a7efe05342ab597f370b1841e7ba8eda9ca38c9072a040f18", "type": "plain" } ] } } } } } ```