search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
51 stars 3.5k forks source link

Add ingest migrator CLI tool #7329

Open suyograo opened 7 years ago

suyograo commented 7 years ago

Based on all the great work that @original-brownbear has done in tools/ingest-convertor it is time now to create a CLI tool that provides an end-user interface to move from ingest pipeline to LS config.

I propose two flavors to make it dead simple and easy for our users -- offline and an online.

Offline

bin/ingest-convert.sh --input file:///tmp/apache.json --output file:///tmp/pipeline.conf

This would call the ingest_pipeline_to_logstash() js function as done in https://github.com/elastic/logstash/blob/master/tools/ingest-converter/src/main/java/org/logstash/ingest/Pipeline.java

We can create all the heavy lifting in Java and just write a shell wrapper around it.

Online (Lower priority)

This assumes the user has access to the ES endpoint where the ingest pipeline is defined.

bin/ingest-convert.sh --input apache --output /tmp/pipeline.conf

defaults to localhost:9200. It then gets the json pipeline using the ES API call GET _ingest/pipeline/apache, stores it in memory or tmp location and then converts it using the utilities created.

can accept optional ES config for credentials:

bin/ingest-convert.sh --pipeline apache --output /tmp/pipeline.conf --user foo --password

For this, maybe we can use the Java high-level client from ES (when it gets shipped)

original-brownbear commented 7 years ago

Done in https://github.com/elastic/logstash/pull/7335

original-brownbear commented 7 years ago

ah nvm part 2 missing