ingest-attachment does not work in Cluster

[paulavijit]

I created the Elastic Cluster following the Deploy steps:

kubectl create -f es-discovery-svc.yaml kubectl create -f es-svc.yaml kubectl create -f es-master.yaml kubectl rollout status -f es-master.yaml

kubectl create -f es-ingest-svc.yaml kubectl create -f es-ingest.yaml kubectl rollout status -f es-ingest.yaml

kubectl create -f es-data.yaml kubectl rollout status -f es-data.yaml

The changes I made was to above YAML files were:

1. Uncomment "type: LoadBalancer" in es-svc.yaml and es-ingest-svc.yaml

2. Add following in es-ingest.yaml

   • name: ES_PLUGINS_INSTALL value: "ingest-attachment"

The cluster deployed successfully. Cluster Health was green:

curl http://a8be743b3a73811e8a9d10a6601a7141-2077302965.us-east-1.elb.amazonaws.com:9200/_cluster/health?pretty { "cluster_name" : "myesdb", "status" : "green", "timed_out" : false, "number_of_nodes" : 7, "number_of_data_nodes" : 2, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

The ingest-attachment plugin was successfully installed:

curl http://a8be743b3a73811e8a9d10a6601a7141-2077302965.us-east-1.elb.amazonaws.com:9200/_cat/plugins?v name component version es-ingest-595954d85c-7hskx ingest-attachment 6.3.2 es-ingest-595954d85c-vwn4j ingest-attachment 6.3.2

But when I try to create a ingest pipeline, it fails:

curl -H 'Content-Type: application/json' -X PUT http://a8be743b3a73811e8a9d10a6601a7141-2077302965.us-east-1.elb.amazonaws.com:9200/_ingest/pipeline/ng-attachment?pretty -d '

{ "description" : "Extract attachment information encoded in Base64 with UTF-8 charset", "processors" : [ { "attachment" : { "field" : "data" } } ] }' { "error" : { "root_cause" : [ { "type" : "parse_exception", "reason" : "No processor type exists with name [attachment]", "header" : { "processor_type" : "attachment" } } ], "type" : "parse_exception", "reason" : "No processor type exists with name [attachment]", "header" : { "processor_type" : "attachment" } }, "status" : 400 }

The same ingest pipeline creation works fine if I have a single node with NODE_MASTER, NODE_INGEST and NODE_DATA all set to true in Kubernetes or if I do "docker run" with the image, quay.io/pires/docker-elasticsearch-kubernetes:6.3.2.

Any help is highly appreciated. Thanks in advance.

[paulavijit]

For ingest-attachment plugin to work in a cluster, it has to be installed on all nodes irrespective of it is an ingest node or not.

That fixed the issue.