search

opensearch-project / data-prepper

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
259 stars 190 forks source link

raw-span-policy delete action #1751

Open arulselvan opened 2 years ago

arulselvan commented 2 years ago

how to replace/override raw-span-policy? I need to include the delete action for otel-v1-apm-span-* during data-prepper setup time

thanks!

sshivanii commented 2 years ago

Hi @arulselvan,

The index reserved will always be otel-v1-apm-span when using trace_analytics_raw index_type, and it cannot be replaced. For reference, here's the configuration for Raw Span Trace Analytics. However, an alternative suggestion would be to switch to custom index_type and then specify the index you want to create. The default index_type is custom so specifying the index should be sufficient, in the following way:

pipeline:
  ...
  sink:
    opensearch:
      hosts: ["https://localhost:9200"]
      cert: path/to/cert
      username: YOUR_USERNAME_HERE
      password: YOUR_PASSWORD_HERE
      index: my-index-name
dlvenable commented 2 years ago

@arulselvan ,

Have you been able to get past this issue? If you are still needing the delete action, can you elaborate on your use-case here?

arulselvan commented 1 year ago

@dlvenable, yes I was able to pass through, during my setup process I've updated the default raw-span-policy using OpenSearch API like below

Get data Prepper SeqNo and PrimaryTerm Using Get API call and then called the Update API

curl --location --request GET 'http://10.81.211.22:5601/api/ism/policies/raw-span-policy'

curl --location --request PUT 'http://10.81.211.22:5601/api/ism/policies/raw-span-policy?seqNo=342071&primaryTerm=5' \ --header 'osd-version: 1.3.1' \ --header 'Content-Type: application/json' \ --data-raw Note : * data-raw will contain updated policy json with additional state and actions like delete

My use case: while spinning up data-prepare container, if we are able to override with docker volume like the below config, I think don't require an explicit update API call like above for the policy. Might be @sshivanii answer would be the solution I'll check it out later this week, thanks a lot!

image