Kibana - Upgrading package causes ingest manager error

[ofiriro3]

Describe the bug: When running Kibana locally and trying to upgrade the CSP integration version, the ingest manager throws an error.

Steps to reproduce:

1. Run the stack locally (elastic-package stack up --version 8.5.0-SNAPSHOT -v -d)
2. Kill Kibana and run Kibana locally
3. Install the CSPM integration.
4. Go back to the integration repository and create a new CSP integration version. (Can be done by updating the manifest.yml version.
5. Build the integration and run the local EPR again - from the packages/cloud_security_posture path run elastic-package build and then elastic-package stack up -v -d --services package-registry
6. Go back to the Kibana page, go to the CSMP integration > Assets, and upgrade the integration.
7. You will receive this error

   IngestManagerError: error deleting pipeline logs-cloud_security_posture.findings-0.0.30: ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: pipeline [logs-cloud_security_posture.findings-0.0.30] cannot be deleted because it is the default pipeline for 1 index(es) including [logs-cloud_security_posture.findings_latest-default]
   at deletePipeline (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/remove.ts:54:15)
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5)
   at async Promise.all (index 0)
   at deletePreviousPipelines (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/remove.ts:29:5)
   at _installPackage (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/packages/_install_package.ts:233:22)
   at installPackageFromRegistry (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:376:12)
   at installPackage (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:589:22)
   at installPackageFromRegistryHandler (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/routes/epm/handlers.ts:271:15)
   at Router.handle (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:163:30)
   at handler (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:124:50)
   at exports.Manager.execute (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
   at Object.internals.handler (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
   at exports.execute (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
   at Request._lifecycle (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
   at Request._execute (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
   [2022-09-07T15:11:03.857+03:00][INFO ][plugins.fleet] Package policy upgraded successfully

Expected behavior: The ingest manager should not throw errors.

Any additional context: Kibana commit - 875a624179a1191f4219da11aa9740b2f6c15f8b

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security Posture)

[kfirpeled]

We should not use the ingest pipeline for the latest findings index

[ofiriro3]

I see that when deleting the asset, there is also an error - I am wondering if it can be related:

Error: error deleting component template logs-cloud_security_posture.findings@custom
    at deleteComponentTemplate (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.ts:202:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 2)
    at deleteAssets (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.ts:173:5)
    at removeInstallation (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.ts:64:3)
    at deletePackageHandler (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/x-pack/plugins/fleet/server/routes/epm/handlers.ts:380:17)
    at Router.handle (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:163:30)
    at handler (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:124:50)
    at exports.Manager.execute (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/Users/ofirlapid/dvlp/kibana/forks/ofiriro3/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

[ofiriro3]

After multiple attempts to reproduce, it seems that the bug is not deterministic.

It is a bit difficult to understand what is the root cause for this bug to happen

[ofiriro3]

It appears that the root cause of this problem was that the latest index used the same ingest pipeline as the findings index. I created a new ingest pipeline for the latest index (as @CohenIdo suggested). It seems to solve it. I am now verifying it