Elastic package with transform does not create the required destination index

[r00tu53r]

A package with a transform does not create the required destination index. When installing the package kibana displays the following error -

Error installing crowdstrike 1.4.0: runtime_exception: [runtime_exception] Reason: Could not create destination index [transform-crowdstrike.fdrv2-aidmaster] for transform [crowdstrike.fdrv2_aidmaster-transform-1.4.0]

Docker logs from the kibana instance -

[2022-07-18T11:48:20.367+00:00][WARN ][plugins.fleet] Failure to install package [crowdstrike]: [ResponseError: runtime_exception: [runtime_exception] Reason: Could not create destination index [transform-crowdstrike.fdr-useridentity] for transform [crowdstrike.fdr_useridentity-transform-1.4.0]]
[2022-07-18T11:48:20.368+00:00][ERROR][plugins.fleet] uninstalling crowdstrike-1.4.0 after error installing: [ResponseError: runtime_exception: [runtime_exception] Reason: Could not create destination index [transform-crowdstrike.fdr-useridentity] for transform [crowdstrike.fdr_useridentity-transform-1.4.0]]
[2022-07-18T11:48:21.155+00:00][INFO ][plugins.fleet] Deleting currently installed transform ids crowdstrike.fdr_useridentity-transform-1.4.0
[2022-07-18T11:48:21.156+00:00][INFO ][plugins.fleet] Deleting currently installed transform ids crowdstrike.fdrv2_aidmaster-transform-1.4.0
[2022-07-18T11:48:21.196+00:00][INFO ][plugins.fleet] Deleted: crowdstrike.fdr_useridentity-transform-1.4.0
[2022-07-18T11:48:21.204+00:00][ERROR][plugins.fleet] ResponseError: security_exception: [security_exception] Reason: action [indices:admin/delete] is unauthorized for service account [elastic/kibana] on indices [transform-crowdstrike.fdr-useridentity], this action is granted by the index privileges [delete_index,manage,all]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/src/core/server/elasticsearch/client/create_transport.js:58:16)
    at /usr/share/kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/remove.js:62:9
    at async Promise.all (index 0)
    at deleteTransforms (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/remove.js:41:3)
    at async Promise.all (index 6)
    at deleteAssets (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.js:178:5)
    at removeInstallation (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.js:70:3)
    at handleInstallPackageFailure (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:238:7)
    at /usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:402:7
    at installPackage (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:556:22)
    at ensureInstalledPackage (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:175:25)
    at async Promise.all (index 0)
    at PackagePolicyService.create (/usr/share/kibana/x-pack/plugins/fleet/server/services/package_policy.js:110:33)
    at createPackagePolicyHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/package_policy/handlers.js:107:27)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
[2022-07-18T11:48:21.207+00:00][INFO ][plugins.fleet] Deleted: crowdstrike.fdrv2_aidmaster-transform-1.4.0
[2022-07-18T11:48:21.245+00:00][ERROR][plugins.fleet] failed to uninstall or rollback package after installation error Error: Saved object [epm-packages/crowdstrike] not found
[2022-07-18T11:48:21.246+00:00][ERROR][plugins.fleet] Error: Error installing crowdstrike 1.4.0: runtime_exception: [runtime_exception] Reason: Could not create destination index [transform-crowdstrike.fdr-useridentity] for transform [crowdstrike.fdr_useridentity-transform-1.4.0]
    at ensureInstalledPackage (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:199:11)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at PackagePolicyService.create (/usr/share/kibana/x-pack/plugins/fleet/server/services/package_policy.js:110:33)
    at createPackagePolicyHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/package_policy/handlers.js:107:27)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

• Relates: #895

[jsoriano]

I think that thsi is not supported in Kibana/Fleet yet, see https://github.com/elastic/kibana/issues/134321

[susan-shu-c]

I am encountering the same issue working on https://github.com/elastic/security-ml/issues/83, commenting here for documentation. I am aware of and following

• https://github.com/elastic/kibana/issues/134321

As a temporary workaround, I created the destination index manually before installing the package with a PUT command.

The curious thing is that for one (out of 2) of the transforms in the package I'm developing, it does seem to be able to create the destination index:

[szeitlin]

@susan-shu-c Sorry I didn't think of this earlier, but I wonder if it's a permissions issue with creating the other index?

[susan-shu-c]

@szeitlin I'm curious if that's the case - I did re-name both indices to a pattern that elastic/kibana service account has access to (.alerts* pattern):

• .alerts-security.host-risk-score (destination for the 1st transform in this package, named "pivot transform")
• .alerts-security.host-risk-score-latest (destination for 2nd transform, named "latest transform")

[Edit with new information] That service account has kibana_system role if I recall correctly

If I do not use a naming pattern that service account has access to, there is this following error when I install the package:

susan@computer host_risk_score % elastic-package install
Install the package
Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"security_exception: [security_exception] Reason: action [indices:admin/delete] is unauthorized for service account [elastic/kibana] on indices [ml_host_risk_score_default], this action is granted by the index privileges [delete_index,manage,all]"}