sei: Update packages to format_version 3.0.0

[andrewkroh]

This meta issue tracks the work related to updating packages to format_version: 3.0.0.

• [x] @kgeller Fix https://github.com/elastic/integrations/issues/7808
• [x] Remove version from ingest pipelines - https://github.com/elastic/integrations/pull/7807
• [x] hashicorp_vault.metrics - Resolve errors related to the dynamic mapping fields - https://github.com/elastic/integrations/issues/7811
• [x] Add DLM to packages that have ILM
  • [x] ti_maltiverse https://github.com/elastic/integrations/pull/7851
  • [x] ti_anomali https://github.com/elastic/integrations/pull/7849
  • [x] ti_recorded_future https://github.com/elastic/integrations/pull/7848
  • [x] ti_rapid7 https://github.com/elastic/integrations/pull/7910
• [x] Fix dotted YAML keys
  • https://github.com/elastic/integrations/pull/7800
  • https://github.com/elastic/integrations/pull/7804
  • https://github.com/elastic/integrations/pull/7803
  • https://github.com/elastic/integrations/pull/7801
  • The remaining keys (all under constraints) will be handled in bulk via ecs-update.
• [x] @marc-gr Change format_version 3.0.0 https://github.com/elastic/integrations/pull/7883
  • This will be handled in bulk via ecs-update.
  • We also need to include owner.type: elastic as part of these changes (see: #6569)
• [x] Integrations requiring manual tests: https://github.com/elastic/integrations/issues/7814
• No elastic.capabilities constraints should be added. This will ensure our packages are available across observability and security projects.

The general command to update packages in bulk is:

go run github.com/andrewkroh/go-examples/ecs-update@main -format-version=3.0.0 -fix-dotted-yaml-keys -owner elastic/security-external-integrations packages/*

but we want to exclude non-deprecated rsa2elk packages from using format_version 3.0.0 so use this command and glob:

zsh
setopt extendedglob
go run github.com/andrewkroh/go-examples/ecs-update@main  \
  -format-version=3.0.0 \
  -fix-dotted-yaml-keys \
  -add-owner-type \
  -owner elastic/security-external-integrations \
  packages/*~packages/cylance~packages/fortinet_forticlient~packages/imperva~packages/netscout~packages/radware~packages/squid

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

116 of the SEI owned packages can be changed to 3.0.0 without issue. The 24 below have some kind of issue that needs resolved. ~This does account for the DLM changes.~

• 1password
• azure_frontdoor
• carbonblack_edr
• cisco_aironet
• cisco_meraki
• cloudflare_logpush
• crowdstrike
• cylance
• fireeye
• fortinet_forticlient
• hashicorp_vault
• imperva
• infoblox_nios
• juniper_srx
• lyve_cloud
• netflow
• netscout
• panw
• radware
• sentinel_one
• squid
• ti_anomali
• ti_maltiverse
• ti_rapid7
• ti_recorded_future
• trend_micro_vision_one

[P1llus]

Added https://github.com/elastic/integrations/issues/7814 to the description as well.

[ebeahan]

No elastic.capabilities constraints should be added. This will ensure our packages are available across observability and security projects.

As we're testing for this update and looking beyond, do we consider having testing in place for both Observability and Security project types?

such as:

# provision security type
elastic-package stack up -v --version 8.10.1 --provider serverless -U stack.serverless.type=security
# test
elastic-package test -v
# spin it down
elastic-package stack down -v
# repeat with observability project
elastic-package stack up -v --version 8.10.1 --provider serverless -U stack.serverless.type=observability
elastic-package test -v
elastic-package stack down -v

[andrewkroh]

After #8025 merges all of the packages owned by SEI (minus deprecated and rsa2elk) will be updated to use format_version 3.0.0.

[ebeahan]

With #8025 merged, any other work to track here? Or is this issue ready to close?