[Cisco Duo] Integration updates

chemamartinez commented 1 week ago

Proposed commit message

This pull request contains several changes for the Cisco Duo integration:

Added new data stream telephony_v2 to support the new v2 API endpoint.
Updated the auth data stream to migrate from the HTTPJSON to CEL.

Both data streams above include a CEL program to make requests to the v2 version of the Cisco Duo API, following the next specifications:
Review documentation and dashboards

The upgrade process has been tested manually to verify that changes in current data streams don't break current users during upgrades.

Data streams that use the new CEL inputs are disabled by default. For the case of the auth data stream, users that are using it will have to enable it again when upgrading the integration, because of the migration from httpjson to CEL.

Checklist

[x] I have reviewed tips for building integrations and this pull request is aligned with them.
[x] I have verified that all data streams collect metrics or logs.
[x] I have added an entry to my package's changelog.yml file.
[x] I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Added pipeline and system tests for affected data streams.

Run asset tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME                                                          │ RESULT │ TIME ELAPSED │
├───────────┼────────────────────┼───────────┼────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-5a0b80af-49ad-42ee-89b7-c89faa927826 is loaded │ PASS   │      1.167µs │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-7a135061-78a3-45d9-951b-4b9b665fa729 is loaded │ PASS   │        625ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-b386f94c-0856-4508-ba08-a525a2f3b70f is loaded │ PASS   │        417ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-c3336a66-68ff-4bcd-95ff-fb388793f721 is loaded │ PASS   │        542ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-e91470e5-2ded-4ff1-8bb5-24e06b949c1d is loaded │ PASS   │        416ns │
│ cisco_duo │                    │ asset     │ map cisco_duo-158c0e80-148c-11ec-9386-31989719f9db is loaded       │ PASS   │        375ns │
│ cisco_duo │ admin              │ asset     │ index_template logs-cisco_duo.admin is loaded                      │ PASS   │        291ns │
│ cisco_duo │ admin              │ asset     │ ingest_pipeline logs-cisco_duo.admin-2.0.0 is loaded               │ PASS   │        333ns │
│ cisco_duo │ auth               │ asset     │ index_template logs-cisco_duo.auth is loaded                       │ PASS   │        375ns │
│ cisco_duo │ auth               │ asset     │ ingest_pipeline logs-cisco_duo.auth-2.0.0 is loaded                │ PASS   │        250ns │
│ cisco_duo │ offline_enrollment │ asset     │ index_template logs-cisco_duo.offline_enrollment is loaded         │ PASS   │        417ns │
│ cisco_duo │ offline_enrollment │ asset     │ ingest_pipeline logs-cisco_duo.offline_enrollment-2.0.0 is loaded  │ PASS   │        334ns │
│ cisco_duo │ summary            │ asset     │ index_template logs-cisco_duo.summary is loaded                    │ PASS   │        416ns │
│ cisco_duo │ summary            │ asset     │ ingest_pipeline logs-cisco_duo.summary-2.0.0 is loaded             │ PASS   │        250ns │
│ cisco_duo │ telephony          │ asset     │ index_template logs-cisco_duo.telephony is loaded                  │ PASS   │        291ns │
│ cisco_duo │ telephony          │ asset     │ ingest_pipeline logs-cisco_duo.telephony-2.0.0 is loaded           │ PASS   │        250ns │
│ cisco_duo │ telephony_v2       │ asset     │ index_template logs-cisco_duo.telephony_v2 is loaded               │ PASS   │        416ns │
│ cisco_duo │ telephony_v2       │ asset     │ ingest_pipeline logs-cisco_duo.telephony_v2-2.0.0 is loaded        │ PASS   │        250ns │
╰───────────┴────────────────────┴───────────┴────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_duo - END   ---
Done
Run pipeline tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME                                              │ RESULT │ TIME ELAPSED │
├───────────┼────────────────────┼───────────┼────────────────────────────────────────────────────────┼────────┼──────────────┤
│ cisco_duo │ admin              │ pipeline  │ (ingest pipeline warnings test-admin.log)              │ PASS   │    214.379ms │
│ cisco_duo │ admin              │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │ 203.127083ms │
│ cisco_duo │ admin              │ pipeline  │ test-admin.log                                         │ PASS   │  168.91625ms │
│ cisco_duo │ admin              │ pipeline  │ test-empty.log                                         │ PASS   │  41.405792ms │
│ cisco_duo │ auth               │ pipeline  │ (ingest pipeline warnings test-auth.log)               │ PASS   │ 213.616375ms │
│ cisco_duo │ auth               │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │ 196.946084ms │
│ cisco_duo │ auth               │ pipeline  │ test-auth.log                                          │ PASS   │ 351.136125ms │
│ cisco_duo │ auth               │ pipeline  │ test-empty.log                                         │ PASS   │  43.347542ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │    196.039ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ (ingest pipeline warnings test-offline-enrollment.log) │ PASS   │ 222.296084ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ test-empty.log                                         │ PASS   │  39.445417ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ test-offline-enrollment.log                            │ PASS   │  44.205709ms │
│ cisco_duo │ summary            │ pipeline  │ (ingest pipeline warnings test-summary.log)            │ PASS   │ 220.213792ms │
│ cisco_duo │ summary            │ pipeline  │ test-summary.log                                       │ PASS   │  39.650167ms │
│ cisco_duo │ telephony          │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │  225.43975ms │
│ cisco_duo │ telephony          │ pipeline  │ (ingest pipeline warnings test-telephony.log)          │ PASS   │ 198.661417ms │
│ cisco_duo │ telephony          │ pipeline  │ test-empty.log                                         │ PASS   │   39.71475ms │
│ cisco_duo │ telephony          │ pipeline  │ test-telephony.log                                     │ PASS   │  43.029333ms │
│ cisco_duo │ telephony_v2       │ pipeline  │ (ingest pipeline warnings test-telephony-v2.log)       │ PASS   │ 198.065125ms │
│ cisco_duo │ telephony_v2       │ pipeline  │ test-telephony-v2.log                                  │ PASS   │  43.282834ms │
╰───────────┴────────────────────┴───────────┴────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_duo - END   ---
Done
Run static tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├───────────┼────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ cisco_duo │ admin              │ static    │ Verify sample_event.json │ PASS   │  46.128333ms │
│ cisco_duo │ auth               │ static    │ Verify sample_event.json │ PASS   │  59.536083ms │
│ cisco_duo │ offline_enrollment │ static    │ Verify sample_event.json │ PASS   │  40.585333ms │
│ cisco_duo │ summary            │ static    │ Verify sample_event.json │ PASS   │  37.666792ms │
│ cisco_duo │ telephony          │ static    │ Verify sample_event.json │ PASS   │  37.640792ms │
│ cisco_duo │ telephony_v2       │ static    │ Verify sample_event.json │ PASS   │    37.3455ms │
╰───────────┴────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_duo - END   ---
Done
Run system tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├───────────┼────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ cisco_duo │ admin              │ system    │ default   │ PASS   │ 35.644630125s │
│ cisco_duo │ auth               │ system    │ default   │ PASS   │ 36.851069708s │
│ cisco_duo │ offline_enrollment │ system    │ default   │ PASS   │ 32.721470209s │
│ cisco_duo │ summary            │ system    │ default   │ PASS   │ 31.743466584s │
│ cisco_duo │ telephony          │ system    │ default   │ PASS   │ 31.538269666s │
│ cisco_duo │ telephony_v2       │ system    │ default   │ PASS   │ 35.694825375s │
╰───────────┴────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: cisco_duo - END   ---
Done