search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
38 stars 448 forks source link

Incorrect `null` handling in `if` conditions and elsewhere #8646

Open chrisberkhout opened 11 months ago

chrisberkhout commented 11 months ago

The problem

In the Painless scripting language it is an error for the result of an if condition to be null. This matches Java but differs from many other languages, in which null is falsy.

The null safe operator ?. is often used to attempt field access or method invocation on objects that may be null, but it doesn't turn nulls in to non-nulls.

After discovering incorrect null handling in my own code, I checked elsewhere and found many other occurrences.

Demonstration of the problem in an if statement ``` POST _scripts/painless/_execute { "script": { "params": { "condition": null }, "source": """ if (params?.condition) { return "it was true"; } else { return "it was false"; } """ } } ```
Demonstration of the problem in an if processor option ``` PUT _ingest/pipeline/my-pipeline { "processors": [ { "set": { "field": "testfield", "value": "it was true", "if": "ctx?.condition" } } ] } POST _ingest/pipeline/my-pipeline/_simulate { "docs": [ { "_source": { "condition": null } } ] } ```

Identified cases

In 3431bbea785a10b3849bdc1899896e5a723f22a8 I looked for lines with an if condition, the null safe operator, and without an equality or instanceof check:

ag -G '\.(yml|hbs)' 'if.*\?\.' | grep -v '==' | grep -v '!=' | grep -v instanceof

And also for lines with a for loop and the null safe operator:

ag -G '\.(yml|hbs)' 'for ?\(.*\?\.'

About half of the candidates were actual problems: 225 lines spread across 39 integrations.

Summary of identified cases ``` 37 lastpass 36 nagios_xi 16 cisco_ise 13 zeek 13 cisco_secure_endpoint 12 ping_one 9 zoom 8 panw_cortex_xdr 7 juniper_srx 7 aws 5 hid_bravura_monitor 5 cisco_nexus 4 trendmicro 4 proofpoint_tap 4 github 4 box_events 4 atlassian_jira 4 atlassian_confluence 3 jamf_compliance_reporter 3 infoblox_nios 2 tenable_sc 2 pfsense 2 mimecast 2 fortinet_fortigate 2 cloudflare_logpush 2 cloudflare 2 cisco_asa 2 atlassian_bitbucket 1 vectra_detect 1 ti_cif3 1 suricata 1 salesforce 1 microsoft_defender_endpoint 1 mattermost 1 elasticsearch 1 elastic_package_registry 1 cyberarkpas 1 cisco_meraki 1 cisco_ftd 225 TOTAL ```
List of identified cases ``` packages/atlassian_bitbucket/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:376: for (def j = 0; j < ctx.bitbucket?.audit?.affected_objects.length; j++) { packages/atlassian_bitbucket/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:396: for (def j = 0; j < ctx.bitbucket?.audit?.changed_values.length; j++) { packages/atlassian_confluence/data_stream/audit/elasticsearch/ingest_pipeline/cloud.yml:59: for (def j = 0; j < ctx.confluence?.audit?.affected_objects.length; j++) { packages/atlassian_confluence/data_stream/audit/elasticsearch/ingest_pipeline/cloud.yml:67: for (def j = 0; j < ctx.confluence?.audit?.changed_values.length; j++) { packages/atlassian_confluence/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:296: for (def j = 0; j < ctx.confluence?.audit?.affected_objects.length; j++) { packages/atlassian_confluence/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:345: for (def j = 0; j < ctx.confluence?.audit?.changed_values.length; j++) { packages/atlassian_jira/data_stream/audit/elasticsearch/ingest_pipeline/cloud.yml:54: for (def j = 0; j < ctx.jira?.audit?.affected_objects.length; j++) { packages/atlassian_jira/data_stream/audit/elasticsearch/ingest_pipeline/cloud.yml:62: for (def j = 0; j < ctx.jira?.audit?.changed_values.length; j++) { packages/atlassian_jira/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:308: for (def j = 0; j < ctx.jira?.audit?.affected_objects.length; j++) { packages/atlassian_jira/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:327: for (def j = 0; j < ctx.jira?.audit?.changed_values.length; j++) { packages/aws/data_stream/cloudfront_logs/elasticsearch/ingest_pipeline/default.yml:168: if: ctx.http?.response?.status_code >= 400 packages/aws/data_stream/cloudfront_logs/elasticsearch/ingest_pipeline/default.yml:172: if: ctx.http?.response?.status_code < 400 && ctx.http?.response?.status_code > 000 packages/aws/data_stream/route53_public_logs/elasticsearch/ingest_pipeline/default.yml:56: if: '!ctx._tmp?.question.endsWith("in-addr.arpa")' packages/aws/data_stream/route53_resolver_logs/elasticsearch/ingest_pipeline/default.yml:228: if(ctx.dns?.question?.name.contains(".in-addr.arpa")) { packages/aws/data_stream/route53_resolver_logs/elasticsearch/ingest_pipeline/default.yml:235: } else if (ctx.dns?.question?.name.contains(".ip6.arpa")) { packages/aws/data_stream/route53_resolver_logs/elasticsearch/ingest_pipeline/default.yml:60: if: '!ctx.json?.query_name.endsWith("in-addr.arpa") && !ctx.json?.query_name.endsWith("ip6.arpa")' packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml:28: if: 'ctx.event?.original.startsWith("version") || ctx.event?.original.startsWith("instance-id")' packages/box_events/data_stream/events/elasticsearch/ingest_pipeline/default.yml:708: if (ctx.event?.action.equals("SHIELD_ALERT")) { packages/box_events/data_stream/events/elasticsearch/ingest_pipeline/default.yml:742: for (session in ctx.box.additional_details?.shield_alert?.alert_summary?.sessions) { packages/box_events/data_stream/events/elasticsearch/ingest_pipeline/default.yml:743: if (session.session_type?.equals("suspicious")) { packages/box_events/data_stream/events/elasticsearch/ingest_pipeline/default.yml:786: for (alert_activity in ctx.box.additional_details?.shield_alert?.alert_summary?.alert_activities) { packages/cisco_asa/data_stream/log/elasticsearch/ingest_pipeline/default.yml:2483: } else if (ctx?.event?.action.startsWith('connection-')) { packages/cisco_asa/data_stream/log/elasticsearch/ingest_pipeline/default.yml:2502: } else if (ctx?.event?.action.startsWith('connection-')) { packages/cisco_ftd/data_stream/log/elasticsearch/ingest_pipeline/default.yml:2117: if: "ctx?.tags?.contains('private_is_internal') && packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_ad_connector.yml:13: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_administrative_and_operational_audit.yml:13: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_authentication_flow_diagnostics.yml:16: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_failed_attempts.yml:13: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_guest.yml:19: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_identity_stores_diagnostics.yml:13: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_internal_operations_diagnostics.yml:16: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_mydevices.yml:19: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_passed_authentications.yml:13: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_policy_diagnostics.yml:19: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_posture_and_client_provisioning_audit.yml:19: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_radius_accounting.yml:19: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_radius_diagnostics.yml:14: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_system_statistics.yml:19: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_tacacs_accounting.yml:19: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_threat_centric_nac.yml:16: if: ctx.cisco_ise?.log?.segment?.number > 0 packages/cisco_meraki/data_stream/log/elasticsearch/ingest_pipeline/events.yml:44: if: ctx?.msgtype.startsWith("type=") packages/cisco_nexus/data_stream/log/elasticsearch/ingest_pipeline/pipeline_extract_message.yml:19: if: "['VSHD_SYSLOG_CONFIG_I','DETECT_MULTIPLE_PEERS','UPDOWN','CFGWRITE_STARTED','LINEPROTO'].contains(ctx.event?.code.toUpperCase())" packages/cisco_nexus/data_stream/log/elasticsearch/ingest_pipeline/pipeline_extract_message.yml:30: if: "['SYSTEM_MSG'].contains(ctx.event?.code.toUpperCase())" packages/cisco_nexus/data_stream/log/elasticsearch/ingest_pipeline/pipeline_extract_message.yml:41: if: "['INVAL_IP','L2FM_MAC_MOVE2','DUPLEX_MISMATCH','NATIVE_VLAN_MISMATCH','THRESHOLD_VIOLATION'].contains(ctx.event?.code.toUpperCase())" packages/cisco_nexus/data_stream/log/elasticsearch/ingest_pipeline/pipeline_extract_message.yml:51: if: "['LOGIN_SUCCESS','LOGOUT','LOGOUT_C6K'].contains(ctx.event?.code.toUpperCase())" packages/cisco_nexus/data_stream/log/elasticsearch/ingest_pipeline/pipeline_extract_message.yml:6: if: "['IF_DOWN_ADMIN_DOWN','IF_ADMIN_UP','SPEED','IF_DUPLEX','IF_RX_FLOW_CONTROL','IF_TX_FLOW_CONTROL','IF_UP','IF_XCVR_WARNING'].contains(ctx.event?.code.toUpperCase())" packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:135: for (technique in ctx?.cisco?.secure_endpoint?.techniques) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:137: if (!ctx?.threat.technique.name.contains(technique.name)) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:142: if (!ctx?.threat.technique.id.contains(technique.external_id)) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:147: if (!ctx?.threat.technique.reference.contains(technique.mitre_url)) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:171: for (tactic in ctx?.cisco?.secure_endpoint?.tactics) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:173: if (!ctx?.threat.tactic.name.contains(tactic.name)) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:178: if (!ctx?.threat.tactic.id.contains(tactic.external_id)) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:183: if (!ctx?.threat.tactic.reference.contains(tactic.mitre_url)) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:217: if (nameArray?.length > 0) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:529: for (addr in ctx?.cisco?.secure_endpoint?.computer?.network_addresses) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:531: if (!ctx?.related?.ip.contains(addr.ip)) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:546: for (addr in ctx?.cisco?.secure_endpoint?.computer?.network_addresses) { packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml:548: if (!ctx?.cisco?.secure_endpoint?.related?.mac.contains(addr.mac)) { packages/cloudflare/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:40: if: ctx.json?.action?.result packages/cloudflare/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:44: if: "!ctx.json?.action?.result" packages/cloudflare_logpush/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:76: if: ctx.json?.ActionResult packages/cloudflare_logpush/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:80: if: '!ctx.json?.ActionResult' packages/cyberarkpas/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:351: if: 'ctx.event?.severity > 6' packages/elastic_package_registry/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml:76: if: ctx?.package_registry?.containsKey("start_time_seconds") packages/elasticsearch/data_stream/audit/elasticsearch/ingest_pipeline/pipeline-json.yml:13: if: '!ctx.elasticsearch.audit.containsKey("type") && !["rest", "transport", "ip_filter", "security_config_change"].contains(ctx.elasticsearch?.audit?.event?.type)' packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/default.yml:68: for (def i = 0; i < arr?.length; i++) { packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/event.yml:203: if: ctx.rule?.description.startsWith('Dynamic address updated') packages/github/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:107: if: ctx.event?.action.startsWith("org.") packages/github/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:191: if: 'ctx._temp?.action.contains("create") || ctx._temp?.action.contains("add")' packages/github/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:195: if: 'ctx._temp?.action.contains("delete") || ctx._temp?.action.contains("remove")' packages/github/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:97: if: ctx.event?.action.startsWith("team.") packages/hid_bravura_monitor/data_stream/log/elasticsearch/ingest_pipeline/default.yml:27: if: ctx?.msg.contains('last message repeated') packages/hid_bravura_monitor/data_stream/log/elasticsearch/ingest_pipeline/default.yml:37: if: ctx?.log?.level.contains('Perf') packages/hid_bravura_monitor/data_stream/log/elasticsearch/ingest_pipeline/default.yml:42: if: ctx?.log?.level.contains('Perf') packages/hid_bravura_monitor/data_stream/log/elasticsearch/ingest_pipeline/default.yml:45: if: ctx?.log?.level.contains('Perf') packages/hid_bravura_monitor/data_stream/log/elasticsearch/ingest_pipeline/default.yml:61: if: ctx?.log?.level.contains('Perf') packages/infoblox_nios/data_stream/log/elasticsearch/ingest_pipeline/pipeline_dns.yml:115: for(data in ctx.dns?.answers?.data){ packages/infoblox_nios/data_stream/log/elasticsearch/ingest_pipeline/pipeline_dns.yml:132: for(name in ctx.dns?.answers?.name){ packages/infoblox_nios/data_stream/log/elasticsearch/ingest_pipeline/pipeline_dns.yml:90: for (def i = 0; i < arr?.length; i++) { packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/pipeline_aue_execve.yml:137: for (Map.Entry m : ctx.json?.args.entrySet()) { packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/pipeline_aue_posix_spawn.yml:43: for (Map.Entry m : ctx.json?.args.entrySet()) { packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/pipeline_prohibited_app_blocked.yml:266: for (Map.Entry m : ctx.json?.args.entrySet()) { packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/flow.yml:29: if: "ctx.juniper?.srx?.tag.endsWith('CREATE') || ctx.juniper?.srx?.tag.endsWith('UPDATE') || ctx.juniper?.srx?.tag.endsWith('CREATE_LS') || ctx.juniper?.srx?.tag.endsWith('UPDATE_LS')" packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/flow.yml:36: if: "ctx.juniper?.srx?.tag.endsWith('CLOSE') || ctx.juniper?.srx?.tag.endsWith('CLOSE_LS')" packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/flow.yml:42: if: "ctx.juniper?.srx?.tag.endsWith('DENY') || ctx.juniper?.srx?.tag.endsWith('DENY_LS')" packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/flow.yml:46: if: "ctx.juniper?.srx?.tag.endsWith('CREATE') || ctx.juniper?.srx?.tag.endsWith('UPDATE') || ctx.juniper?.srx?.tag.endsWith('CREATE_LS') || ctx.juniper?.srx?.tag.endsWith('UPDATE_LS')" packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/flow.yml:50: if: "ctx.juniper?.srx?.tag.endsWith('CLOSE') || ctx.juniper?.srx?.tag.endsWith('CLOSE_LS')" packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/flow.yml:54: if: "ctx.juniper?.srx?.tag.endsWith('DENY') || ctx.juniper?.srx?.tag.endsWith('DENY_LS')" packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/ids.yml:73: if: "ctx.juniper?.srx?.attack_name.startsWith('Tunnel')" packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:146: if: ctx.event?.action?.contains('deleted sites') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:152: if: ctx.event?.action?.contains('employee invited') || ctx.event?.action?.contains('employee account created') || ctx.event?.action?.contains('employee account deleted') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:161: if (ctx.event?.action?.contains('limit shared folder')) { packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:168: if: ctx.event?.action?.contains('log in') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:174: if: ctx.event?.action?.contains('saml login') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:180: if: ctx.event?.action?.contains('failed login attempt') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:186: if: ctx.event?.action?.contains('login to admin console') || ctx.event?.action?.contains('make admin') || ctx.event?.action?.contains('remove admin') || ctx.event?.action?.contains('require password change') || ctx.event?.action?.contains('master password reset by super admin') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:192: if: ctx.event?.action?.contains('site added') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:198: if: ctx.event?.action?.contains('created shared folder') || ctx.event?.action?.contains('deleted shared folder') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:204: if: ctx.event?.action?.contains('add secure note') || ctx.event?.action?.contains('open secure note') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:211: if: ctx.event?.action?.contains('edit secure note') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:217: if: ctx.event?.action?.contains('add to shared folder') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:223: if: ctx.event?.action?.contains('create group') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:229: if: ctx.event?.action?.contains('adding user to group') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:235: if: ctx.event?.action?.contains('created lastpass account') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:241: if: ctx.event?.action?.contains('update folder permissions') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:247: if: ctx.event?.action?.contains('renamed shared folder') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:253: if: ctx.event?.action?.contains('move to shared folder') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:260: if: ctx.event?.action?.contains('move from shared folder') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:268: if: ctx.event?.action?.contains('delete shared sites') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:275: if: ctx.event?.action?.contains('limit shared folder') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:281: if: ctx.event?.action?.contains('removed from shared folder') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:290: if: ctx.event?.action?.contains('employee invited') || ctx.event?.action?.contains('employee account created') || ctx.event?.action?.contains('employee account deleted') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:301: if: "!(ctx.event?.action?.contains('employee invited') || ctx.event?.action?.contains('employee account created') || ctx.event?.action?.contains('employee account deleted'))" packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:40: if: ctx.event?.action?.contains('login verification email sent') || ctx.event?.action?.contains('log in') || ctx.event?.action?.contains('login to admin console') || ctx.event?.action?.contains('saml login') || ctx.event?.action?.contains('multifactor enabled') || ctx.event?.action?.contains('failed login attempt') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:44: if: ctx.event?.action?.contains('add policy') || ctx.event?.action?.contains('delete policy') || ctx.event?.action?.contains('enterprise api secret regenerated') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:48: if: ctx.event?.action?.contains('make admin') || ctx.event?.action?.contains('master password changed') || ctx.event?.action?.contains('master password reset by super admin') || ctx.event?.action?.contains('require password change') || ctx.event?.action?.contains('employee account created') || ctx.event?.action?.contains('created lastpass account') || ctx.event?.action?.contains('employee account deleted') || ctx.event?.action?.contains('remove admin') || ctx.event?.action?.contains('create group') || ctx.event?.action?.contains('adding user to group') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:52: if: ctx.event?.action?.contains('log in') || ctx.event?.action?.contains('login to admin console') || ctx.event?.action?.contains('saml login') || ctx.event?.action?.contains('open secure note') || ctx.event?.action?.contains('failed login attempt') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:56: if: ctx.event?.action?.contains('make admin') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:60: if: ctx.event?.action?.contains('enterprise api secret regenerated') || ctx.event?.action?.contains('update folder permissions') || ctx.event?.action?.contains('master password changed') || ctx.event?.action?.contains('master password reset by super admin') || ctx.event?.action?.contains('edit secure note') || ctx.event?.action?.contains('renamed shared folder') || ctx.event?.action?.contains('move to shared folder') || ctx.event?.action?.contains('move from shared folder') || ctx.event?.action?.contains('limit shared folder') || ctx.event?.action?.contains('update folder permissions') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:64: if: ctx.event?.action?.contains('employee account created') || ctx.event?.action?.contains('created lastpass account') || ctx.event?.action?.contains('created shared folder') || ctx.event?.action?.contains('add secure note') || ctx.event?.action?.contains('site added') || ctx.event?.action?.contains('add to shared folder') || ctx.event?.action?.contains('add policy') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:68: if: ctx.event?.action?.contains('deleted sites') || ctx.event?.action?.contains('delete policy') || ctx.event?.action?.contains('deleted shared folder') || ctx.event?.action?.contains('employee account deleted') || ctx.event?.action?.contains('remove admin') || ctx.event?.action?.contains('removed from shared folder') || ctx.event?.action?.contains('delete shared sites') || ctx.event?.action?.contains('deleted sites') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:72: if: ctx.event?.action?.contains('create group') || ctx.event?.action?.contains('adding user to group') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:76: if: ctx.event?.action?.contains('login verification email sent') || ctx.event?.action?.contains('get shared folder data') || ctx.event?.action?.contains('get user data') || ctx.event?.action?.contains('employee invited') || ctx.event?.action?.contains('reporting') || ctx.event?.action?.contains('require password change') || ctx.event?.action?.contains('multifactor enabled') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:80: if: ctx.event?.action?.contains('login verification email sent') || ctx.event?.action?.contains('log in') || ctx.event?.action?.contains('login to admin console') || ctx.event?.action?.contains('saml login') || ctx.event?.action?.contains('open secure note') || ctx.event?.action?.contains('make admin') || ctx.event?.action?.contains('enterprise api secret regenerated') || ctx.event?.action?.contains('update folder permissions') || ctx.event?.action?.contains('master password changed') || ctx.event?.action?.contains('master password reset by super admin') || ctx.event?.action?.contains('employee account created') || ctx.event?.action?.contains('created lastpass account') || ctx.event?.action?.contains('created shared folder') || ctx.event?.action?.contains('add secure note') || ctx.event?.action?.contains('site added') || ctx.event?.action?.contains('add to shared folder') || ctx.event?.action?.contains('add policy') || ctx.event?.action?.contains('deleted sites') || ctx.event?.action?.contains('delete policy') || ctx.event?.action?.contains('deleted shared folder') || ctx.event?.action?.contains('employee account deleted') || ctx.event?.action?.contains('remove admin') || ctx.event?.action?.contains('create group') || ctx.event?.action?.contains('adding user to group') || ctx.event?.action?.contains('get shared folder data') || ctx.event?.action?.contains('get user data') || ctx.event?.action?.contains('employee invited') || ctx.event?.action?.contains('edit secure note') || ctx.event?.action?.contains('renamed shared folder') || ctx.event?.action?.contains('move to shared folder') || ctx.event?.action?.contains('move from shared folder') || ctx.event?.action?.contains('limit shared folder') || ctx.event?.action?.contains('removed from shared folder') || ctx.event?.action?.contains('update folder permissions') || ctx.event?.action?.contains('delete shared sites') || ctx.event?.action?.contains('multifactor enabled') || ctx.event?.action?.contains('deleted sites') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:84: if: ctx.event?.action?.contains('failed login attempt') packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml:88: if: ctx.event?.action?.contains('require password change') || ctx.event?.action?.contains('reporting') packages/mattermost/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:323: if: "ctx.event?.category.contains('iam')" packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml:173: if: (ctx.json?.description).length() < 1020 packages/mimecast/data_stream/archive_search_logs/elasticsearch/ingest_pipeline/default.yml:82: if: 'ctx.user?.parts?.length > 1' packages/mimecast/data_stream/archive_search_logs/elasticsearch/ingest_pipeline/default.yml:86: if: 'ctx.user?.parts?.length > 1' packages/nagios_xi/data_stream/host/elasticsearch/ingest_pipeline/default.yml:262: if: ctx?.json?.current_state?.contains("0") packages/nagios_xi/data_stream/host/elasticsearch/ingest_pipeline/default.yml:268: if: ctx?.json?.current_state?.contains("1") packages/nagios_xi/data_stream/host/elasticsearch/ingest_pipeline/default.yml:274: if: ctx?.json?.current_state?.contains("2") packages/nagios_xi/data_stream/host/elasticsearch/ingest_pipeline/default.yml:280: if: ctx?.json?.current_state?.contains("3") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:242: if: ctx?.json?.current_state?.contains("0") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:248: if: ctx?.json?.current_state?.contains("1") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:254: if: ctx?.json?.current_state?.contains("2") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:260: if: ctx?.json?.current_state?.contains("3") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:398: if: ctx?.nagios_xi?.service?.check_command?.contains("check_local_users") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:404: if: ctx?.nagios_xi?.service?.containsKey("current_users") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:415: if: ctx?.nagios_xi?.service?.containsKey("current_users") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:423: if: ctx?.nagios_xi?.service?.check_command?.contains("check_local_load") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:429: if: ctx?.nagios_xi?.service?.containsKey("current_load") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:450: if: ctx?.nagios_xi?.service?.containsKey("current_load") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:458: if: ctx?.nagios_xi?.service?.check_command?.contains("check_ssh") || ctx?.nagios_xi?.service?.check_command?.contains("check_xi_service_ssh") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:464: if: ctx?.nagios_xi?.service?.containsKey("ssh") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:475: if: ctx?.nagios_xi?.service?.containsKey("ssh") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:483: if: ctx?.nagios_xi?.service?.check_command?.contains("check_ping") || ctx?.nagios_xi?.service?.check_command?.contains("check_xi_service_ping") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:489: if: ctx?.nagios_xi?.service?.containsKey("ping") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:505: if: ctx?.nagios_xi?.service?.containsKey("ping") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:513: if: ctx?.nagios_xi?.service?.check_command?.contains("check_local_swap") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:519: if: ctx?.nagios_xi?.service?.containsKey("swap_usage") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:532: if(ctx?.nagios_xi?.service?.containsKey("swap_usage")) { packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:541: if: ctx?.nagios_xi?.service?.containsKey("swap_usage") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:549: if: ctx?.nagios_xi?.service?.check_command?.contains("check_local_procs") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:555: if: ctx?.nagios_xi?.service?.containsKey("process") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:566: if: ctx?.nagios_xi?.service?.containsKey("process") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:574: if: ctx?.nagios_xi?.service?.check_command?.contains("check_http") || ctx?.nagios_xi?.service?.check_command?.contains("check_xi_service_http") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:580: if: ctx?.nagios_xi?.service?.containsKey("http") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:596: if: ctx?.nagios_xi?.service?.containsKey("http") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:604: if: ctx?.nagios_xi?.service?.check_command?.contains("check_local_disk") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:610: if: ctx?.nagios_xi?.service?.containsKey("root_partition") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:623: if(ctx?.nagios_xi?.service?.containsKey("root_partition")) { packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:632: if: ctx?.nagios_xi?.service?.containsKey("root_partition") packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:639: if: '!(ctx?.nagios_xi?.service?.containsKey("current_users") || ctx?.nagios_xi?.service?.containsKey("current_load") || ctx?.nagios_xi?.service?.containsKey("ssh") || ctx?.nagios_xi?.service?.containsKey("ping") || ctx?.nagios_xi?.service?.containsKey("swap_usage") || ctx?.nagios_xi?.service?.containsKey("process") || ctx?.nagios_xi?.service?.containsKey("http") || ctx?.nagios_xi?.service?.containsKey("root_partition"))' packages/nagios_xi/data_stream/service/elasticsearch/ingest_pipeline/default.yml:646: if: ctx?.event?.dataset?.contains("nagios_xi.custom") packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml:204: if (!ctx.threat?.technique?.id.contains(x)) { packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml:207: if (!ctx.threat?.technique?.name.contains(y)) { packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml:211: for (mitre_technique in ctx.panw_cortex?.xdr?.mitre_technique_id_and_name) { packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml:232: if (!ctx.threat?.tactic?.id.contains(x)) { packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml:235: if (!ctx.threat?.tactic?.name.contains(y)) { packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml:239: for (mitre_tactic in ctx.panw_cortex?.xdr?.mitre_tactic_id_and_name) { packages/panw_cortex_xdr/data_stream/incidents/elasticsearch/ingest_pipeline/default.yml:126: for (mitre_technique in ctx.panw_cortex?.xdr?.mitre_techniques_ids_and_names) { packages/panw_cortex_xdr/data_stream/incidents/elasticsearch/ingest_pipeline/default.yml:154: for (mitre_tactic in ctx.panw_cortex?.xdr?.mitre_tactics_ids_and_names) { packages/pfsense/data_stream/log/elasticsearch/ingest_pipeline/php-fpm.yml:21: if: 'ctx._tmp?.action.toLowerCase().contains("success")' packages/pfsense/data_stream/log/elasticsearch/ingest_pipeline/php-fpm.yml:25: if: 'ctx._tmp?.action.toLowerCase().contains("authentication error")' packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:24: if: ctx.json?.action?.type?.toLowerCase().contains('created') || ctx.json.action.type.toLowerCase().contains('deleted') || ctx.json.action.type.toLowerCase().contains('updated') || ctx.json.action.type.toLowerCase().contains('access_allowed') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:28: if: ctx.json?.action?.type?.toLowerCase().contains('created') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:32: if: ctx.json?.action?.type?.toLowerCase().contains('deleted') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:36: if: ctx.json?.action?.type?.toLowerCase().contains('updated') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:40: if: ctx.json?.action?.type?.toLowerCase().contains('user') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:44: if: ctx.json?.action?.type?.toLowerCase().contains('group') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:48: if: ctx.json?.action?.type?.toLowerCase().contains('allowed') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:52: if: ctx.json?.action?.type?.toLowerCase().contains('denied') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:56: if: ctx.json?.action?.type?.toLowerCase().contains('started') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:60: if: ctx.json?.action?.type?.toLowerCase().contains('access_allowed') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:64: if: ctx.json?.action?.type?.toLowerCase().contains('password.check_succeeded') packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:68: if: ctx.json?.action?.type?.toLowerCase().contains('email') packages/proofpoint_tap/data_stream/message_blocked/elasticsearch/ingest_pipeline/default.yml:237: for (artifact in ctx.json?.threatsInfoMap) { packages/proofpoint_tap/data_stream/message_blocked/elasticsearch/ingest_pipeline/default.yml:237: for (artifact in ctx.json?.threatsInfoMap) { packages/proofpoint_tap/data_stream/message_delivered/elasticsearch/ingest_pipeline/default.yml:233: for (artifact in ctx.json?.threatsInfoMap) { packages/proofpoint_tap/data_stream/message_delivered/elasticsearch/ingest_pipeline/default.yml:233: for (artifact in ctx.json?.threatsInfoMap) { packages/salesforce/data_stream/login_stream/elasticsearch/ingest_pipeline/default.yml:218: if: '!ctx.source?.geo?.location?.containsKey("lat") && !ctx.source?.geo?.location?.containsKey("lon")' packages/suricata/data_stream/eve/elasticsearch/ingest_pipeline/dns-answer-v2.yml:16: for (def answer : ctx?.dns?.answers) { packages/tenable_sc/data_stream/asset/elasticsearch/ingest_pipeline/default.yml:190: if (nameArray?.length > 0) { packages/tenable_sc/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml:462: if (nameArray?.length > 0) { packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml:72: if: "['md5', 'sha1', 'sha256', 'sha512', 'ssdeep'].contains(ctx.cif3?.itype) && !ctx.cif3?.tags.contains('ja3')" packages/trendmicro/data_stream/deep_security/elasticsearch/ingest_pipeline/default.yml:40: if: "ctx.trendmicro?.event?.signature_id >= 6000000 && ctx.trendmicro?.event?.signature_id <= 6999999" packages/trendmicro/data_stream/deep_security/elasticsearch/ingest_pipeline/default.yml:52: if: "ctx.trendmicro?.event?.signature_id >= 4000000 && ctx.trendmicro?.event?.signature_id <= 4999999" packages/trendmicro/data_stream/deep_security/elasticsearch/ingest_pipeline/default.yml:55: if: "ctx.trendmicro?.event?.signature_id >= 100 && ctx.trendmicro?.event?.signature_id <= 7499" packages/trendmicro/data_stream/deep_security/elasticsearch/ingest_pipeline/default.yml:61: if: "ctx.trendmicro?.event?.signature_id >= 5000000 && ctx.trendmicro?.event?.signature_id <= 5999999" packages/vectra_detect/data_stream/log/elasticsearch/ingest_pipeline/pipeline-account-detection.yml:107: if: ctx.vectra_detect?.log?.account?.uid.contains('@') packages/zeek/data_stream/kerberos/elasticsearch/ingest_pipeline/default.yml:185: if: ctx.zeek?.kerberos?.client.contains('/') packages/zeek/data_stream/known_certs/elasticsearch/ingest_pipeline/default.yml:46: if: ctx.host?.ip.contains('.') packages/zeek/data_stream/known_certs/elasticsearch/ingest_pipeline/default.yml:50: if: ctx.host?.ip.contains(':') packages/zeek/data_stream/known_hosts/elasticsearch/ingest_pipeline/default.yml:46: if: ctx.host?.ip.contains('.') packages/zeek/data_stream/known_hosts/elasticsearch/ingest_pipeline/default.yml:50: if: ctx.host?.ip.contains(':') packages/zeek/data_stream/known_services/elasticsearch/ingest_pipeline/default.yml:45: if: ctx.host?.ip.contains('.') packages/zeek/data_stream/known_services/elasticsearch/ingest_pipeline/default.yml:49: if: ctx.host?.ip.contains(':') packages/zeek/data_stream/ntp/elasticsearch/ingest_pipeline/default.yml:95: if: ctx.source?.ip.contains('.') packages/zeek/data_stream/ntp/elasticsearch/ingest_pipeline/default.yml:99: if: ctx.source?.ip.contains(':') packages/zeek/data_stream/signature/elasticsearch/ingest_pipeline/default.yml:83: if: ctx.source?.ip.contains('.') packages/zeek/data_stream/signature/elasticsearch/ingest_pipeline/default.yml:87: if: ctx.source?.ip.contains(':') packages/zeek/data_stream/software/elasticsearch/ingest_pipeline/default.yml:50: if: ctx.host?.ip.contains('.') packages/zeek/data_stream/software/elasticsearch/ingest_pipeline/default.yml:54: if: ctx.host?.ip.contains(':') packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:100: if: "ctx?.event?.action.startsWith('account')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:103: if: "ctx?.event?.action.startsWith('chat_message')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:106: if: "ctx?.event?.action.startsWith('chat_channel')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:109: if: "ctx?.event?.action.startsWith('phone')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:112: if: "ctx?.event?.action.startsWith('recording')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:115: if: "ctx?.event?.action.startsWith('user')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:118: if: "ctx?.event?.action.startsWith('webinar')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:121: if: "ctx?.event?.action.startsWith('zoomroom')" packages/zoom/data_stream/webhook/elasticsearch/ingest_pipeline/default.yml:97: if: "ctx?.event?.action.startsWith('meeting')" ```

Variations of the problem

In these identified cases there were a number of different types of null handling errors:

Constructed examples ``` POST _scripts/painless/_execute { "script": { "params": { "weather": { "temperature": null }, "plans": { "beAlone": null, "swim": null, "sit": true }, "books": { "finished": null } }, "source": """ def beachBag = []; if (params.plans?.beAlone) { // if condition beachBag.add("thoughts"); } else { beachBag.add("a friend"); } if (params.plans?.swim || params.plans?.sit) { // operand of boolean operator beachBag.add("a towel"); } if (params.weather?.temperature <= 20) { // operand of comparison operator beachBag.add("a coat"); } else { beachBag.add("a hat"); } def books = ["The Little Prince"]; beachBag.addAll(books); for (finishedBook in params.books?.finished) { // iteration statement of for loop beachBag.removeIf(b -> b == finishedBook); } if (params.weather?.sun.bright == true) { // incomplete null safe beachBag.add("sunglasses"); } if (params.plans?.swim.equals(true)) { // equals() beachBag.add("a swimsuit"); } return "I'm going to the beach and I'm bringing " + beachBag; """ } } ```

Checklist of fixes

It's probably best to fix these with a separate pull request for each integration.

sec-deployment-and-devices

sec-windows-platform

security-service-integrations

stack-monitoring

obs-infraobs-intergrations

obs-ds-hosted-services

ecosystem

elasticmachine commented 11 months ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

taylor-swanson commented 9 months ago

Related to this, we need to also watch out for the following:

if: ctx.zeek?.kerberos?.client.contains('/') == true

Note that client is missing a null conditional operator and if it is null, then the processor will fail due to null dereference. This just came up in zeek in a few places.