elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
21 stars 436 forks source link

[ti_threatq] Fix confidence scoring, scale ingestion, update ECS mappings & dashboard improvements #10783

Open cpascale43 opened 2 months ago

cpascale43 commented 2 months ago

ThreatQuotient has surfaced a few issues with our integration. This may be turned into a meta issue, but the main areas of concern are:

Sample logs are available here.

Update confidence scoring logic

The currently implementation doesn't match ThreatQ's 0-10 scale. Attempts to modify pipeline.yml haven't resolved the issue.

https://github.com/elastic/integrations/blob/0c7c64ebaf9c7985aa7acde83b1b20433a2813e6/packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml#L92-L114

ThreatQuotient's scoring system is based on a 0-10 scale with no decimal points. All IOCs have a score, so this should look close to the script below.

 - script:
      lang: painless
      if: ctx.json?.score != null
      description: >
        Normalize confidence level.
      source: >
        def value = ctx.json.score;
        }
        if (value >= 0 && value <= 4) {
          ctx.threat.indicator.confidence = "Low";
          return;
        }
        if (value >= 5 && value <=7) {
          ctx.threat.indicator.confidence = "Med";
          return;
        }
        if (value >= 8 && value <= 10) {
          ctx.threat.indicator.confidence = "High";
          return;
        }

Increase IOC ingestion batch size

The current ingest limit is 100 IOCs at a time, which causes slow ingestion of large numbers of IOCs. For example, if a customer wants to transfer 130,000+ IOCs, this would take about 21 hours. https://github.com/elastic/integrations/blob/e6362aa697abb3c2b2ff0f6e200a4b7bb5fcb638/packages/ti_threatq/data_stream/threat/agent/stream/httpjson.yml.hbs#L26-L36

Update ECS field mappings

See helpers_js.json. Several fields need to be remapped to align with ECS. Specifically, file type, URL attributes and domain fields need to be corrected:

Dashboard data source issues

The Total Indicators per Provider visualization in the ThreatQ Overview Dashboard doesn't seem to be displaying any data: Screenshot 2024-07-10 at 7 18 11 AM

Seephishtank_source.json. Missing TLP markings on some indicators might be causing source attribution problems. Some IOCs have parsing errors, but still show up as threatq.indicator_value.

elasticmachine commented 2 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)