[ti_threatq] Fix confidence scoring, scale ingestion, update ECS mappings & dashboard improvements

cpascale43 commented 3 months ago

ThreatQuotient has surfaced a few issues with our integration. This may be turned into a meta issue, but the main areas of concern are:

Updating confidence scoring logic
Scaling up IOC ingestion batch size
Resolving ECS field mapping issues
Improving dashboard data source handling

Sample logs are available here.

Update confidence scoring logic

The currently implementation doesn't match ThreatQ's 0-10 scale. Attempts to modify pipeline.yml haven't resolved the issue.

https://github.com/elastic/integrations/blob/0c7c64ebaf9c7985aa7acde83b1b20433a2813e6/packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml#L92-L114

ThreatQuotient's scoring system is based on a 0-10 scale with no decimal points. All IOCs have a score, so this should look close to the script below.

 - script:
      lang: painless
      if: ctx.json?.score != null
      description: >
        Normalize confidence level.
      source: >
        def value = ctx.json.score;
        }
        if (value >= 0 && value <= 4) {
          ctx.threat.indicator.confidence = "Low";
          return;
        }
        if (value >= 5 && value <=7) {
          ctx.threat.indicator.confidence = "Med";
          return;
        }
        if (value >= 8 && value <= 10) {
          ctx.threat.indicator.confidence = "High";
          return;
        }

Increase IOC ingestion batch size

The current ingest limit is 100 IOCs at a time, which causes slow ingestion of large numbers of IOCs. For example, if a customer wants to transfer 130,000+ IOCs, this would take about 21 hours. https://github.com/elastic/integrations/blob/e6362aa697abb3c2b2ff0f6e200a4b7bb5fcb638/packages/ti_threatq/data_stream/threat/agent/stream/httpjson.yml.hbs#L26-L36

Update ECS field mappings

See helpers_js.json. Several fields need to be remapped to align with ECS. Specifically, file type, URL attributes and domain fields need to be corrected:

Change threat.indicator.file.type to threat.indicator.file.name for correct file type identification
Ensure proper assignment of attributes such as Port (threat.indicator.url.port) and Scheme (threat.indicator.url.scheme)
Address the incorrect use of threat.indicator.domain for FQDNs, changing it to threat.indicator.url.domain

Dashboard data source issues

The Total Indicators per Provider visualization in the ThreatQ Overview Dashboard doesn't seem to be displaying any data: Screenshot 2024-07-10 at 7 18 11 AM

Seephishtank_source.json. Missing TLP markings on some indicators might be causing source attribution problems. Some IOCs have parsing errors, but still show up as threatq.indicator_value.

elasticmachine commented 3 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 commented 2 weeks ago

The events that are provided in the issue are ES documents rather than the original data from ThreatQuotient. The documents do not have event.original and some contain an error.message making them an unreliable source for testing. Can we get some documents that are as provided by the endpoint or at the very least which contain an event.original to reconstruct the original document from?

elastic / integrations