[ti_threatq] Fix confidence scoring, scale ingestion, update ECS mappings & dashboard improvements

ThreatQuotient has surfaced a few issues with our integration. This may be turned into a meta issue, but the main areas of concern are:

Updating confidence scoring logic
Scaling up IOC ingestion batch size
Resolving ECS field mapping issues
Improving dashboard data source handling

Sample logs are available here.

Update confidence scoring logic

The currently implementation doesn't match ThreatQ's 0-10 scale. Attempts to modify pipeline.yml haven't resolved the issue.

https://github.com/elastic/integrations/blob/0c7c64ebaf9c7985aa7acde83b1b20433a2813e6/packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml#L92-L114

ThreatQuotient's scoring system is based on a 0-10 scale with no decimal points. All IOCs have a score, so this should look close to the script below.

 - script:
      lang: painless
      if: ctx.json?.score != null
      description: >
        Normalize confidence level.
      source: >
        def value = ctx.json.score;
        }
        if (value >= 0 && value <= 4) {
          ctx.threat.indicator.confidence = "Low";
          return;
        }
        if (value >= 5 && value <=7) {
          ctx.threat.indicator.confidence = "Med";
          return;
        }
        if (value >= 8 && value <= 10) {
          ctx.threat.indicator.confidence = "High";
          return;
        }

Increase IOC ingestion batch size

The current ingest limit is 100 IOCs at a time, which causes slow ingestion of large numbers of IOCs. For example, if a customer wants to transfer 130,000+ IOCs, this would take about 21 hours. https://github.com/elastic/integrations/blob/e6362aa697abb3c2b2ff0f6e200a4b7bb5fcb638/packages/ti_threatq/data_stream/threat/agent/stream/httpjson.yml.hbs#L26-L36

Update ECS field mappings

See helpers_js.json. Several fields need to be remapped to align with ECS. Specifically, file type, URL attributes and domain fields need to be corrected:

Change threat.indicator.file.type to threat.indicator.file.name for correct file type identification
Ensure proper assignment of attributes such as Port (threat.indicator.url.port) and Scheme (threat.indicator.url.scheme)
Address the incorrect use of threat.indicator.domain for FQDNs, changing it to threat.indicator.url.domain

Dashboard data source issues

The Total Indicators per Provider visualization in the ThreatQ Overview Dashboard doesn't seem to be displaying any data: Screenshot 2024-07-10 at 7 18 11 AM

Seephishtank_source.json. Missing TLP markings on some indicators might be causing source attribution problems. Some IOCs have parsing errors, but still show up as threatq.indicator_value.

elastic / integrations