Sightings Ecosystem gives cyber defenders visibility into what adversaries actually do in the wild. With your help, we are tracking MITRE ATT&CK® techniques observed to give defenders real data on technique prevalence.
This can become untenable for large JSON blobs of sightings data, as python can be inefficient at processing larger files. This is especially true for high throughput applications that can log millions to tens-of-millions of logs to sightings per day.
I recommend the ability to load files as jsonl, which are newline delimited json blobs for sightings. You can then setup sidecards/forwarders (i am using https://vector.dev) to append to a file within the sightings ecosystem
When loading sightings data under
./data
, https://github.com/center-for-threat-informed-defense/sightings_ecosystem/blob/main/src/pipeline/pipeline.py#L218 assumes that a file will be one massive JSON blob.This can become untenable for large JSON blobs of sightings data, as python can be inefficient at processing larger files. This is especially true for high throughput applications that can log millions to tens-of-millions of logs to sightings per day.
I recommend the ability to load files as
jsonl
, which are newline delimited json blobs for sightings. You can then setup sidecards/forwarders (i am using https://vector.dev) to append to a file within the sightings ecosystem