center-for-threat-informed-defense / sightings_ecosystem

Sightings Ecosystem gives cyber defenders visibility into what adversaries actually do in the wild. With your help, we are tracking MITRE ATT&CK® techniques observed to give defenders real data on technique prevalence.
https://ctid.io/sightings-ecosystem
Apache License 2.0
33 stars 8 forks source link

Add additional logic for pipelining sightings. #9

Closed zmallen closed 6 months ago

zmallen commented 2 years ago

When someone wants to experiment with sightings with less than 1000 sightings, they are unable to do so due to the constraint of db_batch_ingest_size being 1000. Change this to an environment variable, as well as logic around per file commit processing rather than waiting for the 1000 number.

zmallen commented 2 years ago

relates to #8