center-for-threat-informed-defense / sightings_ecosystem

Sightings Ecosystem gives cyber defenders visibility into what adversaries actually do in the wild. With your help, we are tracking MITRE ATT&CK® techniques observed to give defenders real data on technique prevalence.
https://ctid.io/sightings-ecosystem
Apache License 2.0
33 stars 8 forks source link

Sightings data does not make it into database #8

Closed zmallen closed 6 months ago

zmallen commented 2 years ago

Hi there!

Trying to add some sightings data to the database. I am sending in two files, with 1 direct sighting and some techniques each. Feel free to use the attached files here and rename to .json

data.json.txt data3.json.txt

When running the pipeline I get the following logs:

[+] postgres:5432 is available after 0 seconds

Counting Sightings...

2 Sightings counted

Inserting Sightings into database...

Process complete in 0.0 hours

When running analysis, I get the following error from logs

Matplotlib created a temporary config/cache directory at /tmp/matplotlib-m2osgmch because the default path (/home/sightings_analysis/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.

Traceback (most recent call last):

  File "/analysis/sightings_dash.py", line 31, in <module>

    G, stylesheet = sequences.get_graph(sighting_db_conn_str=sighting_db_conn_str)

  File "/analysis/support_files/rule_mining_sequences.py", line 35, in get_graph

    rules_sets.find_assoc_rules()

  File "/analysis/helper_functions/rules_object.py", line 118, in find_assoc_rules

    raise ValueError(

ValueError: No items in groups. Ensure you have processed data via the pipeline prior to running analysis.

Logging into postgres shows no data in the sighting or technique table. Any idea what Im doing wrong?

Thanks!