InQuest / ThreatIngestor

Extract and aggregate threat intelligence.
https://inquest.readthedocs.io/projects/threatingestor/
GNU General Public License v2.0
821 stars 135 forks source link

Tagging not working as expected #104

Closed virtuallaik closed 3 years ago

virtuallaik commented 3 years ago

I attempted to query twitter for specific malware types so that I could could tag them in MISP. Is this the correct way to go about this? When the config was much smaller it was correctly tagging. As it got bigger it now tags them all the same. All of the sources seem to come into my misp instance with the same emotet tags. Even though ive used allowed sources. Can someone else sanity check this config for me.

Config.yml

general:

    daemon: true
    sleep: 900
    state_path: state.db

credentials:

  - name: twitter-auth
    api_key: <redacted>
    api_secret_key: <redacted>
    access_token: <redacted>
    access_token_secret: <redacted>

  - name: misp-auth
    url: <redacted>
    key: <redacted>
    ssl: False

sources:
  - name: Twitter-Emotet
    module: twitter
    credentials: twitter-auth
    q: "#emotet 'hxxp' OR 'md5' OR 'sha256'"

  - name: Twitter-Trickbot
    module: twitter
    credentials: twitter-auth
    q: "#trickbot"

  - name: Twitter-Ryuk
    module: twitter
    credentials: twitter-auth
    q: "#ryuk OR #ryukransomware"

  - name: Twitter-Njrat
    module: twitter
    credentials: twitter-auth
    q: "#njrat AND 'hxxp' OR 'md5' OR 'sha256' OR 'C2:'"

  - name: Twitter-AgentTesla
    module: twitter
    credentials: twitter-auth
    q: "#agenttesla"

  - name: Twitter-Qbot
    module: twitter
    credentials: twitter-auth
    q: "#qbot"

  - name: Twitter-Nanocore
    module: twitter
    credentials: twitter-auth
    q: "#nanocore"    

  - name: Twitter-Remcos
    module: twitter
    credentials: twitter-auth
    q: "#remcos"

  - name: Twitter-Dridex
    module: twitter
    credentials: twitter-auth
    q: "#dridex"

  - name: Twitter-Lokibot
    module: twitter
    credentials: twitter-auth
    q: "#lokibot"

  - name: Twitter-Formbook
    module: twitter
    credentials: twitter-auth
    q: "#formbook"

  - name: Twitter-Crimsonrat
    module: twitter
    credentials: twitter-auth
    q: "#crimsonrat"

  - name: Twitter-Ursnif
    module: twitter
    credentials: twitter-auth
    q: "#ursnif"

  - name: Twitter-Wannacry
    module: twitter
    credentials: twitter-auth
    q: "#wannacry"

  - name: Twitter-Quasarrat
    module: twitter
    credentials: twitter-auth
    q: "#quasarrat"

  - name: Twitter-Azorult
    module: twitter
    credentials: twitter-auth
    q: "#azorult"

  - name: Twitter-Avemaria
    module: twitter
    credentials: twitter-auth
    q: "#avemaria AND @RedBeardIOCs"

  - name: Twitter-Hancitor
    module: twitter
    credentials: twitter-auth
    q: "#hancitor"

  - name: Twitter-Ragnarlocker
    module: twitter
    credentials: twitter-auth
    q: "#ragnarlocker"

  - name: Twitter-Netwire
    module: twitter
    credentials: twitter-auth
    q: "#netwire"

  - name: Twitter-Maze
    module: twitter
    credentials: twitter-auth
    q: "#mazeransomware"

  - name: Twitter-Wshrat
    module: twitter
    credentials: twitter-auth
    q: "#wshrat"

  - name: Twitter-AdWind
    module: twitter
    credentials: twitter-auth
    q: "#Adwind"

  - name: Twitter-Asyncrat
    module: twitter
    credentials: twitter-auth
    q: "#Asyncrat"

  - name: Twitter-Zloader
    module: twitter
    credentials: twitter-auth
    q: "#zloader"

operators:
    # This section defines outputs for the information extracted from your
    # sources. All filtering and flow control is done here, with options like
    # "allowed_sources", "artifact_types", and "filter".

  - name: misp-Emotet
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Emotet
    tags: [tlp:white, mwdb:family="emotet", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Trickbot
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Trickbot
    tags: [tlp:white, mwdb:family="trickbot", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Ryuk
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Ryuk
    tags: [tlp:white, mwdb:family="ryuk", malware_classification:malware-category="Ransomware", osint:source-type="microblog-post"]

  - name: misp-Njrat
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Njrat
    tags: [tlp:white, mwdb:family="njrat", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-AgentTesla
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-AgentTelsa
    tags: [tlp:white, mwdb:family="agenttesla", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Qbot
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Qbot
    tags: [tlp:white, mwdb:family="qbot", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Nanocore
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Nanocore
    tags: [tlp:white, mwdb:family="nanocore", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Remcos
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Remcos
    tags: [tlp:white, mwdb:family="remcos", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]  

  - name: misp-Dridex
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Dridex
    tags: [tlp:white, mwdb:family="dridex", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Lokibot
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Lokibot
    tags: [tlp:white, mwdb:family="lokibot", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Formbook
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Formbook
    tags: [tlp:white, mwdb:family="formbook", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"] 

  - name: misp-Crimsonrat
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Crimsonrat
    tags: [tlp:white, mwdb:family="crimsonrat", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Ursnif
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Ursnif
    tags: [tlp:white, mwdb:family="ursnif", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"] 

  - name: misp-Wannacry
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Wannacry
    tags: [tlp:white, mwdb:family="wannacry", malware_classification:malware-category="Ransomware", osint:source-type="microblog-post"]

  - name: misp-Quasarrat
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Quasarrat
    tags: [tlp:white, mwdb:family="quasarrat", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Azorult
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Azorult
    tags: [tlp:white, mwdb:family="azorult", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Avemaria
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Avemaria
    tags: [tlp:white, mwdb:family="avemaria", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Hancitor
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Hancitor
    tags: [tlp:white, mwdb:family="hancitor", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Ragnarlocker
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Ragnarlocker
    tags: [tlp:white, mwdb:family="ragnarlocker", malware_classification:malware-category="Ransomware", osint:source-type="microblog-post"]

  - name: misp-Netwire
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Netwire
    tags: [tlp:white, mwdb:family="netwire", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Maze
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Maze
    tags: [tlp:white, mwdb:family="maze", malware_classification:malware-category="Ransomware", osint:source-type="microblog-post"]

  - name: misp-Wshrat
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Wshrat
    tags: [tlp:white, mwdb:family="wshrat", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Adwind
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Adwind
    tags: [tlp:white, mwdb:family="adwind", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Asyncrat
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Asyncrat
    tags: [tlp:white, mwdb:family="asyncrat", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]

  - name: misp-Zloader
    module: misp
    credentials: misp-auth
    allowed_sources: Twitter-Zloader
    tags: [tlp:white, mwdb:family="zloader", malware_classification:malware-category="Trojan", osint:source-type="microblog-post"]
virtuallaik commented 3 years ago

Remember to wrap allowed sources in [ ]. GG

cmmorrow commented 3 years ago

Hey @virtuallaik. Sorry I just saw this issue and glad you figured it out. Reach out if you have any other questions.