config.yml follow links for IOC's

[mathurin68]

Didn't there used to be an option in the config.yml to follow links? I.E. pastebin etc? I can't seem to find it.

Thanks!

[mathurin68]

alright so the issue isn't "defanged = false" , the issue is it doesn't seem to work.

below is my config.yml

general:
    # You always need this section!
    # Here are some sane values to include:
    daemon: true
    sleep: 900
    state_path: state.db

credentials:
    # This section is optional. Use it to define credentials to reference below
    # in the source and operator sections.
  - name: twitter-auth
    # https://dev.twitter.com/oauth/overview/application-owner-access-tokens
    api_key: token
    api_secret_key: tokey
    access_token: token-KJNNBgxQT9bLP6zBFCPeZIBdbU7MPY
    access_token_secret: token
    defanged_only: false

  - name: misp-auth
    url: https://misp.org
    key: key
    ssl: False

sources:
    # This section defines each of the input sources for ThreatIngestor.
    # Define as many as you want. ThreatIngestor maintains a "state" for each of
    # your sources, which helps pull in only new content since the last run.
  - name: Twitter_ingest
    module: twitter
    credentials: twitter-auth
    # https://dev.twitter.com/rest/reference/get/lists/statuses
    owner_screen_name: oasdfas
    slug: Ioc

  - name: twitter-open-directory
    module: twitter
    credentials: twitter-auth
    q: '"payload" #APT'

  - name: twitter-open-directory
    module: twitter
    credentials: twitter-auth
    q: '"md5" #APT'

  - name: twitter-open-directory
    module: twitter
    credentials: twitter-auth
    q: '"url" #phishing "phishing"'

  - name: twitter-open-directory
    module: twitter
    credentials: twitter-auth
    q: '"url" #APT'

  - name: twitter-open-directory
    module: twitter
    credentials: twitter-auth
    q: '"ip" #APT'

  - name: twitter-open-directory
    module: twitter
    credentials: twitter-auth
    q: '"c2" #APT'

operators:
    # This section defines outputs for the information extracted from your
    # sources. All filtering and flow control is done here, with options like
    # "allowed_sources", "artifact_types", and "filter".
  - name: csv
    # Write artifacts to a CSV file
    module: csv
    filename: output.csv

  - name: sqlite-db
    module: sqlite
    filename: artifacts.db

  - name: misp-instance
    module: misp
    credentials: misp-auth
    tags: [type:OSINT, twitter]

[cmmorrow]

Hi @mathurin68, I'm looking into defanged_only: false not working. Do you have an example of a tweet with a non-defanged URL that isn't being extracted?

[cmmorrow]

According to the logic here: URLs in the tweet should be extracted. I'll look into the possibility that defanged_only: false in the config file isn't being picked up.

[mathurin68]

Hey @cmmorrow!
Like for this one... https://twitter.com/Cryptolaemus1/status/1299363102107013120

is it supposed to follow the pastebin link?

Haha could always be something I don't have configured right too...

[mathurin68]

Hmm it seems to be missing a ton of stuff too -- https://twitter.com/_re_fox/status/1301564536575733760

[RatherBland]

+1 Would love to be able to follow links in tweets. Pastebin being a great example as often the majority of IOC's are not in the tweet body, but the attached link.

Having a set of whitelisted domains to follow could negate opening dangerous links.

[RatherBland]

Looking at a way to implement this now, but due to the static nature of sources, it's seeming a bit tricky.