Issue Template

Please use this template!

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'onioningestor' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'onioningestor'. Thanks.

Make sure you've checked the following:

[] Python version is 3.10.6

OnionIngestor Output

Please provide the output of OnionIngestor

OnionIngestor Yaml file

This is an example ThreatIngestor config file with some preconfigured RSS

sources, feeding extracted artifacts into a CSV file.

general:

Run forever, check feeds once an hour.

daemon: True
sleep: 10
onion_validation: ([a-z2-7]{16,56}\.onion)
blacklist: blacklist,keywords,go,here
interestingKeywords: Interesting,Keywords,Go,Here
save-thread: no         # Use a separate thread to save onions
TorController:
    port: 9051
    password: mine password

monitor: filename: monitoring.txt

sources:

# A few threat intel blogs to get you started!
- name: simple-text-file
  module: simplefile
  filename: onion_master_list.txt

- name: hunchly
  module: hunchly
  domain: https://www.dropbox.com/sh/wdleu9o7jj1kk7v/AADq2sapbxm7rVtoLOnFJ7HHa/HiddenServices.xlsx

- name: pystemon
  module: pystemon
  dirname: pystemon/alerts/

- name: dark.fail
  module: dark.fail
  domain: https://dark.fail/

#  - name: source-gist
#    module: gist
#    url: https://gist.github.com/search?l=Text&q=.onion

#  - name: source-reddit
#    module: reddit
#    url: https://api.pushshift.io/reddit/search/comment/?subreddit=onions&limit=1000000
#    feed_type: messy
#
#  - name: pastebin
#    module: pastebin-account
#    url: https://gist.github.com/search?l=Text&q=.onion
#    feed_type: messy
#
#  - name: hunchly-report
#    module: gmail-hunchly
#    url: https://gist.github.com/search?l=Text&q=.onion
#    feed_type: messy
#
#  - name: onionland-search
#    module: collect-onions
#    url: http://3bbaaaccczcbdddz.onion/discover
#    feed_type: messy
#
#  - name: torch
#    module: collect-onions
#    url: http://xmh57jrzrnw6insl.onion
#    feed_type: messy

operators:

name: simple-html module: html timeout: 300 retries: 2 interestingKeywords: YOUR,INTERESTING,KEYWORDS,GO,HERE socks5: http: 'socks5h://127.0.0.1:9050' https: 'socks5h://127.0.0.1:9050'

- name: onionscan-go

module: onionscan

binpath: /PATH/TO/YOUR/ONIONSCAN/GO/BINARY

- name: simple-screenshot

module: screenshot

screenshots_path: null

- name: yara-rule

module: yara

filename: categories.yar

base_score: 50

database_Engines:

# Simple telegram notifier

name: telegram-notifer # see https://core.telegram.org/bots/api#authorizing-your-bot module: telegram chat_id: YOUR-TELEGRAM-CHAT token: YOUR-TELEGRAM-TOKEN

- name: elasticsearch

module: elasticsearch

index: YOUR-ELASTICSEARCH-INDEX_NAME

port : 9200

host : 127.0.0.1

- name: email

module: send_email

alert: no # Enable/disable email alerts

from: alert@example.com

to: alert@example.com

server: 127.0.0.1 # Address of the server (hostname or IP)

port: 25 # Outgoing SMTP port: 25, 587, ...

tls: no # Enable/disable tls support

username: '' # (optional) Username for authentication. Leave blank for no authentication.

password: '' # (optional) Password for authentication. Leave blank for no authentication.

subject: '[onioningestor] - {subject}'

size-limit: 1048576 # Size limit for pastie, above it's sent as attachement

Description of Issue

sudo python3 -m onioningestor -c config.yml [30 Mar 2023 21:48:00] - INFO - Starting OnionScraper Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/Desktop/tools/OnionIngestor/onioningestor/main.py", line 47, in app = Ingestor(args) File "/home/Desktop/tools/OnionIngestor/onioningestor/init.py", line 27, in init self.config = config.Config(args.configFile, self.logger) File "/home/Desktop/tools/OnionIngestor/onioningestor/config.py", line 26, in init with io.open(self.filename, "r") as f: FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'

Environment Details

Linux Ubuntu 22.04 Additional context Add any other context about the problem here.

danieleperera / OnionIngestor

[BUG] #16