add collector

[hsninbil]

Hi, how can I add a new collector? or collectors?

[milankowww]

Hello,

Since collectors are separate to the core of Taranis NG, there are two paths in general. One of them is to modify the existing collector code in src/collectors. The other is to create a completely independent application which collects your desired type of data, and make it talk to the core using the REST API.

Option 1: modify the existing collector code

Advantages: use the infrastructure of the original collectors, only need to focus on the business part of the code Disadvantages: limited to Python How to:

• visit src/collectors
• create a new file in collectors/ subdirectory, subclass BaseCollector in that file, use RSS collector as an example
• if your collector requires some configuration (like source to harvest or anything else), in variable parameters, define parameters that will be collected from the user when adding a source based on that collector
• implement a method collect which should create array of NewsItems and use BaseCollector.publish(news_items, source) to publish the data to Taranis NG

Option 2: develop a separate application

Advantages: use any language / framework you chose Disadvantages: possibly more work How to:

• create an app in your favorite language
• the app should expose (provide) a REST API, and consume (use) a REST API of Core
• the REST API calls that a collector needs to implement:
  • GET /api/v1/collectors, GET /api/v1/collectors/NAME
  • communicate a JSON with a list of types of data this collector could provide (collector schemas). Reference: src/collectors/api/collectors.py, src/shared/shared/schema/collector.py
    • the REST API calls that a collector uses to update its status and send the crawled data to code
  • GET /api/v1/collectors/COLLECTOR_ID/osint-sources?api_key=KEY&collector_type=TYPE - get a list of configured sources for this collector
    • POST /api/v1/collectors/news-items?api_key=KEY&collector_type=TYPE to publish collected data from the collector to the core
  • GET /api/v1/collectors/COLLECTOR_ID to mark the collector "alive" right now
  • references:
    • src/core/model/news_item.py - a schema for the news item (array of NewsItemData in JSON format)
    • src/collectors/remote/core_api.py - functions to access core api from a collector: CoreApi.add_news_items, CoreApi.get_osint_sources, CoreApi.update_collector_status

[hsninbil]

Dear milanKowww, and Greetings, First of all thank you for your help, for your explanation. Also, I'm trying to follow the points but I got the same result. sorry, but I'm currently presenting your TaranisNG project instead of Taranis3.

[multiflexi]

What is in assess now?

[hsninbil]

nothing.

[hsninbil]

Dear @multiflexi ,

• http://collectors/, should change with my IP?
• I add this line SKIP_DEFAULT_COLLECTOR=true ==== vim docker/.env

[multiflexi]

http://collectors/ is fine. Why would you add SKIP_DEFAULT_COLLECTOR to the configuration? What do you have in Uncategorized? Have you set refresh interval for OSINT sources? Do you see in logs that they are collected?

[hsninbil]

Thank you @multiflexi. Taranis-NG is working.