Shadowserver API bot

[kodebrain]

Hi,

By default, the bot seems to download a set of reports covering a four-day span (e.g., from "2024-02-26" to "2024-02-29"). However, I have specific requirements regarding the types of reports and the dates for which I need the data.

Could you please guide me on how to customise the bot's configuration to download only specific types of reports for a specific date or based on the current date? For example, If I am interested in fetching only the 'sinkhole', 'http_scan', 'compromised-website','honeypot_ddos' reports for the current date.

Thank you

[kamil-certat]

@elsif2 Could you please have a look at this? I have also got a similar report during private channels (I didn't have a time to dig into yet)

[elsif2]

The bot downloads the latest reports by checking the past two days for availability and downloads only those that have not been fetched previously.

You can specify the types of reports to download using the types option.

[kodebrain]

Thanks @elsif2 I've been attempting to configure the bot to selectively download subsets of scan reports, instead of pulling all the reports for type scan that can contain 1M plus events. Can it be filtered to download a specific set of reports for example scan_ssl, scan6_ssl, scan_rdpeudp, scan_http, scan6_http etc... The specific report types I've defined in the configuration don't seem to be filtering as expected. Could you offer any advice on how to effectively target these report types in the bot's settings?

[elsif2]

What parameters do you have configured for the collector?

Example:

    parameters:
        types: [scan_ssl, scan6_ssl, scan_rdpeudp, scan_http]

[kodebrain]

Yes, that's how I defined them but when i did that way, it did not download any data so I've to apply only scan in the types field which isn't ideal as it downloads an overwhelming volume of reports.

[elsif2]

The following config only downloads the blocklist and scan_rdpeudp types on my system:

 parameters:
   types: [blocklist,scan_rdpeudp]

What version of IntelMQ are you running?

[kodebrain]

intelmqctl --version
3.3.0

ShadowServerAPI-Collector:
  bot_id: ShadowServerAPI-Collector
  enabled: true
  group: Collector
  module: intelmq.bots.collectors.shadowserver.collector_reports_api
  name: ShadowServerAPI
  parameters:
    api_key: "$API_KEY_received_from_the_shadowserver_foundation"
    bottype: Collector
    destination_queues:
      _default: [Shadowserver-Parser-queue]
    http_header: {}
    provider: Shadowserver
    rate_limit: 86400
    reports: null
    secret: $SECRET_received_from_the_shadowserver_foundation"
    types: blocklist
  run_mode: continuous

Even when the types was set to blocklist, it still downloaded all scan reports, I will do some testing over the next few days and see if any changes The report field, not sure what values should go in there or it can be left as null

[elsif2]

The types parameter must be a list:

parameters:
      types: [blocklist]