Stats: Allow filtering for aggregate_identifier "tag"

[bernhardreiter]

One way to model groups is to add something to the aggregate_identifier of a directive by a certbund-contact notification_rule which then will get written into the eventdb.

intelmq-cb-mailgen operators will want to search for a special key/value in the aggregate_identifier of events to see how many events where send for the group.

[bernhardreiter]

Implementation details

creation

Here an example line of a rule

base_directive.aggregate_key["recipient_group"]  = "CNI_water"

Could be inserted in certbund-contact-git/example-rules/50shadowserver.py like

--- 50shadowserver.py.out       2019-10-16 17:22:14.225905644 +0200
+++ 50shadowserver.py   2019-10-16 17:23:55.281491630 +0200
@@ -16,6 +16,10 @@
             directive = Directive.from_contact(contact)
             directive.update(shadowserver_params)
             directive.aggregate_by_field(context.section + ".asn")
+
+            if directive.recipient_address == 'abuse.de@example.com':
+                directive.aggregate_key["recipient_group"]  = "CNI_water"
+
             context.add_directive(directive)
         return True

see in place diff at https://github.com/Intevation/intelmq-certbund-contact/commit/e690a7dc0669254c494e7c9f209cb38290d739af

example select

counting

SELECT count(*), json_object(aggregate_identifier) ->> 'recipient_group'  as rg                           
    FROM directives
    WHERE inserted_at >= '2019-10-16' 
        AND  json_object(aggregate_identifier) ->> 'recipient_group' = 'CNI_water'
   GROUP BY rg;

db

It makes sense for speeding up to add indexes to the eventdb table directives for

• inserted_at
• json_object(aggregate_identifier) ->> 'recipient_group'

[bernhardreiter]

The improvement done for https://github.com/Intevation/intelmq-fody-backend/issues/22 solves this by enhancing the old endpoints, so this is solved with fody-backend v>=0.6.4.