Enhance recertification workflow as IP owner-based

[tpurschke]

design decisions

• build one "best-of-breed" recertification logic based on owner = IP<-->APP mapping
• start with fixed logic: filtering for owner in both src AND destination

pre-requisites

• [x] add demo data
• [x] owner mapping processes with ip mapping (as view)
• [x] issue: views cannot be combined with other tables in graphql
• [x] add a flag for local groups "is_owner_group"
• [x] add UI settings for editing owner_network
• [x] add column "next_recert_date" to "recertification" table
• [x] import process to contain "add new recert entry into recertifaction table"
• [x] find out if hasura has other mechanism for automatic changes (other than db trigger)
• [x] add trigger to owner table --> change should result in recertication table open entry adjustments (if no trigger alternative is found)
• [x] the super owner will be a fixed owner with id 0 who always exists and is indelible. The super owner will have no owner_networks defined. It will be created during install in db fill script
• [ ] add "owner" and "owner_id" to filter line
• [ ] add tests for recert logic
  • [x] create simple config data @ fwodemodata.cactus.de: https://fwodemodata.cactus.de/test08-recert.json
  • [ ] add ansible test with this demodata to verify logic using creation of json reports
• [x] repair settings owner - default owner (set as default) and deletion of default owner should both work
• [x] only create sample owner data during fresh install not during upgrade
• [ ] also show ip matches for super owner (all IPs that do not match for another owner)
• [x] just the initial recertification of 155.000 rules takes 7.5h with just a single owner (super-owner)
• [x] remove triggers, do not recalculate during import, and replace with nightly calls to recert_ functions for each management/owner - also offer a recalc recert data button in addition for ad-hoc recalculation

missing database fields

• [x] might better add mgm_id where using rule_metadata_id for uniqueness? e.g. in recertification table (implemented by storing rule_id ref)
• [x] add recertification table to store all recert events and map to owner and rule_metadata
• [x] currently we can only store one last recertifier per rule but this should be actually a list of multiple recertifiers (need to decide on how to handle recert periods)

UI elements owner settings

• [x] #1995
• [x] allow for setting a special "super owner" who is responsible for all non-app-related IP addresses
  • [x] for the super owner no networks should be addable
• [x] remove demo owners data functionality

UI elements recertification report

• [x] recert report - add owner filter as drop-down for all reporters
• [x] add "show any-rules" option (checkbox)
• [x] grey-out time filter for recert report --> better hide altogether
• [x] add filter by owner
• [x] migrate certification page from views to recertification table entries
  • [x] super owner will become the default (replaces the current view) serving as a single recert use if no owner data is defined. Still need to find a way to implement this without demo user!
• [x] add export functionality
• [x] show all rules without owner / default-owner rules (reset of owner filter is still missing here)
• [x] show an overview of rule recert status - one line for each rule
  • green - fully recertified within the given recert time range for each app
  • red - not fully recertified within the given recert time range for each app
• [x] show detailed recert status of rule-base (use rule-details component to reveal more information on given and missing recertifications)
  • green - fully recertified within the given recert time range for each app (listing all recert events in details)
  • red - not fully recertified within the given recert time range for each app - (listing all recert events - and missing events - in details app-based)
• [x] #2003
• [x] #2154
• [ ] add option for manually setting the next_recert_date via UI (needs extra function)
• [ ] need a way to report on overdue recerts for decertified rules (these are currently neither shown in recert report nor on recert page)
• [ ] For installation with many owners, not all owners should be displayed immediately - only show the first 3 owners of each rule and allow for expanding the rule to show all owners.

enhance recertification page filtered by owner(s)

• [x] lookup logged-in user in all groups referenced in owner.id field
• [x] filter rules only for the logged-in owner
• [x] also allow for auditor and admin to have the view (read-only) with a manual filter option for arbitrary owners
• [x] also allow for auditor and admin to have a read-only view across all rules (without owner-filter)

advanced stuff / enhancements

• [x] add settings page for outgoing email server (hostname, port, tls y/n, optional user & password) - works for plain smtp and with or without authentication
• [x] allow for sending smtps mails (currently runs into timeout) --> move to MailKit library (https://github.com/dnnsoftware/Dnn.Platform/issues/3768)
• [x] add smart email notifications for owners
• [ ] provide import mechanism to bulk-import a list of owner definitions
• [ ] always create a single local ldap group "app-id_autogenerated_owner_group" for all owners, containing both the main owner and the owner group
• [ ] add manual rule/owner mapping UI
• [ ] use (fortinet et al.) zone information for additional mapping - would need to store zone information in addition to ip information - currently a zone-based any-rule matches with all ip addresses
• [ ] add exceptions to define which rules will not be recertified with standard recert workflow but need to be recertified differently (based on source/destination ip and port/proto)
• [ ] currently the first owner which matches an ip object within a rule "owns" it. So if e.g., two owners have the same ip range, only the first one is assigned as owner. This is too arbitrary depending on the order of owner - rule labeling.
• [ ] (this is not correct!) Not all parts of a subnet are assigned an owner. E.g. if the source of a rule is "0.0.0.0/0 and an owner has network 1.0.0.0/24 then this owner gets the whole 0.0.0.0/0 object and no other owners need to recertify this object/rule - not even the super-owner.
• [ ] Provide report listing all recent recertifiers and the recertification date per rule.
• [ ] do we need a flag in the internal ldap for "is owner group"?
• [ ] add filter-line to recert page to allow for filtering by e.g. ip address
• [ ] add the same export functionality which exists for the reporting section

add details for view:

• [x] allow for filtering out any in src or dst fields
• [ ] add src & dst cell negation awareness
• [ ] add port filtering
• [ ] add ip range support for views

performance optimization

• [x] add field for rule "is_allow_rule" and fill it for all importers as well as for existing rule data during upgrade
• [ ] add field for rule "is_any_rule" and fill it (if either source or dst is any/all) for all importers as well as for existing rule data during upgrade to allow for faster filtering
• [x] if view becomes to slow for big data, build basic rule: fill rule_owner table nightly --> (trying first with dynamic view - need to test this with large data sets)
• [x] when importing, only calculate super owner once
• [x] add indices for full views (object.obj_ip, owner_network.ip, owner_network.owner_id

[gammelich]

Could be interesting to compare this approach to the current JAVA based repo created. In there an Application based approach running on a subscriber pattern allows to decouple management of Firewall Rules into a relation of "Communication relation A -> D" matching via Firewall Architect or Automation based relation creation to "installed basis ( firewall rules)" of the firewalls identified in path to provide the requested connectivity.

This approach as well provides Actual vs. Target and Target vs. Actual reporting.