search

CactuseSecurity / firewall-orchestrator

Network Security data repository for automation, reporting and compliance of firewall rules
https://fwo.cactus.de
Apache License 2.0
33 stars 11 forks source link

Importer rework - merge FWO.Basics #2618

Closed tpurschke closed 3 weeks ago

tpurschke commented 1 month ago

also fix AppRules report intermediate broken state

alf-cactus commented 4 weeks ago

My first Problem was:

TASK [docker : apt update] ***** fatal: [localhost]: FAILED! => changed=false msg: 'Failed to update apt cache: W:Updating from such a repository can''t be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., E:The repository ''https://download.docker.com/linux/debian jammy Release'' does not have a Release file.'

I solved it by changing file nano roles/docker/tasks/main.yml replace line 27:

line: "deb [arch={{ linux_architecture }}] https://download.docker.com/linux/debian {{ os_codename }} stable"

with:

line: "deb [arch={{ linux_architecture }}] https://download.docker.com/linux/{{ ansible_facts['distribution']|lower }} {{ os_codename }} stable"

alf-cactus commented 4 weeks ago

For my second problem I have no solution yet:

TASK [api : import API metadata via metadata API directly from local file] *****
fatal: [localhost]: FAILED! => changed=false 
    content: '{"error":"cannot continue due to inconsistent metadata","path":"$.args","code":"unexpected","internal":[{"definition":{"comment":null,"name":"rule","source":"default","table":{"name":"rulebase_on_gateway","schema":"public"},"using":{"foreign_key_constraint_on":"layer_guard_rule"}},"name":"object_relation rule in table rulebase_on_gateway in source default","reason":"Inconsistent object: in table \"rulebase_on_gateway\": in relationship \"rule\": no foreign constraint exists on the given column(s)","type":"object_relation"},{"definition":{"comment":null,"definition":{"function":{"name":"get_rules_for_owner","schema":"public"}},"name":"get_rules_for_owner","source":"default","table":{"name":"device","schema":"public"}},"name":"computed_field get_rules_for_owner in table device in source default","reason":"Inconsistent object: in table \"device\": in computed field \"get_rules_for_owner\": no such function exists: \"get_rules_for_owner\"","type":"computed_field"},{"definition":{"comment":null,"name":"rulebase_on_gateways","source":"default","table":{"name":"rule","schema":"public"},"using":{"foreign_key_constraint_on":{"column":"layer_guard_rule","table":{"name":"rulebase_on_gateway","schema":"public"}}}},"name":"array_relation
        rulebase_on_gateways in table rule in source default","reason":"Inconsistent object: in table \"rule\": in relationship \"rulebase_on_gateways\": no foreign constraint exists on the given column(s)","type":"array_relation"},{"definition":{"comment":"","permission":{"backend_only":false,"check":{},"columns":["layer_guard_rule","dev_id","order_no","rulebase_id"]},"role":"importer","source":"default","table":{"name":"rulebase_on_gateway","schema":"public"}},"name":"insert_permission importer in table rulebase_on_gateway in source default","reason":"Inconsistent object: in table \"rulebase_on_gateway\": in permission for role \"importer\": column \"layer_guard_rule\" does not exist","type":"insert_permission"},{"definition":{"comment":"","permission":{"allow_aggregations":true,"columns":["layer_guard_rule","dev_id","order_no","rulebase_id"],"computed_fields":[],"filter":{}},"role":"importer","source":"default","table":{"name":"rulebase_on_gateway","schema":"public"}},"name":"select_permission
        importer in table rulebase_on_gateway in source default","reason":"Inconsistent object: in table \"rulebase_on_gateway\": in permission for role \"importer\": column \"layer_guard_rule\" does not exist","type":"select_permission"},{"definition":{"comment":"","permission":{"backend_only":false,"check":{},"columns":["access_rule","action_id","active","dev_id","enforced_on_gateway","is_global","last_change_admin","mgm_id","nat_rule","parent_rule_id","parent_rule_type","removed","rule_action","rule_comment","rule_create","rule_custom_fields","rule_disabled","rule_dst","rule_dst_neg","rule_dst_refs","rule_from_zone","rule_head_text","rule_id","rule_implied","rule_installon","rule_last_seen","rule_name","rule_num","rule_num_numeric","rule_ruleid","rule_src","rule_src_neg","rule_src_refs","rule_svc","rule_svc_neg","rule_svc_refs","rule_time","rule_to_zone","rule_track","rule_uid","rulebase_id","track_id","xlate_rule"]},"role":"importer","source":"default","table":{"name":"rule","schema":"public"}},"name":"insert_permission
        importer in table rule in source default","reason":"Inconsistent object: in table \"rule\": in permission for role \"importer\": column \"enforced_on_gateway\" does not exist","type":"insert_permission"},{"definition":{"comment":"","permission":{"allow_aggregations":false,"columns":["parent_rule_id","removed","rule_create","rule_id","rule_last_seen","xlate_rule","access_rule","active","enforced_on_gateway","is_global","nat_rule","rule_disabled","rule_dst_neg","rule_implied","rule_src_neg","rule_svc_neg","rule_installon","rule_name","rule_ruleid","rule_time","action_id","dev_id","last_change_admin","mgm_id","rulebase_id","rule_from_zone","rule_num","rule_to_zone","track_id","rule_custom_fields","rule_num_numeric","parent_rule_type","rule_action","rule_comment","rule_dst","rule_dst_refs","rule_head_text","rule_src","rule_src_refs","rule_svc","rule_svc_refs","rule_track","rule_uid"],"computed_fields":["rule_relevant_for_tenant"],"filter":{}},"role":"importer","source":"default","table":{"name":"rule","schema":"public"}},"name":"select_permission
        importer in table rule in source default","reason":"Inconsistent object: in table \"rule\": in permission for role \"importer\": column \"enforced_on_gateway\" does not exist","type":"select_permission"},{"definition":{"comment":"","permission":{"backend_only":false,"check":{},"columns":["created","removed","is_global","name","id","mgm_id"]},"role":"importer","source":"default","table":{"name":"rulebase","schema":"public"}},"name":"insert_permission importer in table rulebase in source default","reason":"Inconsistent object: in table \"rulebase\": in permission for role \"importer\": column \"removed\" does not exist","type":"insert_permission"},{"definition":{"comment":"","permission":{"allow_aggregations":true,"columns":["created","removed","is_global","name","id","mgm_id"],"computed_fields":[],"filter":{}},"role":"importer","source":"default","table":{"name":"rulebase","schema":"public"}},"name":"select_permission importer in table rulebase in source default","reason":"Inconsistent
        object: in table \"rulebase\": in permission for role \"importer\": column \"removed\" does not exist","type":"select_permission"},{"definition":{"comment":"","permission":{"backend_only":false,"check":{},"columns":["created","removed","is_global","name","id","mgm_id"],"filter":{}},"role":"importer","source":"default","table":{"name":"rulebase","schema":"public"}},"name":"update_permission importer in table rulebase in source default","reason":"Inconsistent object: in table \"rulebase\": in permission for role \"importer\": column \"removed\" does not exist","type":"update_permission"},{"definition":{"name":"get_rules_for_owner","schema":"public"},"name":"function get_rules_for_owner in source default","reason":"Inconsistent object: in function \"get_rules_for_owner\": no such function exists: \"get_rules_for_owner\"","type":"function"}]}'
    content_length: '6190'
    content_type: application/json; charset=utf-8
    date: Thu, 31 Oct 2024 09:17:48 GMT
    elapsed: 8
    json:
        code: unexpected
        error: cannot continue due to inconsistent metadata
        internal:
        - definition:
                comment: null
                name: rule
                source: default
                table:
                    name: rulebase_on_gateway
                    schema: public
                using:
                    foreign_key_constraint_on: layer_guard_rule
            name: object_relation rule in table rulebase_on_gateway in source default
            reason: 'Inconsistent object: in table "rulebase_on_gateway": in relationship "rule": no foreign constraint exists on the given column(s)'
            type: object_relation
        - definition:
                comment: null
                definition:
                    function:
                        name: get_rules_for_owner
                        schema: public
                name: get_rules_for_owner
                source: default
                table:
                    name: device
                    schema: public
            name: computed_field get_rules_for_owner in table device in source default
            reason: 'Inconsistent object: in table "device": in computed field "get_rules_for_owner": no such function exists: "get_rules_for_owner"'
            type: computed_field
        - definition:
                comment: null
                name: rulebase_on_gateways
                source: default
                table:
                    name: rule
                    schema: public
                using:
                    foreign_key_constraint_on:
                        column: layer_guard_rule
                        table:
                            name: rulebase_on_gateway
                            schema: public
            name: array_relation rulebase_on_gateways in table rule in source default
            reason: 'Inconsistent object: in table "rule": in relationship "rulebase_on_gateways": no foreign constraint exists on the given column(s)'
            type: array_relation
        - definition:
                comment: ''
                permission:
                    backend_only: false
                    check: {}
                    columns:
                    - layer_guard_rule
                    - dev_id
                    - order_no
                    - rulebase_id
                role: importer
                source: default
                table:
                    name: rulebase_on_gateway
                    schema: public
            name: insert_permission importer in table rulebase_on_gateway in source default
            reason: 'Inconsistent object: in table "rulebase_on_gateway": in permission for role "importer": column "layer_guard_rule" does not exist'
            type: insert_permission
        - definition:
                comment: ''
                permission:
                    allow_aggregations: true
                    columns:
                    - layer_guard_rule
                    - dev_id
                    - order_no
                    - rulebase_id
                    computed_fields: []
                    filter: {}
                role: importer
                source: default
                table:
                    name: rulebase_on_gateway
                    schema: public
            name: select_permission importer in table rulebase_on_gateway in source default
            reason: 'Inconsistent object: in table "rulebase_on_gateway": in permission for role "importer": column "layer_guard_rule" does not exist'
            type: select_permission
        - definition:
                comment: ''
                permission:
                    backend_only: false
                    check: {}
                    columns:
                    - access_rule
                    - action_id
                    - active
                    - dev_id
                    - enforced_on_gateway
                    - is_global
                    - last_change_admin
                    - mgm_id
                    - nat_rule
                    - parent_rule_id
                    - parent_rule_type
                    - removed
                    - rule_action
                    - rule_comment
                    - rule_create
                    - rule_custom_fields
                    - rule_disabled
                    - rule_dst
                    - rule_dst_neg
                    - rule_dst_refs
                    - rule_from_zone
                    - rule_head_text
                    - rule_id
                    - rule_implied
                    - rule_installon
                    - rule_last_seen
                    - rule_name
                    - rule_num
                    - rule_num_numeric
                    - rule_ruleid
                    - rule_src
                    - rule_src_neg
                    - rule_src_refs
                    - rule_svc
                    - rule_svc_neg
                    - rule_svc_refs
                    - rule_time
                    - rule_to_zone
                    - rule_track
                    - rule_uid
                    - rulebase_id
                    - track_id
                    - xlate_rule
                role: importer
                source: default
                table:
                    name: rule
                    schema: public
            name: insert_permission importer in table rule in source default
            reason: 'Inconsistent object: in table "rule": in permission for role "importer": column "enforced_on_gateway" does not exist'
            type: insert_permission
        - definition:
                comment: ''
                permission:
                    allow_aggregations: false
                    columns:
                    - parent_rule_id
                    - removed
                    - rule_create
                    - rule_id
                    - rule_last_seen
                    - xlate_rule
                    - access_rule
                    - active
                    - enforced_on_gateway
                    - is_global
                    - nat_rule
                    - rule_disabled
                    - rule_dst_neg
                    - rule_implied
                    - rule_src_neg
                    - rule_svc_neg
                    - rule_installon
                    - rule_name
                    - rule_ruleid
                    - rule_time
                    - action_id
                    - dev_id
                    - last_change_admin
                    - mgm_id
                    - rulebase_id
                    - rule_from_zone
                    - rule_num
                    - rule_to_zone
                    - track_id
                    - rule_custom_fields
                    - rule_num_numeric
                    - parent_rule_type
                    - rule_action
                    - rule_comment
                    - rule_dst
                    - rule_dst_refs
                    - rule_head_text
                    - rule_src
                    - rule_src_refs
                    - rule_svc
                    - rule_svc_refs
                    - rule_track
                    - rule_uid
                    computed_fields:
                    - rule_relevant_for_tenant
                    filter: {}
                role: importer
                source: default
                table:
                    name: rule
                    schema: public
            name: select_permission importer in table rule in source default
            reason: 'Inconsistent object: in table "rule": in permission for role "importer": column "enforced_on_gateway" does not exist'
            type: select_permission
        - definition:
                comment: ''
                permission:
                    backend_only: false
                    check: {}
                    columns:
                    - created
                    - removed
                    - is_global
                    - name
                    - id
                    - mgm_id
                role: importer
                source: default
                table:
                    name: rulebase
                    schema: public
            name: insert_permission importer in table rulebase in source default
            reason: 'Inconsistent object: in table "rulebase": in permission for role "importer": column "removed" does not exist'
            type: insert_permission
        - definition:
                comment: ''
                permission:
                    allow_aggregations: true
                    columns:
                    - created
                    - removed
                    - is_global
                    - name
                    - id
                    - mgm_id
                    computed_fields: []
                    filter: {}
                role: importer
                source: default
                table:
                    name: rulebase
                    schema: public
            name: select_permission importer in table rulebase in source default
            reason: 'Inconsistent object: in table "rulebase": in permission for role "importer": column "removed" does not exist'
            type: select_permission
        - definition:
                comment: ''
                permission:
                    backend_only: false
                    check: {}
                    columns:
                    - created
                    - removed
                    - is_global
                    - name
                    - id
                    - mgm_id
                    filter: {}
                role: importer
                source: default
                table:
                    name: rulebase
                    schema: public
            name: update_permission importer in table rulebase in source default
            reason: 'Inconsistent object: in table "rulebase": in permission for role "importer": column "removed" does not exist'
            type: update_permission
        - definition:
                name: get_rules_for_owner
                schema: public
            name: function get_rules_for_owner in source default
            reason: 'Inconsistent object: in function "get_rules_for_owner": no such function exists: "get_rules_for_owner"'
            type: function
        path: $.args
    msg: 'Status code was 400 and not [200]: HTTP Error 400: Bad Request'
    redirected: false
    status: 400
    url: http://127.0.0.1:8080/v1/metadata
tpurschke commented 4 weeks ago

line: "deb [arch={{ linux_architecture }}] https://download.docker.com/linux/{{ ansible_facts['distribution']|lower }} {{ os_codename }} stable"

this change should actually also go directly into the standard develop branch via a small PR - @alf-cactus can you create one please?

alf-cactus commented 4 weeks ago

done

tpurschke commented 3 weeks ago

now we only have test errors but the system should be usable (maybe after restarting the UI)

    /usr/local/fworch/test/csharp/FWO.Test/ExportTest.cs(1083,33): error CS0117: 'DeviceReport' does not contain a definition for 'Rules' [/usr/local/fworch/test/csharp/FWO.Test/FWO.Test.csproj]
    /usr/local/fworch/test/csharp/FWO.Test/ExportTest.cs(1134,33): error CS0117: 'DeviceReport' does not contain a definition for 'Rules' [/usr/local/fworch/test/csharp/FWO.Test/FWO.Test.csproj]
    /usr/local/fworch/test/csharp/FWO.Test/ExportTest.cs(1177,55): error CS0117: 'DeviceReport' does not contain a definition for 'Rules' [/usr/local/fworch/test/csharp/FWO.Test/FWO.