fix(panos_security_rule): state merged with existing values

[alperenkose]

Description

When state: merged is used with panos_security_rule for existing entries, it adds default values to existing configuration, like "any" being added to the list of source_zone, or source_ip. And similar issue exist for all the fields with default values in the module.

This PR removes "default" values from sdk_params in module which results in None (null) values for non-provided arguments, and arguments with null values in the invocation are not updated on the device for updating an existing rule. Whereas if rule doesn't exist a new record is being created with default values provided in the newly introduced default_values helper param. If no default values exist in default_values, it fallbacks to SDK default values, and if none found it is set to None.

PR also introduces preset_values for module parameters, which are available fixed selections for list type parameters like "any" or "application-default" for service param. With the addition of preset_values, it fixes updating the list parameters to or from the preset values or user defined entries.

Motivation and Context

Fixes issue #564 but we will keep it open for a while when we have the change in development branch. fixes #563, fixes #467 Also fixes #551 regarding panos_template mode xpath error on second run

How Has This Been Tested?

Tested manually on vmseries.

_Similar issue exists for modules using "default" values in the sdk_params like panos_zone , panos_pbf_rule or others which causes existing values to revert to default if not provided with the merged state. These will be fixed on a separate PR._

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist

• [x] I have updated the documentation accordingly.
• [ ] I have read the CONTRIBUTING document.
• [ ] I have added tests to cover my changes if appropriate.
• [ ] All new and existing tests passed.

[alperenkose]

We need to clarify how to document module default values while they are not in spec but actually applied with present or merged states while creating new resources.

[dmurarasu]

This seems to have broken the fix from #314 and now all security rules show as if they had changes even when no changes were made.

[alperenkose]

Hi @dmurarasu could you please send an example task and output of the issue?

[dmurarasu]

Hi @alperenkose, sorry, I probably should have raised an issue, details below This is ran on a PA-440 unit running 11.1.2

ansible@a801c38859c8:/ansible$ ansible-galaxy collection install paloaltonetworks.panos Starting galaxy collection install process Process install dependency map Starting collection install process Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/paloaltonetworks-panos-2.21.2.tar.gz to /home/ansible/.ansible/tmp/ansible-local-14wf7mps93/tmpev8pz_j5/paloaltonetworks-panos-2.21.2-14miom3s Installing 'paloaltonetworks.panos:2.21.2' to '/home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos' paloaltonetworks.panos:2.21.2 was installed successfully

I'm just running a few rules in this example, just to see the changed output; the rules are already in place but I'll run it twice to show it's the same every time.

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules PLAY [palo] ***

TASK [palo_configure : Create security rules] ***** changed: [pa01] => (item={'rule_name': 'Dynamic Block List Inbound', 'source_zone': ['any'], 'source_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'destination_zone': ['any'], 'destination_ip': ['any'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'top', 'description': 'Blocking known bad IPs inbound'}) changed: [pa01] => (item={'rule_name': 'Dynamic Block List Outbound', 'source_zone': ['any'], 'source_ip': ['any'], 'destination_zone': ['any'], 'destination_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'after', 'existing_rule': 'Dynamic Block List Inbound', 'description': 'Blocking known bad IPs outbound'}) changed: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'})

PLAY RECAP **** pa01 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules PLAY [palo] ***

TASK [palo_configure : Create security rules] ***** changed: [pa01] => (item={'rule_name': 'Dynamic Block List Inbound', 'source_zone': ['any'], 'source_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'destination_zone': ['any'], 'destination_ip': ['any'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'top', 'description': 'Blocking known bad IPs inbound'}) changed: [pa01] => (item={'rule_name': 'Dynamic Block List Outbound', 'source_zone': ['any'], 'source_ip': ['any'], 'destination_zone': ['any'], 'destination_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'after', 'existing_rule': 'Dynamic Block List Inbound', 'description': 'Blocking known bad IPs outbound'}) changed: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'})

PLAY RECAP **** pa01 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Just to show it was ok before

ansible@a801c38859c8:/ansible$ ansible-galaxy collection install paloaltonetworks.panos:2.20.0 Starting galaxy collection install process Process install dependency map Starting collection install process Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/paloaltonetworks-panos-2.20.0.tar.gz to /home/ansible/.ansible/tmp/ansible-local-37472fsj_7b/tmpx42tjxmv/paloaltonetworks-panos-2.20.0-oqng52gi Installing 'paloaltonetworks.panos:2.20.0' to '/home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos' paloaltonetworks.panos:2.20.0 was installed successfully

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules

PLAY [palo] ***

TASK [palo_configure : Create security rules] ***** ok: [pa01] => (item={'rule_name': 'Dynamic Block List Inbound', 'source_zone': ['any'], 'source_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'destination_zone': ['any'], 'destination_ip': ['any'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'top', 'description': 'Blocking known bad IPs inbound'}) ok: [pa01] => (item={'rule_name': 'Dynamic Block List Outbound', 'source_zone': ['any'], 'source_ip': ['any'], 'destination_zone': ['any'], 'destination_ip': ['Cisco Talos', 'panw-torexit-ip-list', 'panw-bulletproof-ip-list', 'panw-highrisk-ip-list', 'panw-known-ip-list'], 'application': ['any'], 'service': ['any'], 'action': 'drop', 'location': 'after', 'existing_rule': 'Dynamic Block List Inbound', 'description': 'Blocking known bad IPs outbound'}) ok: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'})

PLAY RECAP **** pa01 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

When it detects changes all the time and I run it with verbosity the only changes that I can see in the before and after section:

before: "group_profile": [ "Inbound Network Security" ], "group_tag": null, "hip_profiles": null,

after: "group_profile": "Inbound Network Security", "group_tag": null, "hip_profiles": [ "any" ],

Commiting and rerunning just shows the same again.

Full output below:

ansible@a801c38859c8:/ansible$ ansible-playbook -i local.inventory config_playbook.yml --tags=security_rules -vvvv ansible-playbook [core 2.15.1] config file = None configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible-playbook python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3) jinja version = 3.1.2 libyaml = True No config file found; using defaults setting up inventory plugins Loading collection ansible.builtin from host_list declined parsing /ansible/local.inventory as it did not pass its verify_file() method auto declined parsing /ansible/local.inventory as it did not pass its verify_file() method yaml declined parsing /ansible/local.inventory as it did not pass its verify_file() method Parsed /ansible/local.inventory inventory source with ini plugin Loading collection paloaltonetworks.panos from /home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos statically imported: /ansible/roles/palo_configure/tasks/mgmt_config.yml statically imported: /ansible/roles/palo_configure/tasks/email_profile.yml statically imported: /ansible/roles/palo_configure/tasks/zones.yml statically imported: /ansible/roles/palo_configure/tasks/interfaces.yml statically imported: /ansible/roles/palo_configure/tasks/objects.yml statically imported: /ansible/roles/palo_configure/tasks/security_profiles.yml statically imported: /ansible/roles/palo_configure/tasks/security_rules.yml statically imported: /ansible/roles/palo_configure/tasks/nat_rules.yml statically imported: /ansible/roles/palo_configure/tasks/pbf_rules.yml Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.10/dist-packages/ansible/plugins/callback/default.py Skipping callback 'default', as we already have a stdout callback. Skipping callback 'minimal', as we already have a stdout callback. Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: config_playbook.yml ***** Positional arguments: config_playbook.yml verbosity: 4 connection: smart timeout: 10 become_method: sudo tags: ('security_rules',) inventory: ('/ansible/local.inventory',) forks: 5 1 plays in config_playbook.yml

PLAY [palo] ***

TASK [palo_configure : Create security rules] ***** task path: /ansible/roles/palo_configure/tasks/security_rules.yml:1

EXEC /bin/sh -c 'echo ~ansible && sleep 0' EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818 `" && echo ansible-tmp-1728555898.4642293-732-217420695437818="` echo /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818 `" ) && sleep 0' Using module file /home/ansible/.ansible/collections/ansible_collections/paloaltonetworks/panos/plugins/modules/panos_security_rule.py PUT /home/ansible/.ansible/tmp/ansible-local-727vaemhhyn/tmpfb9sfkfk TO /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/AnsiballZ_panos_security_rule.py EXEC /bin/sh -c 'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/ /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/AnsiballZ_panos_security_rule.py && sleep 0' EXEC /bin/sh -c '/usr/bin/python3 /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/AnsiballZ_panos_security_rule.py && sleep 0' EXEC /bin/sh -c 'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1728555898.4642293-732-217420695437818/ > /dev/null 2>&1 && sleep 0' changed: [pa01] => (item={'rule_name': 'GP Connection', 'source_zone': ['External_LAN'], 'source_ip': ['any'], 'destination_zone': ['External_LAN'], 'destination_ip': ['any'], 'application': ['ipsec', 'panos-global-protect', 'ssl'], 'service': ['application-default'], 'action': 'allow', 'description': 'Permit GP Gateway Login', 'group_profile': 'Inbound Network Security', 'tag': ['Global Protect'], 'location': 'after', 'existing_rule': 'Dynamic Block List Outbound'}) => { "after": { "action": "allow", "antivirus": null, "application": [ "ipsec", "panos-global-protect", "ssl" ], "category": [ "any" ], "data_filtering": null, "description": "Permit GP Gateway Login", "destination_devices": [ "any" ], "destination_ip": [ "any" ], "destination_zone": [ "External_LAN" ], "disable_server_response_inspection": false, "disabled": false, "file_blocking": null, "group_profile": "Inbound Network Security", "group_tag": null, "hip_profiles": [ "any" ], "icmp_unreachable": null, "log_end": true, "log_setting": null, "log_start": false, "negate_destination": false, "negate_source": false, "negate_target": null, "rule_name": "GP Connection", "rule_type": "universal", "schedule": null, "service": [ "application-default" ], "source_devices": [ "any" ], "source_ip": [ "any" ], "source_user": [ "any" ], "source_zone": [ "External_LAN" ], "spyware": null, "tag_name": null, "target": null, "url_filtering": null, "uuid": "e74b64af-9806-4011-bf9a-23f2fc41fe28", "vulnerability": null, "wildfire_analysis": null }, "ansible_loop_var": "item", "before": { "action": "allow", "antivirus": null, "application": [ "ipsec", "panos-global-protect", "ssl" ], "category": [ "any" ], "data_filtering": null, "description": "Permit GP Gateway Login", "destination_devices": [ "any" ], "destination_ip": [ "any" ], "destination_zone": [ "External_LAN" ], "disable_server_response_inspection": false, "disabled": false, "file_blocking": null, "group_profile": [ "Inbound Network Security" ], "group_tag": null, "hip_profiles": null, "icmp_unreachable": null, "log_end": true, "log_setting": null, "log_start": false, "negate_destination": false, "negate_source": false, "negate_target": null, "rule_name": "GP Connection", "rule_type": "universal", "schedule": null, "service": [ "application-default" ], "source_devices": [ "any" ], "source_ip": [ "any" ], "source_user": [ "any" ], "source_zone": [ "External_LAN" ], "spyware": null, "tag_name": null, "target": null, "url_filtering": null, "uuid": "e74b64af-9806-4011-bf9a-23f2fc41fe28", "vulnerability": null, "wildfire_analysis": null }, "changed": true, "diff": { "after": "\n\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tipsec\n\t\tpanos-global-protect\n\t\tssl\n\t\n\t\n\t\tapplication-default\n\t\n\t\n\t\tany\n\t\n\tallow\n\tno\n\tyes\n\tPermit GP Gateway Login\n\tuniversal\n\tno\n\tno\n\tno\n\t\n\t\tno\n\t\n\t\n\t\t\n\t\t\tInbound Network Security\n\t\t\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\n", "before": "\n\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tExternal_LAN\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\t\n\t\tipsec\n\t\tpanos-global-protect\n\t\tssl\n\t\n\t\n\t\tapplication-default\n\t\n\t\n\t\tany\n\t\n\tallow\n\tno\n\tyes\n\tPermit GP Gateway Login\n\tuniversal\n\tno\n\tno\n\tno\n\t\n\t\tno\n\t\n\t\n\t\t\n\t\t\tInbound Network Security\n\t\t\n\t\n\t\n\t\tany\n\t\n\t\n\t\tany\n\t\n\n" }, "invocation": { "module_args": { "action": "allow", "antivirus": null, "api_key": null, "application": [ "ipsec", "panos-global-protect", "ssl" ], "audit_comment": null, "category": null, "commit": false, "data_filtering": null, "description": "Permit GP Gateway Login", "destination_ip": [ "any" ], "destination_zone": [ "External_LAN" ], "device_group": "shared", "devicegroup": null, "disable_server_response_inspection": null, "disabled": null, "existing_rule": "Dynamic Block List Outbound", "file_blocking": null, "gathered_filter": null, "group_profile": "Inbound Network Security", "group_tag": null, "hip_profiles": null, "icmp_unreachable": null, "ip_address": null, "location": "after", "log_end": null, "log_setting": null, "log_start": null, "negate_destination": null, "negate_source": null, "negate_target": null, "password": null, "port": 443, "provider": { }, "rule_name": "GP Connection", "rule_type": null, "rulebase": null, "schedule": null, "service": [ "application-default" ], "source_ip": [ "any" ], "source_user": null, "source_zone": [ "External_LAN" ], "spyware": null, "state": "present", "tag_name": null, "target": null, "url_filtering": null, "username": "admin", "uuid": null, "vsys": "vsys1", "vulnerability": null, "wildfire_analysis": null } }, "item": { "action": "allow", "application": [ "ipsec", "panos-global-protect", "ssl" ], "description": "Permit GP Gateway Login", "destination_ip": [ "any" ], "destination_zone": [ "External_LAN" ], "existing_rule": "Dynamic Block List Outbound", "group_profile": "Inbound Network Security", "location": "after", "rule_name": "GP Connection", "service": [ "application-default" ], "source_ip": [ "any" ], "source_zone": [ "External_LAN" ], "tag": [ "Global Protect" ] } }

[alperenkose]

@dmurarasu could you please create an issue for this to be tracked with the config_playbook.yml content you have run?