removeWhereUsed:delete failing

bethatasitmay commented 1 year ago

Describe the bug

removeWhereUsed:delete is not working correctly - it is stopping with errors when trying to remove a rule which matches the filter (e.g., the host is the last source or destination in the rule).

Expected behavior

Rules where an object is the last member should get deleted and the command should continue running until the end.

I last ran this back in January 2023 on what I believe was a release of v2.0.64 and it worked OK.

Current behavior

It appears to delete one rule and then stop with errors. I say it that was as multiple runs have a different rulename just before the errors. I did confirm from the Configuration log that the rule name listed just before the errors is getting deleted.

I have an earlier run on v2.0.73 and then I upgraded to the current release as of this writing (v2.1.13) and got these results for two runs:

PS C:\tools\pan> PS C:\tools\pan> PS C:\tools\pan> php -r "require_once 'C:/tools/pan/pan-os-php/utils/pan-os-php.php';" type=address in=api://panorama.domain.com location=any actions=removeWhereUsed:delete 'filter=(name eq object01) or (name eq object02) or (name eq object03) or (name eq object04) or (name eq object05) or (name eq object06) or (name eq object07) or (name eq object08) or (name eq object09) or (name eq object10) or (name eq object11) or (name eq object12) or (name eq object13) or (name eq object14)'

*** pan-os-php.php type=address UTILITY **

PAN-OS-PHP version: 2.1.13 [WIN] [8.1.16]
Downloading config from API...
Detected platform type is 'panorama'
No 'template' provided so using default ='any'
filter after sanitization : (name eq object01) or (name eq object02) or (name eq object03) or (name eq object04) or (name eq object05) or (name eq object06) or (name eq object07) or (name eq object08) or (name eq object09) or (name eq object10) or (name eq object11) or (name eq object12) or (name eq object13) or (name eq object14)
Loading configuration through PAN-OS-PHP library... (1.49 seconds, 113.25 mb memory)
PAN-OS version: 91
PAN-OS APP-ID version: 8741-8213
PAN-OS Device timezone: US/Pacific is used. actual time: 2023/08/25 17:01:39

processing store 'PanoramaConf: / AddressStore:addresses' that holds 7151 objects
- object 'object01' passing through Action='removeWhereUsed' Args: actionIfLastMemberInRule=delete,
  - last member so deleting /firewall01/security:Artiva-02/source PHP Fatal error: Uncaught TypeError: count(): Argument #1 ($value) must be of type Countable|array, null given in C:\tools\pan\pan-os-php\lib\container-classes\ObjRuleContainer.php:41 Stack trace:
    0 C:\tools\pan\pan-os-php\lib\object-classes\trait\AddressCommon.php(258): ObjRuleContainer->count()
    
    1 C:\tools\pan\pan-os-php\lib\object-classes\trait\AddressCommon.php(459): Address->__removeWhereIamUsed()
    
    2 C:\tools\pan\pan-os-php\utils\common\actions-address.php(329): Address->API_removeWhereIamUsed()
    
    3 C:\tools\pan\pan-os-php\utils\common\CallContext.php(112): {closure}()
    
    4 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(1805): CallContext->executeAction()
    
    5 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(227): UTIL->time_to_process_objects()
    
    6 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(211): UTIL->utilStart()
    
    7 C:\tools\pan\pan-os-php\lib\misc-classes\PH.php(1162): UTIL->__construct()
    
    8 C:\tools\pan\pan-os-php\utils\pan-os-php.php(117): PH::callPANOSPHP()
    
    9 Command line code(1): require_once('...')
    
    10 {main}
    
    thrown in C:\tools\pan\pan-os-php\lib\container-classes\ObjRuleContainer.php on line 41 PS C:\tools\pan> PS C:\tools\pan> PS C:\tools\pan> php -r "require_once 'C:/tools/pan/pan-os-php/utils/pan-os-php.php';" type=address in=api://panorama.domain.com location=any actions=removeWhereUsed:delete 'filter=(name eq object01) or (name eq object02) or (name eq object03) or (name eq object04) or (name eq object05) or (name eq object06) or (name eq object07) or (name eq object08) or (name eq object09) or (name eq object10) or (name eq object11) or (name eq object12) or (name eq object13) or (name eq object14)'

*** pan-os-php.php type=address UTILITY **

PAN-OS-PHP version: 2.1.13 [WIN] [8.1.16]
Downloading config from API...
Detected platform type is 'panorama'
No 'template' provided so using default ='any'
filter after sanitization : (name eq object01) or (name eq object02) or (name eq object03) or (name eq object04) or (name eq object05) or (name eq object06) or (name eq object07) or (name eq object08) or (name eq object09) or (name eq object10) or (name eq object11) or (name eq object12) or (name eq object13) or (name eq object14)
Loading configuration through PAN-OS-PHP library... (1.45 seconds, 113.25 mb memory)
PAN-OS version: 91
PAN-OS APP-ID version: 8741-8213
PAN-OS Device timezone: US/Pacific is used. actual time: 2023/08/25 17:02:10

processing store 'PanoramaConf: / AddressStore:addresses' that holds 7151 objects
- object 'object01' passing through Action='removeWhereUsed' Args: actionIfLastMemberInRule=delete,
  - last member so deleting /firewall01/security:Artiva-15/source PHP Fatal error: Uncaught TypeError: count(): Argument #1 ($value) must be of type Countable|array, null given in C:\tools\pan\pan-os-php\lib\container-classes\ObjRuleContainer.php:41 Stack trace:
    0 C:\tools\pan\pan-os-php\lib\object-classes\trait\AddressCommon.php(258): ObjRuleContainer->count()
    
    1 C:\tools\pan\pan-os-php\lib\object-classes\trait\AddressCommon.php(459): Address->__removeWhereIamUsed()
    
    2 C:\tools\pan\pan-os-php\utils\common\actions-address.php(329): Address->API_removeWhereIamUsed()
    
    3 C:\tools\pan\pan-os-php\utils\common\CallContext.php(112): {closure}()
    
    4 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(1805): CallContext->executeAction()
    
    5 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(227): UTIL->time_to_process_objects()
    
    6 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(211): UTIL->utilStart()
    
    7 C:\tools\pan\pan-os-php\lib\misc-classes\PH.php(1162): UTIL->__construct()
    
    8 C:\tools\pan\pan-os-php\utils\pan-os-php.php(117): PH::callPANOSPHP()
    
    9 Command line code(1): require_once('...')
    
    10 {main}
    
    thrown in C:\tools\pan\pan-os-php\lib\container-classes\ObjRuleContainer.php on line 41

*** pan-os-php.php type=address UTILITY **

PAN-OS-PHP version: 2.1.13 [WIN] [8.1.16]
Downloading config from API...
Detected platform type is 'panorama'
No 'template' provided so using default ='any'
filter after sanitization : (name eq object01) or (name eq object02) or (name eq object03) or (name eq object04) or (name eq object05) or (name eq object06) or (name eq object07) or (name eq object08) or (name eq object09) or (name eq object10) or (name eq object11) or (name eq object12) or (name eq object13) or (name eq object14)
Loading configuration through PAN-OS-PHP library... (1.45 seconds, 113.25 mb memory)
PAN-OS version: 91
PAN-OS APP-ID version: 8741-8213
PAN-OS Device timezone: US/Pacific is used. actual time: 2023/08/25 17:02:10

processing store 'PanoramaConf: / AddressStore:addresses' that holds 7151 objects
- object 'object01' passing through Action='removeWhereUsed' Args: actionIfLastMemberInRule=delete,
  - last member so deleting /LV-INT-FW1-2/security:Artiva-15/source PHP Fatal error: Uncaught TypeError: count(): Argument #1 ($value) must be of type Countable|array, null given in C:\tools\pan\pan-os-php\lib\container-classes\ObjRuleContainer.php:41 Stack trace:
    0 C:\tools\pan\pan-os-php\lib\object-classes\trait\AddressCommon.php(258): ObjRuleContainer->count()
    
    1 C:\tools\pan\pan-os-php\lib\object-classes\trait\AddressCommon.php(459): Address->__removeWhereIamUsed()
    
    2 C:\tools\pan\pan-os-php\utils\common\actions-address.php(329): Address->API_removeWhereIamUsed()
    
    3 C:\tools\pan\pan-os-php\utils\common\CallContext.php(112): {closure}()
    
    4 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(1805): CallContext->executeAction()
    
    5 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(227): UTIL->time_to_process_objects()
    
    6 C:\tools\pan\pan-os-php\utils\lib\UTIL.php(211): UTIL->utilStart()
    
    7 C:\tools\pan\pan-os-php\lib\misc-classes\PH.php(1162): UTIL->__construct()
    
    8 C:\tools\pan\pan-os-php\utils\pan-os-php.php(117): PH::callPANOSPHP()
    
    9 Command line code(1): require_once('...')
    
    10 {main}
    
    thrown in C:\tools\pan\pan-os-php\lib\container-classes\ObjRuleContainer.php on line 41

Steps to reproduce

php -r "require_once 'C:/tools/pan/pan-os-php/utils/pan-os-php.php';" type=address in=api://panorama.domain.com location=any actions=removeWhereUsed:delete 'filter=(name eq object01) or (name eq object02) or (name eq object03) or (name eq object04) or (name eq object05) or (name eq object06) or (name eq object07) or (name eq object08) or (name eq object09) or (name eq object10) or (name eq object11) or (name eq object12) or (name eq object13) or (name eq object14)'

Context

I use this for decommissioning hosts and to remove rules & address objects.

Your Environment

Version used: 2.0.73 (PHP 8.1.16) [1 attempt] and 2.1.13 (PHP 8.1.16) [2 attempts, shown above]
Operating System and version (desktop or mobile): Windows 10 & PowerShell 5.1.19041.3031

swaschkut commented 1 year ago

Please can you share more details about the SecurityRule configuration, best via the privat way done in the past? I need to create a similar configuration to reproduce your issue

bethatasitmay commented 1 year ago

Thanks.

How would like it? Screenshot, rule dump (either from GUI or PAN-OS-PHP), XML, set commands?

Oh, if it's XML I'll need you to tell me how to do that.

swaschkut commented 1 year ago

the output of this is enough:

pan-os-php type=rule in=api://panorama.domain.com location=LV-INT-FW1-2 'filter=(name eq Artiva-15)'

or in your case: php -r "require_once 'C:/tools/pan/pan-os-php/utils/pan-os-php.php';" type=rule in=api://panorama.domain.com location=LV-INT-FW1-2 'filter=(name eq Artiva-15)'

please send it via E-Mail, swaschkut (at) paloaltonetworks.com

bethatasitmay commented 1 year ago

Since it is deleting one rule prior to erroring out, that rule is gone. I'll send the next hit.

swaschkut commented 1 year ago

thanks for sharing, now I am getting where the problem is:

the Rules which error out does have only one address object configured for source and for destination; and in a more detail the same object is configured in source and destination;

so what the script is doing: 1) checking rule source and remove the address object -> it is last object so delete complete rule 2) script has still in memory that the same rule in destination has also the object which need to be remove 3) as rule is no longer => error

I will work on this, as I my assumption about the issue is correct.

There was no change in between, if such a rule would be available in the past, you would also run into this issue in the past.

bethatasitmay commented 1 year ago

Ah, yes, that makes sense. I do know that at least some of the previous runs where it worked only the source or destination was the last member, but not both. Since I always ExportToExcel first using the same filter, I can go back and confirm if you like.

swaschkut commented 1 year ago

I just published version 2.1.14 - which include a fix for you problem; please validate and give feedback

bethatasitmay commented 1 year ago

That fixed - it's working great now. Thanks again!

swaschkut commented 1 year ago

fixed with #773

PaloAltoNetworks / pan-os-php