[Bug Report] ASG Lambda delicensing not working?

[kverstr]

Describe the bug

I enabled delicensing in the asg module. I can see in the lambda logs that is succesfully delicensed the instance. However it is still available in panorama and is also still using a license in the paloalto support portal.

Am I missing something here? What will the lambda do exactly? As far as I can tell you only have to set delicense_enabled = true and create the SSM_param with your panorama credentials. It seems the lambda is doing nothing regarding to delicensing .

Module Version

v1.0.9

Terraform version

v1.6.2

Expected behavior

when I scale-down the ASG from 3 to 2:

• 1 instance starts draining connections and gets removed from target_group
• the ASG lambda kicks in and delicenses the instance from panorama
• (license is removed from instance in paloalto support portal)

Current behavior

when I scale-down the ASG from 3 to 2:

• instances scales down

• lambda kicks in and says in the logs it successfully delicensed the instance

• instance is still there in panorama

• instance is still taking a license in paloalto support portal

Anything else to add?

Let me know if you need more info, Not sure if this issue is related to the module or if this is a problem with paloalto/panorama. But since the feature is here in the module I expect that this should work.

[sebastianczech]

Hi @kverstr , have you configured API key on Panorama as described in prerequisites e.g. step 5 for combined design: https://github.com/PaloAltoNetworks/terraform-aws-vmseries-modules/tree/main/examples/combined_design_autoscale#prerequisites ?

If you don't configured it, then Lambda will send request to Panorama to delicense firewall (so this will be successful), but VM-Series will not be delicensed, because there is no valid API key on Panorama and Panorama cannot properly authenticate in support portal in order to finish delicense process.

[kverstr]

@sebastianczech I wasn't aware of these prerequisites. Maybe these prereqs should be added to the ASG module readme? (also the link to an example in the readme is broken).

This is all configuration on panorama it seems. Currently we had to manually delicense in the paloalto support portal as well so I guess that api_key can automate that as well and that we don't have that set up. I will contact our onpremise network team about this, thanks! Any chance you know if I can also check this configuration in the UI in panorama? As the docs refer to cli commands to setup this api_key.

Kr

[sebastianczech]

@kverstr I fixed links in README for modules and added direct links to example with autoscaling in #398 .

As far as I know API key can be set only via CLI in Panorama. In UI you can only verify if delicense requests was finished successfuly or not - in order to check it, please go in Panorama on monitor tab and them from left menu go to system logs - here you can filter logs, which contains license in description.

If possible please let us know, if after you set API key on Panorama, delicensing via Lambda works.

[kverstr]

@sebastianczech Thanks! So the network team told me they set up an api_key. However it still seems to be doing nothing.. I only see this in the logs, which seems to be an empty response?

[sebastianczech]

In the lab I checked current version of asg module.

When I have API key defined in Panorama, then VM-Series was delicensed:

When I removed API key from Panorama, then I have the same situation as yours - empty response:

So please check if API key defined in Panorama:

admin@Panorama> request license api-key show

is the same as you have in Customer Support Portal (in menu Products -> API Key Management and then in selectbox Licensing API).

If it's the same, then please provide me exact versions of:

• PAN-OS on Panorama
• Panorama Software Firewall License Plugin

I can then replicate it in the lab with the same versions of PAN-OS and plugin.

[kverstr]

Ok thanks, so I was able to verify that the api_key was not set by running the cli command. But since I don't have super user privileges on the support portal, I could not see the "API Key Management" tab. Talked to the network team again , they confirmed something must've gone wrong last time. I re-tested scaling the ASG and now the license was released successfully in the support portal as well as in panorama!

However, what I see is that the delicense event happens a few mins before the actual termination of the EC2. I'm wondering if this could be an issue and if maybe I need to play with some timeout variables. The ASG drains the connections to the instance and seems to remove the instance from the target_group as a first step, about 5mins later the delicense happens. So maybe it's fine and the delicense happens after all connections were drained. Do you know what exactly happens in this process? (just want to be sure cycling instances doesn't cause impact)

[sebastianczech]

I'm happy, that Lambda is working correctly. Thank you for feedback.

Regarding delicense process - VM-Series is removed after ASG drains connections and then it's delicensed by Lambda. Nevertheless please take into account that:

• there is default cooldown, which is set to 300 seconds, so when a scaling activity completes then another scaling activity can start after that time (https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-scaling-cooldowns.html) - in meantime destroyed VM-Series is already delicensed
• while bootstrapping new VM-Series from scratch it takes ~30 minutes to finish whole process (create an alarm according to metrics, create new EC2 instance, bootstrap VM-Series correctly, connect to Panorama, push device group and template from Panorama to VM-Series and do optional content update) - so you have a lot of time to delicense destroyed VM-Series

I hope it clarifies the process. Please let me know if you need anything more regarding that subject or we can close issue, as root cause of the problem was found ?

[kverstr]

Thanks for all the help and info. Will close the issue.

Kr