Pan-os version 11 genericpolling DT fails

adambaumeister commented 7 months ago

All generic polling playbooks that are looking at jobs like download/install software fail on version 11 due to duplicate job ids in the context.

GrantGabbert commented 2 months ago

This issue seems to've started in 10.2 NGFW code

alperenkose commented 1 month ago

PANOS XML API literally returns multiple records for the same job ids after 10.2, which I believe is the root cause for this. See below response I tested on 10.2.9-h1.

<response status="success">
    <result>
        <job>
            <tenq>2024/07/08 04:57:51</tenq>
            <tdeq>04:57:51</tdeq>
            <id>4</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/08 04:58:03</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/08 04:58:03</progress>
            <details>
                <line>Successfully downloaded</line>
                <line>Transferring a copy of image to HA peer</line>
                <line>Preloading into software manager</line>
                <line>Successfully loaded into software manager</line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/03 01:02:03</tenq>
            <tdeq>01:02:03</tdeq>
            <id>3</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/03 01:02:05</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/03 01:02:05</progress>
            <details>
                <line>File successfully downloaded </line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/01 06:39:16</tenq>
            <tdeq>06:39:29</tdeq>
            <id>2</id>
            <user/>
            <type>AutoCom</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/01 06:40:26</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>100</progress>
            <details>
                <line>Configuration committed successfully</line>
                <line>Successfully committed last configuration</line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/08 04:57:51</tenq>
            <tdeq>04:57:51</tdeq>
            <id>4</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/08 04:58:03</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/08 04:58:03</progress>
            <details>
                <line>Successfully downloaded</line>
                <line>Transferring a copy of image to HA peer</line>
                <line>Preloading into software manager</line>
                <line>Successfully loaded into software manager</line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/03 01:02:03</tenq>
            <tdeq>01:02:03</tdeq>
            <id>3</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/03 01:02:05</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/03 01:02:05</progress>
            <details>
                <line>File successfully downloaded </line>
            </details>
            <warnings/>
        </job>
    </result>
</response>

GrantGabbert commented 1 month ago

I did notice that the PAN job API was returning duplicate entries in the 10.2/11 codebase, but not before. And that seemed like the root cause. But, there's also something else going on in XSOAR.

The DT for the poller is - PANOS.JobStatus(val.status!=="FIN").id. So even with duplicate job entries returned from PANOS, I would expect that DT should still give 0 results once the job is completed.

What I'm seeing in the context is more of a history, instead of just the last run.

For example, the last upgrade I did, the command result show the 2 "FIN" entries in the war room...

PAN-OS Job Status

hostid	id	positionInQ	progress	result	status	stoppable	tenq	tfin	type
021201010640	37	0	2024/06/17 16:22:25	OK	FIN	no	2024/06/17 16:16:09	2024/06/17 16:22:25	SWInstall
021201010640	37	0	2024/06/17 16:22:25	OK	FIN	no	2024/06/17 16:16:09	2024/06/17 16:22:25	SWInstall

But the context shows 4 entries with different percentages....

{
  "JobStatus": [
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "0",
      "result": "PEND",
      "status": "ACT",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": null,
      "type": "SWInstall",
      "user": null
    },
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "37",
      "result": "PEND",
      "status": "ACT",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": null,
      "type": "SWInstall",
      "user": null
    },
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "30",
      "result": "PEND",
      "status": "ACT",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": null,
      "type": "SWInstall",
      "user": null
    },
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "2024/06/17 16:22:25",
      "result": "OK",
      "status": "FIN",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": "2024/06/17 16:22:25",
      "type": "SWInstall",
      "user": null
    }
  ]
}

alperenkose commented 6 days ago

@GrantGabbert I did a PR on demisto/content to filter by job id using show jobs id command. It resolves this issue as it returns a single entry when you query with a job id now. But your point on DT filtering remains the same. Maybe we can close this issue and investigate DT filtering separately.

GrantGabbert commented 3 days ago

So the issue was for generic polling failures, and that hasn't been resolved yet. So I think it would make the most sense to keep this issue open until that is resolved (or it's decided to not fix it).

Also, with the changes to the pan-os integration, are there also changes that need to be added in this repo to utilize new show job functionality?

PaloAltoNetworks / xsoar-panos-upgrade-automation

Pan-os version 11 genericpolling DT fails #14

PAN-OS Job Status