FMC ACP rule hit count - feature request

[NetDevAutomate]

Are there any plans to use the 6.4 features to be able to get ACP rule hit counts?

[daxm]

Our goal is to eventually support access to ALL FMC API objects. That said, we do them as we can. Are you interested/willing to help the project out and work on that?

[NetDevAutomate]

I can certainly try, I would love to be able to contribute as a great project!

[daxm]

Any progress on this feature? If not, I'll give it a try.

[daxm]

I've got HitCount to work for 'device' and 'fetchZeroHitCount' but the prefilter doesn't work because I don't have a Class for prefilter yet. It is in the Develop branch currently: https://github.com/daxm/fmcapi/tree/develop

[daxm]

NOTE: Only for the GET method. I'm still working on PUT and DELETE methods. FYI

[NetDevAutomate]

You legend, SO sorry for not getting this done, I’ve been so swamped with work.

Hopefully I can also learn from this so a second reason for most sincere thanks.

Kind regards,

Andy

On 12 Sep 2019, at 19:02, daxm notifications@github.com wrote:

NOTE: Only for the GET method. I'm still working on PUT and DELETE methods. FYI

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/daxm/fmcapi/issues/39?email_source=notifications&email_token=ACM6ZXSTDNOGXU2PPPAK5DTQJJ7Z5A5CNFSM4IKVWBFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6SXWBI#issuecomment-530938629, or mute the thread https://github.com/notifications/unsubscribe-auth/ACM6ZXUEA4TI4UDAP6O3AL3QJJ7Z5ANCNFSM4IKVWBFA.

[daxm]

hitcount and prefilter_policy GET features are now in Master branch. However, hitcount also can do PUT or DELETE but I haven't written those features into fmcapi yet. Test it out and see what you think!

[NetDevAutomate]

Do you have an example/unit test for hit count? There was nothing in the repo and the code base has had a pretty major refactor.

[daxm]

I’m not in front of my computer right now but I’ll describe that later

It is far better to grasp the Universe as it really is than to persist in delusion, however satisfying and reassuring. — Carl Sagan

On Oct 26, 2019, at 11:51, Andy Taylor notifications@github.com wrote:

 Do you have an example/unit test for hit count? There was nothing in the repo and the code base has had a pretty major refactor.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[daxm]

Yes, we did a major refactor to better organize the code. Hopefully (other than trying to find things) this reorg didn't break your workflow!

I must have lost some code or "thought" HitCount was done. I just spent the day getting the GET features to work. There is still an issue that I can't seem to resolve (you can't add an AccessRule to the HitCount's filter via the rule's name... it provides/expects the wrong value!) Anyway, here is the sample code: https://github.com/daxm/fmcapi/blob/master/unit_tests/hit_counts.py

Note: These changes aren't published to PyPi yet so you'll have to clone the repo to try them out.

[NetDevAutomate]

When I run the test, sadly I get:

WARNING:root:Either:

1. Payload too large. FMC can only handle a payload of 2048000 bytes. 2.The payload contains an unprocessable or unreadable entity such as a invalid attribut name or incorrect JSON syntax

The code is largely untouched but removed the add acp rule but did add a caching function as found it makes a huge difference when running more than one task on a requests dataset:

import logging
import fmcapi
import time
import requests_cache

# ### Set these variables to match your environment. ### #

host = ""
username = ""
password = ""
autodeploy = False
logname = "TestingUserScript.log"
pagelimit = 500
debug = False
device = ""
cache_name = "fmcacpi_cache"

requests_cache.install_cache(
    cache_name=cache_name, expire_after=21600
)

def main():
    with fmcapi.FMC(
        host=host,
        username=username,
        password=password,
        autodeploy=autodeploy,
        limit=pagelimit,
        file_logging=logname,
        debug=debug,
    ) as fmc1:
        logging.info("# ### Starting!!! ### #\n\n")
        hitcounts(fmc=fmc1, device_name=device)

def hitcounts(fmc, device_name="", prefilter_id=""):
    if not device_name and not prefilter_id:
        return f"Name of an actual device or prefilter ID is required for the HitCounts test to work... skipping test."
    # logging.info(
    #     "In preparation for testing HitCounts method, set up some known objects in the FMC."
    # )
    # starttime = str(int(time.time()))
    # namer = f"test__hitcounts_{starttime}"

    # Get the device
    device1 = fmcapi.DeviceRecords(fmc=fmc, name=device_name)
    device1.get()

    # In case there is no ACP Rule build a temp one.
    acprule1 = fmcapi.AccessRules(fmc=fmc, acp_id=device1.accessPolicy["id"])
    # acprule1 = fmcapi.AccessRules(fmc=fmc, acp_name=device1.accessPolicy['name'])
    # acprule1.name = namer
    # acprule1.action = "ALLOW"
    # acprule1.post()
    # time.sleep(1)
    acprule1.get()

    hitcounter1 = None
    if prefilter_id:
        hitcounter1 = fmcapi.HitCounts(
            fmc=fmc, prefilter_id=prefilter_id, device_name=device_name
        )
    else:
        hitcounter1 = fmcapi.HitCounts(
            fmc=fmc, acp_id=device1.accessPolicy["id"], device_name=device_name
        )
        """ 
        Searching for AccessRule by name returns the "correct" ID for the rule but HitCount shows a completely
        different ID so it doesn't match.
        # hitcounter1.acp_rules(action="add", name="Permit HQ LAN")
        If you know the ID that HitCount is looking for a specific rule:
        # hitcounter1.acp_rules(action="add", acp_rule_id="005056B5-44E6-0ed3-0000-000268434433")
        """
    for result in hitcounter1.get():
        print(result)

    logging.info("Test HitCount done.")

    logging.info("Cleanup of testing HitCount methods.")
    # acprule1.delete()
    time.sleep(1)
    logging.info("Cleanup of objects for HitCount test done.\n\n")

if __name__ == "__main__":
    main()

[daxm]

Note the """ quoted note near the end of the code. I haven't figured out why but the HitCount "ID" for the AccessRule doesn't match what the AccessRule's ID is.

Correction: You need to find the acp_rule_id that HitCount wants... which isn't the same thing as the AccessRule id.

[daxm]

Check out the debug output for what is returned from GET'ting the AccessRule vs what the HitCounts sees as the id for that particular AccessRule.

[NetDevAutomate]

Same problem when using name when using:

acprule1 = fmcapi.AccessRules(fmc=fmc, acp_name=device1.accessPolicy['name'])

Log file:

2019/11/04-15:53:36 - INFO:apiclasstemplate.py:135 - GET success. Object with name: "<device name>" and id: "<device id>" fetched from FMC.
2019/11/04-15:53:36 - INFO:apiclasstemplate.py:135 - GET success. Object with name: "<policy name>" and id: "<policy id>" fetched from FMC.
2019/11/04-16:09:27 - INFO:apiclasstemplate.py:135 - GET success. Object with name: "<device name>" and id: "<device id>" fetched from FMC.
2019/11/04-16:09:27 - WARNING:fmc.py:245 - Either:
    1. Payload too large.  FMC can only handle a payload of 2048000 bytes.
    2.The payload contains an unprocessable or unreadable entity such as a invalid attribut name or incorrect JSON syntax
2019/11/04-16:09:27 - ERROR:fmc.py:253 - Error in POST operation --> 422 Client Error: Unprocessable Entity for url: https://<IP address>/api/fmc_config/v1/domain/<domain>/policy/accesspolicies/<ACP>/operational/hitcounts?filter=%22deviceId:<device id>%22&expanded=true&limit=500?filter=%22deviceId:<device id>%22&expanded=true&limit=500
2019/11/04-16:09:27 - ERROR:fmc.py:254 - json_response -->  {'error': {'category': 'VALIDATION', 'messages': [{'description': 'The limit in query is not a number.', 'code': 'invalidLimit', 'location': 'limit'}], 'severity': 'ERROR'}}
2019/11/04-16:09:27 - INFO:fmc.py:154 - Auto deploy changes set to False.  Use the Deploy button in FMC to push changes to FTDs.

[daxm]

Yeah. I corrected my last message.

It is far better to grasp the Universe as it really is than to persist in delusion, however satisfying and reassuring. — Carl Sagan

On Nov 4, 2019, at 09:16, Andy Taylor notifications@github.com wrote:

 Same problem when using name when using:

acprule1 = fmcapi.AccessRules(fmc=fmc, acp_name=device1.accessPolicy['name'])

Log file: 2019/11/04-15:53:36 - INFO:apiclasstemplate.py:135 - GET success. Object with name: "" and id: "" fetched from FMC. 2019/11/04-15:53:36 - INFO:apiclasstemplate.py:135 - GET success. Object with name: "" and id: "" fetched from FMC. 2019/11/04-16:09:27 - INFO:apiclasstemplate.py:135 - GET success. Object with name: "" and id: "" fetched from FMC. 2019/11/04-16:09:27 - WARNING:fmc.py:245 - Either:

1. Payload too large. FMC can only handle a payload of 2048000 bytes. 2.The payload contains an unprocessable or unreadable entity such as a invalid attribut name or incorrect JSON syntax 2019/11/04-16:09:27 - ERROR:fmc.py:253 - Error in POST operation --> 422 Client Error: Unprocessable Entity for url: https:///api/fmc_config/v1/domain//policy/accesspolicies//operational/hitcounts?filter=%22deviceId:%22&expanded=true&limit=500?filter=%22deviceId:%22&expanded=true&limit=500 2019/11/04-16:09:27 - ERROR:fmc.py:254 - json_response --> {'error': {'category': 'VALIDATION', 'messages': [{'description': 'The limit in query is not a number.', 'code': 'invalidLimit', 'location': 'limit'}], 'severity': 'ERROR'}} 2019/11/04-16:09:27 - INFO:fmc.py:154 - Auto deploy changes set to False. Use the Deploy button in FMC to push changes to FTDs.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[NetDevAutomate]

When using acprule1 = fmcapi.AccessRules(fmc=fmc, acp_name=device1.accessPolicy["name"])

The log shows the FTD device and ACP rule IDs, using acp_id=device1.accessPolicy["id"]) only the FTD device ID is found.

Neither option get's as far as showing HitCount logs.

ERROR:fmc.py:254 - json_response -->     {'error': {'category': 'VALIDATION', 'messages': [{'description': 'The limit in query is not a number.', 'code': 'invalidLimit', 'location': 'limit'}], 'severity': 'ERROR'}}

However the limit does appear to be a number:

ERROR:fmc.py:253 - Error in POST operation --> 422 Client Error: Unprocessable Entity for url: https://<snip>&expanded=true&limit=500

[daxm]

I'm going to have to look into this some more (obviously).  I know I can get it to work if I already know the "correct" ID that HitCounts is looking for.  That said, I don't know how to get that ID without first pulling all the HitCounts.  I need to find someone who knows what is going on here.

On 11/4/19 10:31 AM, Andy Taylor wrote:

When using acprule1 = fmcapi.AccessRules(fmc=fmc, acp_name=device1.accessPolicy["name"])

The log shows the FTD device and ACP rule IDs, using acp_id=device1.accessPolicy["id"]) only the FTD device ID is found.

Neither option get's as far as showing HitCount logs.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/daxm/fmcapi/issues/39?email_source=notifications&email_token=AAZOMZ2XXPFOCYEVEOYRNZLQSBL6VA5CNFSM4IKVWBFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDACEZQ#issuecomment-549462630, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZOMZ6Z2KZVRWXSBAW57QTQSBL6VANCNFSM4IKVWBFA.

[NetDevAutomate]

Thank you so much, you have been incredibly helpful as always! Happy to perform any testing as where I am working we have large policies so a good test case.

[daxm]

Thank you.  That said, you aren't off the hook.  Maybe you can help me figure out what ID the HitCount is using when it references the AccessRule.  :-)

On 11/5/19 9:27 AM, Andy Taylor wrote:

Thank you so much, you have been incredibly helpful as always!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/daxm/fmcapi/issues/39?email_source=notifications&email_token=AAZOMZ6WSVE2VYAPEAWFS73QSGNGTA5CNFSM4IKVWBFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDDJ5VY#issuecomment-549887703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZOMZZY3E63RCLBKRNE5M3QSGNGTANCNFSM4IKVWBFA.

[NetDevAutomate]

I knocked up some rough python and used:

api_path = f"/api/fmc_config/v1/domain/{domain}/policy/accesspolicies/{acp}/operational/hitcounts?filter=%22deviceId%3A{deviceID}%3BfetchZeroHitCount%3A{ZeroCount}%22&expanded={expanded}&limit={LIMIT}"

acp = ACP ID

The code isn't pretty but did give me hit counts for each ACP entry.

When I set ZeroCount to true oddly enough got nothing, setting it to false I got everything including ZeroCount which is actually the only thing I was interested in.

[MysticRyuujin]

So I just tried to use this, and it works...but there are a few things wrong with it (and one thing wrong with the actual API itself...)

First what's wrong with the API itself? Using version 6.6.1 of the FMC, the hitcounts endpoint will not return more than 1,000 objects, even if the limit is set to be greater than 1,000. Additionally, it does not page. It will always say that the limit was set to 1,000 in the reply and will only return 1,000 objects. Basically if you send limit=2000, the response link to self will say limit=1000 and the paging property will say there's only 1 page.

Now, back to this module...

There are 2 things not working:

1. There is no offset or index number available. This is important because of the issue above, if I get back 1,000 objects, there's a very good chance that there are more objects that I need to go get, but I can't because I can't set an offset/index. The module doesn't know there are more objects to get because the paging is broken :smile: so there needs to be a way to set an offset in the module...

2. You cannot get an HA pair device using its name. I don't know if this is a bigger problem than just the hitcounts endpoint, but using the device_name only works for non-HA pairs. You must use the deviceId of the HA pair as the containerId

[Niltak]

Offset was added in #127