Factorial Duplication - Access Rules via post

Describe the bug While using the same fmc object in a loop and posting a set new access rule, it will post every rule before it.

If I am looping 6 new rules, it will post 1 - 12 -123 -1234 -12345 - 123456 for a total of 21 rules created.

To Reproduce Loop creation of access rules while using a with statement of a fmc object.

Expected behavior Only posting 6 rules when supplied 6 rules.

import fmcapi

class Connection(fmcapi.FMC):
    def __init__(
        self,
        host="fmc01",
        username="admin",
        password="Admin123",
        domain='test',
        autodeploy=False,
        file_logging=None,
        logging_level="INFO",
        debug=False,
        limit=1000,
        timeout=15):
        super().__init__(
            host, username, password,
            domain, autodeploy,
            file_logging, logging_level,
            debug, limit, timeout)

    def rule_add(self, rule_list):
        '''
        '''
        if not isinstance(rule_list, list):
            rule_list = [rule_list]

        rule_results = []
        for rule in rule_list:
            rule_details = fmcapi.AccessRules(
                fmc=self, acp_name=rule.access_policy)

          # prep rules

            rule_results.append(rule_details.post())
            print('---')

        return rule_results

Python environment

asttokens==2.0.5
autopep8==1.6.0
backcall==0.2.0
bcrypt==3.2.2
Brotli==1.0.9
certifi==2022.6.15
cffi==1.15.0
charset-normalizer==2.1.0
click==8.1.3
colorama==0.4.5
cryptography==37.0.2
dash==2.5.1
dash-core-components==2.0.0
dash-cytoscape==0.3.0
dash-html-components==2.0.0
dash-table==5.0.0
DateTime==4.4
decorator==5.1.1
diffios==0.0.9
executing==0.8.3
flake8==4.0.1
Flask==2.1.2
Flask-Compress==1.12
fmcapi==20220914.0
future==0.18.2
idna==3.3
ipaddress==1.0.23
ipython==8.4.0
itsdangerous==2.1.2
jedi==0.18.1
Jinja2==3.1.2
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mccabe==0.6.1
netmiko==4.1.0
networkx==2.8.4
ntc-templates==3.0.0
paramiko==2.11.0
parso==0.8.3
passlib==1.7.4
pickleshare==0.7.5
plotly==5.9.0
prompt-toolkit==3.0.30
pure-eval==0.2.2
pycodestyle==2.8.0
pycparser==2.21
pycryptodome==3.15.0
pyflakes==2.4.0
Pygments==2.12.0
PyNaCl==1.5.0
pyserial==3.5
pytz==2022.1
PyYAML==6.0
requests==2.28.1
scp==0.14.4
six==1.16.0
stack-data==0.3.0
tenacity==8.0.1
textfsm==1.1.2
toml==0.10.2
traitlets==5.3.0
urllib3==1.26.9
wcwidth==0.2.5
Werkzeug==2.1.2
zope.interface==5.4.0

Additional context Add any other context about the problem here.

File running the module:

rule_list = [
    {'ports': 'HTTPS', 'network': 'obj1'},
    {'ports': 'obj_tcp_5100', 'network': 'obj2'},
    {'ports': 'obj_tcp_1521', 'network': 'obj3'},
    {'ports': 'obj_tcp_1525', 'network': 'obj4'},
    {'ports': 'obj_tcp_1529', 'network': 'obj5'},
    {'ports': 'obj_tcp_1530', 'network': 'obj6'}]

with Connection(
    username=fmc_user,
    password=fmc_pwd,
    timeout=30) as test:

    count = 1
    for rule in rule_list:
        test.rule_add(RuleTemplate(
            f'test_rule_soa_{count}',
            'automation_testing',
            source_zones='test1',
            source_networks='target_obj',
            destination_networks=rule['network'],
            destination_ports=rule['ports'],
            new_comments='test_rule_api'))
        count += 1

This appears to be an incomplete example of the problem. What is RuleTemplate?

Sorry I was trying to reduce the complexity since the code works outside using the same fmc object.

class Rule(dict):
    def __init__(
        self, name, access_policy,
        action=None, enabled=None, sendEventsToFMC=None,
        logFiles=None, logBegin=None, logEnd=None,
        intrusion_policy=None, variable_set=None,
        vlan_tags=None, urls_info=None,
        source_zones=None, destination_zones=None,
        source_ports=None, destination_ports=None,
        source_networks=None, destination_networks=None,
        remove_source_zones=None, remove_destination_zones=None,
        remove_source_ports=None, remove_destination_ports=None,
        remove_source_networks=None, remove_destination_networks=None,
        new_comments=None, id=None):
        self.name = name
        self.access_policy = access_policy
        self.action = action
        self.enabled = enabled
        self.sendEventsToFMC = sendEventsToFMC
        self.logFiles = logFiles
        self.logBegin = logBegin
        self.logEnd = logEnd
        self.intrusion_policy = intrusion_policy
        self.variable_set = variable_set
        self.vlan_tags = vlan_tags
        self.urls_info = urls_info
        self.source_zones = source_zones
        self.destination_zones = destination_zones
        self.source_ports = source_ports
        self.destination_ports = destination_ports
        self.source_networks = source_networks
        self.destination_networks = destination_networks
        self.remove_source_zones = remove_source_zones
        self.remove_destination_zones = remove_destination_zones
        self.remove_source_ports = remove_source_ports
        self.remove_destination_ports = remove_destination_ports
        self.remove_source_networks = remove_source_networks
        self.remove_destination_networks = remove_destination_networks
        self.new_comments = new_comments
        self.id = id

    def info(self):
        '''
        Returns a dictionary of the class.
        '''
        return self.__dict__

class RuleTemplate(Rule):
    def __init__(
        self, name, access_policy,
        action='ALLOW', enabled=False, sendEventsToFMC=True,
        logFiles=False, logBegin=False, logEnd=True,
        intrusion_policy='t_intrusion_dc1', variable_set='t_variableset',
        **kwargs):

        super().__init__(
            name, access_policy,
            action=action, enabled=enabled, sendEventsToFMC=sendEventsToFMC,
            logFiles=logFiles, logBegin=logBegin, logEnd=logEnd,
            intrusion_policy=intrusion_policy, variable_set=variable_set,
            **kwargs)

example of a work around:

rule_list = [
    {'ports': 'HTTPS', 'network': 'obj1'},
    {'ports': 'obj_tcp_5100', 'network': 'obj2'},
    {'ports': 'obj_tcp_1521', 'network': 'obj3'},
    {'ports': 'obj_tcp_1525', 'network': 'obj4'},
    {'ports': 'obj_tcp_1529', 'network': 'obj5'},
    {'ports': 'obj_tcp_1530', 'network': 'obj6'}]

count = 1
for rule in rule_list:
    with Connection(
        username=fmc_user,
        password=fmc_pwd,
        timeout=30) as test:
            test.rule_add(RuleTemplate(
                f'test_rule_soa_{count}',
                'automation_testing',
                source_zones='test1',
                source_networks='target_obj',
                destination_networks=rule['network'],
                destination_ports=rule['ports'],
                new_comments='test_rule_api'))
            count += 1

I do not see the problem you are reporting in either the FMCAPI code that was released this week or the previous version. Here are my FMCAPI logs:

2022-09-16 08:05:38,999  257:apiclasstemplate          INFO     POST success. Object with name: "marksull-dup" and id: "0050568E-B2F8-0ed3-0000-176094164300" created in FMC.
2022-09-16 08:06:01,204  170:apiclasstemplate          INFO     GET success. Object with name: "marksull-dup" and id: "0050568E-B2F8-0ed3-0000-176094164300" fetched from FMC.
2022-09-16 08:06:10,113  257:apiclasstemplate          INFO     POST success. Object with name: "test_rule_soa_1" and id: "0050568E-B2F8-0ed3-0000-000268849882" created in FMC.
---
2022-09-16 08:06:20,185  170:apiclasstemplate          INFO     GET success. Object with name: "marksull-dup" and id: "0050568E-B2F8-0ed3-0000-176094164300" fetched from FMC.
2022-09-16 08:06:21,699  257:apiclasstemplate          INFO     POST success. Object with name: "test_rule_soa_2" and id: "0050568E-B2F8-0ed3-0000-000268849883" created in FMC.
---
2022-09-16 08:06:25,587  170:apiclasstemplate          INFO     GET success. Object with name: "marksull-dup" and id: "0050568E-B2F8-0ed3-0000-176094164300" fetched from FMC.
2022-09-16 08:06:27,175  257:apiclasstemplate          INFO     POST success. Object with name: "test_rule_soa_3" and id: "0050568E-B2F8-0ed3-0000-000268849884" created in FMC.
---
2022-09-16 08:06:30,198  170:apiclasstemplate          INFO     GET success. Object with name: "marksull-dup" and id: "0050568E-B2F8-0ed3-0000-176094164300" fetched from FMC.
2022-09-16 08:06:31,679  257:apiclasstemplate          INFO     POST success. Object with name: "test_rule_soa_4" and id: "0050568E-B2F8-0ed3-0000-000268849885" created in FMC.
---
2022-09-16 08:06:36,085  170:apiclasstemplate          INFO     GET success. Object with name: "marksull-dup" and id: "0050568E-B2F8-0ed3-0000-176094164300" fetched from FMC.
2022-09-16 08:06:37,578  257:apiclasstemplate          INFO     POST success. Object with name: "test_rule_soa_5" and id: "0050568E-B2F8-0ed3-0000-000268849886" created in FMC.
---
2022-09-16 08:06:40,808  170:apiclasstemplate          INFO     GET success. Object with name: "marksull-dup" and id: "0050568E-B2F8-0ed3-0000-176094164300" fetched from FMC.
2022-09-16 08:06:42,307  257:apiclasstemplate          INFO     POST success. Object with name: "test_rule_soa_6" and id: "0050568E-B2F8-0ed3-0000-000268849887" created in FMC.
2022-09-16 08:06:42,308  163:fmc                       INFO     Auto deploy changes set to False.  Use the Deploy button in FMC to push changes to FTDs.
---

Because it was an incomplete example, I simply set the rule name and action the same on each rule, so I don't know what other activities you were performing in the rule_add method.

            rule_details.name = rule.name
            rule_details.action = "ALLOW"

The best way forward is to provide the most minimal example of the problem you are experiencing without any additional complexities, just the minimum raw FMCAPI calls that recreate your problem. Also, include your FMCAPC logs as above.

If I were to guess your issue, it would be a Pass By Reference issue, where you are inadvertently passing around a list between methods, and it is being appended to and reused on successive iterations.

Sorry for not getting back sooner. Emergency at work.

Here is a less complex example:

import fmcapi
from userlist import fmc_user, fmc_pwd

class Connection(fmcapi.FMC):
    def __init__(
        self,
        host="redacted",
        username="admin",
        password="Admin123",
        domain='test-it',
        autodeploy=False,
        file_logging=None,
        logging_level="INFO",
        debug=False,
        limit=1000,
        timeout=15):
        super().__init__(
            host, username, password,
            domain, autodeploy,
            file_logging, logging_level,
            debug, limit, timeout)

    def test_loop(self):
        '''
        '''
        rule_list = [
            {'ports': 'HTTPS', 'network': 'any'},
            {'ports': 'obj_tcp_5100', 'network': 'any-ipv4'},
            {'ports': 'obj_tcp_1521', 'network': 'any-ipv4-test'}]

        count = 0
        for rule in rule_list:
            rule_details = fmcapi.AccessRules(
                fmc=self, acp_name='automation_testing',
                category='Automation', action='ALLOW')

            count += 1
            rule_details.name = f'test_rule_{count}'
            rule_details._enabled = True
            rule_details._sendEventsToFMC = True
            rule_details._logEnd = True
            rule_details.intrusion_policy('set', name='test_intrusion_dc1')
            rule_details.variable_set('set', name='test_variableset')
            rule_details.new_comments('add', value='test_rule_api')
            rule_details.destination_port('add', name=rule['ports'])
            rule_details.source_network('add', name=rule['network'])

            rule_details.post()

with Connection(
    username=fmc_user,
    password=fmc_pwd,
    timeout=30) as test:
    test.test_loop()

INFO:root:Requesting new tokens from https://{hostname redacted}/api/fmc_platform/v1/auth/generatetoken.
INFO:root:Domain name entered not found in FMC, falling back to Global
INFO:root:Domain set to Global/test-it
INFO:root:Building base to URLs.     
INFO:root:Populating vdbVersion, sruVersion, serverVersion, and geoVersion FMC instance variables.
INFO:root:This FMC's version is 7.0.1.1 (build 11)
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-228856506384" fetched from FMC.
INFO:root:GET success. Object with name: "test_intrusion_dc1" and id: "0e3c1b4e-7291-11ec-895d-23b674325230" fetched from FMC.
INFO:root:Intrusion Policy set to "test_intrusion_dc1" for this AccessRules object.
INFO:root:GET success. Object with name: "test_variableset" and id: "7a98350e-5f58-11eb-9a7f-9024cd0a25b7" fetched from FMC.
INFO:root:VariableSet set to "test_variableset" for this AccessRules object.
INFO:root:GET success. Object with name: "HTTPS" and id: "1834bd00-38bb-11e2-86aa-62f0c593a59a" fetched from FMC.
WARNING:root:   GET query for HTTPS is not found.
INFO:root:Adding "HTTPS" to destinationPorts for this AccessRules.
INFO:root:Adding "any" to sourceNetworks for this AccessRules.
INFO:root:POST success. Object with name: "test_rule_1" and id: "3C5731FF-5F3A-0ed3-0000-000268459256" created in FMC.
---
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-228856506384" fetched from FMC.
INFO:root:GET success. Object with name: "test_intrusion_dc1" and id: "0e3c1b4e-7291-11ec-895d-23b674325230" fetched from FMC.
INFO:root:Intrusion Policy set to "test_intrusion_dc1" for this AccessRules object.
INFO:root:GET success. Object with name: "test_variableset" and id: "7a98350e-5f58-11eb-9a7f-9024cd0a25b7" fetched from FMC.
INFO:root:VariableSet set to "test_variableset" for this AccessRules object.
INFO:root:GET success. Object with name: "obj_tcp_5100" and id: "3C5731FF-5F3A-0ed3-0000-008589942277" fetched from FMC.
WARNING:root:   GET query for obj_tcp_5100 is not found.
INFO:root:Adding "obj_tcp_5100" to destinationPorts for this AccessRules.
INFO:root:Adding "any-ipv4" to sourceNetworks for this AccessRules.
INFO:root:POST success. Object with name: "test_rule_1" and id: "3C5731FF-5F3A-0ed3-0000-000268459257" created in FMC.
---
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-228856506384" fetched from FMC.
INFO:root:GET success. Object with name: "test_intrusion_dc1" and id: "0e3c1b4e-7291-11ec-895d-23b674325230" fetched from FMC.
INFO:root:Intrusion Policy set to "test_intrusion_dc1" for this AccessRules object.
INFO:root:GET success. Object with name: "test_variableset" and id: "7a98350e-5f58-11eb-9a7f-9024cd0a25b7" fetched from FMC.
INFO:root:VariableSet set to "test_variableset" for this AccessRules object.
INFO:root:GET success. Object with name: "obj_tcp_1521" and id: "3C5731FF-5F3A-0ed3-0000-008589942450" fetched from FMC.
WARNING:root:   GET query for obj_tcp_1521 is not found.
INFO:root:Adding "obj_tcp_1521" to destinationPorts for this AccessRules.
INFO:root:Adding "any-ipv4-test" to sourceNetworks for this AccessRules.
INFO:root:POST success. Object with name: "test_rule_1" and id: "3C5731FF-5F3A-0ed3-0000-000268459259" created in FMC.
---
INFO:root:Auto deploy changes set to False.  Use the Deploy button in FMC to push changes to FTDs.

To add to the weirdness of this bug, sometimes (mostly when I run it for the first time of the day) it runs this script without the bug. I have restarted my IDE to see if it was some issue with vars getting cached but it doesn't seem to be the case.

You are not making my life easy ;-)

A couple of things:

the example code above was not used for creating the last logs above because there is no "---" in the code separator in the code.
the last example output does not demonstrate any duplication. There are only 3 rules set up, which is exactly as expected so I don't see the problem you are reporting
when I said "the simplest example possible used to recreate the problem", by including a custom category, custom intrusion policy, custom variable set, custom port objects, and custom networks make testing this difficult without me having to set up all those dependencies

I ran your code 4 times with varying complexity (excluding some of the dependencies hoping to recreate the issue in the simplest way possible), but I always would only see 3 records created.

In the example above that you provided, what did the ACP look like on the FMC?

My bad for not removing more complexities. My brain that morning thought I needed those vars to create any rule.

To be clear, my last post has the logs from the above code, but it was edited to take out some redacted vars. I added the dashes and added a print dash line to the code after to make it more readable.

The above logs show that it created 'test_rule_1' 3 times. It doesn't log the output for the other rules created.

Here is my current code with the least amount of complexities for my issue:

import fmcapi
from aa_global import fmc_user, fmc_pwd, fmc_host, fmc_domain

class Connection(fmcapi.FMC):
    def __init__(
        self,
        host="redacted",
        username="admin",
        password="Admin123",
        domain='test-it',
        autodeploy=False,
        file_logging=None,
        logging_level="INFO",
        debug=False,
        limit=1000,
        timeout=15):
        super().__init__(
            host, username, password,
            domain, autodeploy,
            file_logging, logging_level,
            debug, limit, timeout)

    def test_loop(self):
        count = 0
        while count < 3:
            rule_details = fmcapi.AccessRules(
                fmc=self, acp_name='automation_testing', action='ALLOW')

            count += 1
            rule_details.name = f'test_rule_{count}'

            rule_details.post()
            print('---')

with Connection(
    host=fmc_host,
    username=fmc_user,
    password=fmc_pwd,
    domain=fmc_domain,
    timeout=30) as test:
    test.test_loop()

Logging of the Bug

INFO:root:Requesting new tokens from https://prdfmc01/api/fmc_platform/v1/auth/generatetoken.
INFO:root:Domain name entered not found in FMC, falling back to Global
INFO:root:Domain set to {{REDACTED}}
INFO:root:Building base to URLs.
INFO:root:Populating vdbVersion, sruVersion, serverVersion, and geoVersion FMC instance variables.
INFO:root:This FMC's version is 7.0.1.1 (build 11)
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-128856506384" fetched from FMC.
INFO:root:POST success. Object with name: "test_rule_1" and id: "3C5731FF-5F3A-0ed3-0000-000268459283" created in FMC.
---
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-128856506384" fetched from FMC.
INFO:root:POST success. Object with name: "test_rule_1" and id: "3C5731FF-5F3A-0ed3-0000-000268459284" created in FMC.
---
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-128856506384" fetched from FMC.
INFO:root:POST success. Object with name: "test_rule_1" and id: "3C5731FF-5F3A-0ed3-0000-000268459286" created in FMC.
---
INFO:root:Auto deploy changes set to False.  Use the Deploy button in FMC to push changes to FTDs.

Logging when it randomly works:

INFO:root:Requesting new tokens from https://prdfmc01/api/fmc_platform/v1/auth/generatetoken.
INFO:root:Domain name entered not found in FMC, falling back to Global
INFO:root:Building base to URLs.
INFO:root:Populating vdbVersion, sruVersion, serverVersion, and geoVersion FMC instance variables.
INFO:root:This FMC's version is 7.0.1.1 (build 11)
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-128856506384" fetched from FMC.
INFO:root:POST success. Object with name: "test_rule_1" and id: "3C5731FF-5F3A-0ed3-0000-000268459280" created in FMC.
---
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-128856506384" fetched from FMC.
INFO:root:POST success. Object with name: "test_rule_2" and id: "3C5731FF-5F3A-0ed3-0000-000268459281" created in FMC.
---
INFO:root:GET success. Object with name: "automation_testing" and id: "3C5731FF-5F3A-0ed3-0000-128856506384" fetched from FMC.
INFO:root:POST success. Object with name: "test_rule_3" and id: "3C5731FF-5F3A-0ed3-0000-000268459282" created in FMC.
---
INFO:root:Auto deploy changes set to False.  Use the Deploy button in FMC to push changes to FTDs.

Picture of before and after:

Honestly, don't know. I just tested it 4 different times, and it always works fine. We use this code in production many times daily and haven't seen this issue in the wild either.

Thinking aloud... 1) Any chance you can initiate the fmcapi connection with debug=true and capture the logs and attach it as a file here (make sure you removed any potential private information)? 2) What version of FMC are you using? Has there been a recent change to your FMC version, and if so, do you have a previous version to test on to see if the problem follows the FMC or follows the code?

Here you go: fmcapi_logs.txt

FMC v7.0.1.1 I am attempting to get ahold of a test server for v7.0.4 to see if that works. There is a bug report that is similar to what I am experiencing, but it was 'fixed' on v7.0.1 . https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvy03907

I am ok saying this isn't an API issue on fmcapi. Everything seems to be good when it gets handed off to FMC. The only thing that is odd is this bug hasn't happen if I create a new connection object every time I make a new rule.

IMHO CSCvy03907 does not appear related to this issue.

The log definitely highlights an anomaly. Check this out:

I honestly don't know what is going on there.

My next troubleshooting step (I would try this myself, but I cannot recreate the issue) would be to take the URL/JSON Data from that debug log and use postman (or similar) and post the exact same data using a single session key and see if you get the same response (i.e. get back a different rule name). Then repeat the tests with a fresh session key for each post (which should be the equivalent of creating a new FMCAPI connection object each time) and see if this works without issue.

If you see similar results to the above anomaly, then we know this is not FMCAPI package related. If you do not see the same issue, then I am going to further confused ;-)

Confirmed with Postman that this issue is on the FMC side. Thanks for the help with this.

Very unusual problem. If you end up opening a case with TAC and get the bug id, if you could drop it in here so we have reference to it that would be helpful.

marksull / fmcapi

Factorial Duplication - Access Rules via post #131