module_utils/pfsense.py:PFSenseModule.php JSONDecodeError: Expecting value: line 1 column 1 (char 0)

NeonixRIT commented 7 months ago

Using openstack and ansible 2.10.8 to deploy and configure some pfsense 2.6.0 instances. It seems self.module.run_command('/usr/local/bin/php', data=cmd) has .. added to the beginning of stdout causing JSONDecoder to throw an error when trying to parse the returned json string from php. I fixed this by changing the php function to the following.

    def php(self, command):
        """ Run a command in php and return the output """
        cmd = '<?php\n'
        cmd += command
        cmd += '\n?>\n'
        (dummy, stdout, stderr) = self.module.run_command('/usr/local/bin/php', data=cmd)
        start_of_json = stdout.index('{')
        end_of_json = stdout.rindex('}') + 1
        # TODO: check stderr for errors
        return json.loads(stdout[start_of_json:end_of_json])

opoplawski commented 7 months ago

A reproducer would be nice, but I'll also note that anible 2.10 went EOL on 23 May 2022. How do you use openstack to deploy pfsense?

NeonixRIT commented 7 months ago

I have an UbuntuJammy2204 VM setup on Openstack with internet access, running python 3.10.12 with ansible installed. I also have the python-openstackclient python module installed via pip. To deploy the pfsense VM I use this task.

In an example environment I have two networks local_1: 192.168.1.0/24 remote_1: 10.0.1.0/24 each having their own ID as defined in openstack.

- name: "Create pfSense instance - {{ router.key }}_pfsense"
  host: localhost
  command: >
    openstack server create
    --flavor {{ router.value.flavor }}
    --image {{ router.value.image }}
    --boot-from-volume {{ router.value.image_size }}
    --nic net-id={{ local_1.net_id }},v4-fixed-ip="192.168.1.1"
    --nic net-id={{ remote_1.net_id }},v4-fixed-ip="10.0.1.1"
    --key-name {{ ansible_control_key_name }}
    {{ router.key }}_pfsense
  ignore_errors: yes

ansible_control_key_name is the name of ssh public key stored in openstack of the UbuntuJammy2204 VM I am running the ansible on to deploy the VMs from. the router var is in this structure:

      team_1_main:
        image: PFsense-2.6.0-Cloudinit
        image_size: 40
        flavor: medium
        connections:
          local_1:
            address: 192.168.1.1
            control: 192.168.1.250
          remote_1:
            address: 10.0.1.254

I then create an openstack port from the UbuntuJammy2204 VM to local_1 so I have access to one of the router's interfaces. This creates a new NIC interface on UbuntuJammy2204 VM, assigning it the control address on the local_1 network.

Then with a very bare test config like:

pfsenses:
  team_1_main_pfsense: {
    interfaces: {
      WAN:                { ip: 10.0.1.254/24 },
      LAN:                 { ip: 192.168.1.1/24 }
    }
  }

rules:
  options: { log: yes }

  ALLOW_ALL:
    allow_all:              { src: any,             dst: any,                 protocol: any,      action: pass }

hosts_aliases:
  internet:             { ip: 0.0.0.0/0 }

ports_aliases:
  admin_ports:          { port: 22 80 443 }

running the examples/lookup/setup_all_rules.yaml playbook to setup firewall rules is where the fatal error occurs. Note, i am not using connection: paramiko. Error still occurs if I use it though.

my inventory.ini being:

[pfsense]
team_1_main_pfsense ansible_ssh_host=192.168.1.1

[pfsense:vars]
ansible_ssh_user=admin
ansible_ssh_password=pfsense
ansible_ssh_port=22
ansible_ssh_common_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
ansible_python_interpreter=/usr/local/bin/python3.8

and ansible.cfg being:

[defaults]

inventory = ./inventory.ini
collections_paths = ./collections/

The php that is run that causes the error is:

<?php
require_once("/etc/inc/interfaces.inc");
$portlist = get_interface_list();
$lagglist = get_lagg_interface_list();
$portlist = array_merge($portlist, $lagglist);
foreach ($lagglist as $laggif => $lagg) {    
    $laggmembers = explode(',', $lagg['members']);
    foreach ($laggmembers as $lagm)        
        if (isset($portlist[$lagm])) unset($portlist[$lagm]);
}
$list = array();
foreach ($portlist as $ifn => $ifinfo) {  
    $list[$ifn] = $ifn . " (\" . $ifinfo[\"mac\"] . \")";
    $iface = convert_real_interface_to_friendly_interface_name($ifn);
    if (isset($iface) && strlen($iface) > 0) $list[$ifn] .= " - $iface";
}
echo json_encode($list);
?>

stdout is

..{\"vtnet0\":\"vtnet0 (fa:16:3e:d5:fb:fd) - wan\",\"vtnet1\":\"vtnet1 (fa:16:3e:cd:52:c2) - lan\"}

stderr is (sorry for formatting)

Traceback (most recent call last):\r\n  File \"/root/.ansible/tmp/ansible-tmp-1707320150.650548-145558-145929973344656/AnsiballZ_pfsense_aggregate.py\", line 102, in <module>\r\n    _ansiballz_main()\r\n  File \"/root/.ansible/tmp/ansible-tmp-1707320150.650548-145558-145929973344656/AnsiballZ_pfsense_aggregate.py\", line 94, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/root/.ansible/tmp/ansible-tmp-1707320150.650548-145558-145929973344656/AnsiballZ_pfsense_aggregate.py\", line 40, in invoke_module\r\n    runpy.run_module(mod_name='ansible_collections.pfsensible.core.plugins.modules.pfsense_aggregate', init_globals=None, run_name='__main__', alter_sys=True)\r\n  File \"/usr/local/lib/python3.8/runpy.py\", line 207, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/local/lib/python3.8/runpy.py\", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File \"/usr/local/lib/python3.8/runpy.py\", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_pfsensible.core.pfsense_aggregate_payload_tip17utb/ansible_pfsensible.core.pfsense_aggregate_payload.zip/ansible_collections/pfsensible/core/plugins/modules/pfsense_aggregate.py\", line 1143, in <module>\r\n  File \"/tmp/ansible_pfsensible.core.pfsense_aggregate_payload_tip17utb/ansible_pfsensible.core.pfsense_aggregate_payload.zip/ansible_collections/pfsensible/core/plugins/modules/pfsense_aggregate.py\", line 1128, in main\r\n  File \"/tmp/ansible_pfsensible.core.pfsense_aggregate_payload_tip17utb/ansible_pfsensible.core.pfsense_aggregate_payload.zip/ansible_collections/pfsensible/core/plugins/modules/pfsense_aggregate.py\", line 644, in __init__\r\n  File \"/tmp/ansible_pfsensible.core.pfsense_aggregate_payload_tip17utb/ansible_pfsensible.core.pfsense_aggregate_payload.zip/ansible_collections/pfsensible/core/plugins/module_utils/vlan.py\", line 73, in __init__\r\n  File \"/tmp/ansible_pfsensible.core.pfsense_aggregate_payload_tip17utb/ansible_pfsensible.core.pfsense_aggregate_payload.zip/ansible_collections/pfsensible/core/plugins/module_utils/pfsense.py\", line 659, in php\r\n  File \"/usr/local/lib/python3.8/json/__init__.py\", line 357, in loads\r\n    return _default_decoder.decode(s)\r\n  File \"/usr/local/lib/python3.8/json/decoder.py\", line 337, in decode\r\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n  File \"/usr/local/lib/python3.8/json/decoder.py\", line 355, in raw_decode\r\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n

NeonixRIT commented 7 months ago

here is php --info output on the pfsense 2.6.0 vm. Let me know if there is any other information that could help.

opoplawski commented 7 months ago

I think there has to be something strange about your setup - which is unusual and making use of old versions of ansible and pfsense. I can't reproduce it with pfSense 2.6.0. I'd be open to some validation of the output returned, but not all valid JSON starts and ends with braces, so your proposed solution doesn't look correct to me.

NeonixRIT commented 7 months ago

That's fair. My initial solution breaks other modules that use the php function where the returned json doesn't start and end with {} like pfsense_interface. Perhaps a custom JSONDecoder or error handling that increments the start of stdout until line 1 column 1 (char 0) is valid, or stripping invalid characters from the start and end would be better solutions.

pfsensible / core

module_utils/pfsense.py:PFSenseModule.php JSONDecodeError: Expecting value: line 1 column 1 (char 0) #118