search

ndejong / pfsense_fauxapi

REST based API interface for pfSense 2.3.x and 2.4.x to facilitate devops
Apache License 2.0
354 stars 61 forks source link
api devops pfsense pfsense-fauxapi restful-api

readme

FauxAPI - v1.4

A REST API interface for pfSense 2.3.x, 2.4.x, 2.5.x to facilitate devops:-

Additionally available are a set of client libraries that hence make programmatic access and management of pfSense hosts for devops tasks feasible.

Important

API Action Summary

Approach

At its core FauxAPI simply reads the core pfSense config.xml file, converts it to JSON and returns to the API caller. Similarly it can take a JSON formatted configuration and write it to the pfSense config.xml and handles the required reload operations. The ability to programmatically interface with a running pfSense host(s) is enormously useful however it should also be obvious that this provides the API user the ability to create configurations that can break your pfSense system.

FauxAPI provides easy backup and restore API interfaces that by default store configuration backups on all configuration write operations thus it is very easy to roll-back even if the API user manages to deploy a "very broken" configuration.

Multiple sanity checks take place to make sure a user provided JSON config will correctly convert into the (slightly quirky) pfSense XML config.xml format and then reload as expected in the same way. However, because it is not a real per-action application-layer interface it is still possible for the API caller to create configuration changes that make no sense and can potentially disrupt your pfSense system - as the package name states, it is a "Faux" API to pfSense filling a gap in functionality with the current pfSense product.

Because FauxAPI is a utility that interfaces with the pfSense config.xml there are some cases where reloading the configuration file is not enough and you may need to "tickle" pfSense a little more to do what you want. This is not common however a good example is getting newly defined network interfaces or VLANs to be recognized. These situations are easily handled by calling the send_event action with the payload interface reload all - see the example included below and refer to a the resolution to Issue #10

NB: As at FauxAPI v1.2 the function_call action has been introduced that now provides the ability to issue function calls directly into pfSense.

Installation

Until the FauxAPI is added to the pfSense FreeBSD-ports tree you will need to install manually from root as shown:-

set fauxapi_base_package_url='https://raw.githubusercontent.com/ndejong/pfsense_fauxapi_packages/master'
set fauxapi_latest=`fetch -qo - ${fauxapi_base_package_url}/LATEST`
fetch ${fauxapi_base_package_url}/${fauxapi_latest}
pkg-static install ${fauxapi_latest}

Installation and de-installation is quite straight forward, further examples can be found in the README.md located here.

Refer to the published package SHA256SUMS

Hint: if not already, consider installing the jq tool on your local machine (not pfSense host) to pipe and manage JSON outputs from FauxAPI - https://stedolan.github.io/jq/

NB: you MUST at least setup your /etc/fauxapi/credentials.ini file on the pfSense host before you continue, see the API Authentication section below.

Client libraries

Python

A Python interface to pfSense was perhaps the most desired end-goal at the onset of the FauxAPI package project. Anyone that has tried to parse the pfSense config.xml files using a Python based library will understand that things don't quite work out as expected or desired.

The Python client-library can be easily installed from PyPi as such

pip3 install pfsense-fauxapi

Package Status: PyPi Build Status

Use of the package should be easy enough as shown

import pprint, sys
from PfsenseFauxapi.PfsenseFauxapi import PfsenseFauxapi
PfsenseFauxapi = PfsenseFauxapi('<host-address>', '<fauxapi-key>', '<fauxapi-secret>')

aliases = PfsenseFauxapi.config_get('aliases')
## perform some kind of manipulation to `aliases` here ##
pprint.pprint(PfsenseFauxapi.config_set(aliases, 'aliases'))

It is recommended to review the Python code examples to observe worked examples with the client library. Of small note is that the Python library supports the ability to get and set single sections of the pfSense system, not just the entire system configuration as with the Bash library.

Python examples

Command Line

As distinct from the Bash library as described below the Python pip also introduces a command-line tool to interact with the API, which makes a wide range of actions possible directly from the command line, for example

fauxapi --host 192.168.1.200 gateway_status | jq .

Bash

The Bash client library makes it possible to add a line with source pfsense-fauxapi.sh to your bash script and then access a pfSense host configuration directly as a JSON string

source pfsense-fauxapi.sh
export fauxapi_auth=$(fauxapi_auth <fauxapi-key> <fauxapi-secret>)

fauxapi_config_get <host-address> | jq .data.config > /tmp/config.json
## perform some kind of manipulation to `/tmp/config.json` here ##
fauxapi_config_set <host-address> /tmp/config.json

It is recommended to review the commented out samples in the provided fauxapi-sample.sh file that cover all possible FauxAPI calls to gain a better idea on usage.

NodeJS/TypeScript

A NodeJS client has been developed by a third party and is available here

PHP

A PHP client has been developed by a third party and is available here

API Authentication

A deliberate design decision to decouple FauxAPI authentication from both the pfSense user authentication and the pfSense config.xml system. This was done to limit the possibility of an accidental API change that removes access to the host. It also seems more prudent to only establish API user(s) manually via the FauxAPI /etc/fauxapi/credentials.ini file - happy to receive feedback about this approach.

The two sample FauxAPI keys (PFFAexample01 and PFFAexample02) and their associated secrets in the sample credentials.sample.ini file are hard-coded to be inoperative, you must create entirely new values before your client scripts will be able to issue commands to FauxAPI.

You can start your own /etc/fauxapi/credentials.ini file by copying the sample file provided in credentials.sample.ini

API authentication itself is performed on a per-call basis with the auth value inserted as an additional fauxapi-auth HTTP request header, it can be calculated as such:-

fauxapi-auth: <apikey>:<timestamp>:<nonce>:<hash>

For example:-
fauxapi-auth: PFFA4797d073:20161119Z144328:833a45d8:9c4f96ab042f5140386178618be1ae40adc68dd9fd6b158fb82c99f3aaa2bb55

Where the <hash> value is calculated like so:-

<hash> = sha256(<apisecret><timestamp><nonce>)

NB: that the timestamp value is internally passed to the PHP strtotime function which can interpret a wide variety of timestamp formats together with a timezone. A nice tidy timestamp format that the strtotime PHP function is able to process can be obtained using bash command date --utc +%Y%m%dZ%H%M%S where the Z date-time seperator hence also specifies the UTC timezone.

This is all handled in the client libraries provided, but as can be seen it is relatively easy to implement even in a Bash shell script.

Getting the API credentials right seems to be a common source of confusion in getting started with FauxAPI because the rules about valid API keys and secret values are pedantic to help make ensure poor choices are not made.

The API key + API secret values that you will need to create in /etc/fauxapi/credentials.ini have the following rules:-

To make things easier consider using the following shell commands to generate valid values:-

apikey_value

echo PFFA`head /dev/urandom | base64 -w0 | tr -d /+= | head -c 20`

apisecret_value

echo `head /dev/urandom | base64 -w0 | tr -d /+= | head -c 60`

NB: Make sure the client side clock is within 60 seconds of the pfSense host clock else the auth token values calculated by the client will not be valid - 60 seconds seems tight, however, provided you are using NTP to look after your system time it's quite unlikely to cause issues - happy to receive feedback about this.

Shout Out: Seeking feedback on the API authentication, many developers seem to stumble here - if you feel something could be improved without compromising security then submit an Issue ticket via Github.

API Authorization

The file /etc/fauxapi/credentials.ini additionally provides a method to restrict the API actions available to the API key using the permit configuration parameter. Permits are comma delimited and may contain * wildcards to match more than one rule as shown in the example below.

[PFFAexample01]
secret = abcdefghijklmnopqrstuvwxyz0123456789abcd
permit = alias_*, config_*, gateway_*, rule_*, send_*, system_*, function_*
comment = example key PFFAexample01 - hardcoded to be inoperative

Debugging

FauxAPI comes with awesome debug logging capability, simply insert __debug=true as a URL request parameter and the response data will contain rich debugging log data about the flow of the request.

If you are looking for more debugging at various points feel free to submit a pull request or lodge an issue describing your requirement and I'll see what can be done to accommodate.

Logging

FauxAPI actions are sent to the system syslog via a call to the PHP syslog() function thus causing all FauxAPI actions to be logged and auditable on a per action (callid) basis which provide the full basis for the call, for example:-

Jul  3 04:37:59 pfSense php-fpm[55897]: {"INFO":"20180703Z043759 :: fauxapi\\v1\\fauxApi::__call","DATA":{"user_action":"alias_update_urltables","callid":"5b3afda73e7c9","client_ip":"192.168.1.5"},"source":"fauxapi"}
Jul  3 04:37:59 pfSense php-fpm[55897]: {"INFO":"20180703Z043759 :: valid auth for call","DATA":{"apikey":"PFFAdevtrash","callid":"5b3afda73e7c9","client_ip":"192.168.1.5"},"source":"fauxapi"}

Enabling debugging yields considerably more logging data to assist with tracking down issues if you encounter them - you may review the logs via the pfSense GUI as usual unser Status->System Logs->General or via the console using the clog tool

$ clog /var/log/system.log | grep fauxapi

Configuration Backups

All configuration edits through FauxAPI create configuration backups in the same way as pfSense does with the webapp GUI.

These backups are available in the same way as edits through the pfSense GUI and are thus able to be reviewed and diff'd in the same way under Diagnostics->Backup & Restore->Config History.

Changes made through the FauxAPI carry configuration change descriptions that name the unique callid which can then be tied to logs if required for full usage audit and change tracking.

FauxAPI functions that cause write operations to the system config config.xml return reference to a backup file of the configuration immediately previous to the change.

API REST Actions

The following REST based API actions are provided, example cURL call request examples are provided for each. The API user is perhaps more likely to interface with the client libraries as documented above rather than directly with these REST end-points.

The framework around the FauxAPI has been put together with the idea of being able to easily add more actions at a later time, if you have ideas for actions that might be useful be sure to get in contact.

NB: the cURL requests below use the '--insecure' switch because many pfSense deployments do not deploy certificate chain signed SSL certificates. A reasonable improvement in this regard might be to implement certificate pinning at the client side to hence remove scope for man-in-middle concerns.

alias_update_urltables

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=alias_update_urltables"

Example Response

{
  "callid": "598ec756b4d09",
  "action": "alias_update_urltables",
  "message": "ok",
  "data": {
    "updates": {
      "bruteforceblocker": {
        "url": "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bruteforceblocker.ipset",
        "status": [
          "no changes."
        ]
      }
    }
  }
}

config_backup

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=config_backup"

Example Response

{
  "callid": "583012fea254f",
  "action": "config_backup",
  "message": "ok",
  "data": {
    "backup_config_file": "/cf/conf/backup/config-1479545598.xml"
  }
}

config_backup_list

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=config_backup_list"

Example Response

{
  "callid": "583065cb670db",
  "action": "config_backup_list",
  "message": "ok",
  "data": {
    "backup_files": [
      {
        "filename": "/cf/conf/backup/config-1479545598.xml",
        "timestamp": "20161119Z144635",
        "description": "fauxapi-PFFA4797d073@192.168.10.10: update via fauxapi for callid: 583012fea254f",
        "version": "15.5",
        "filesize": 18535
      },
      ....

config_get

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=config_get"

Example Response

{
    "callid": "583012fe39f79",
    "action": "config_get",
    "message": "ok",
    "data": {
      "config_file": "/cf/conf/config.xml",
      "config": {
        "version": "15.5",
        "staticroutes": "",
        "snmpd": {
          "syscontact": "",
          "rocommunity": "public",
          "syslocation": ""
        },
        "shaper": "",
        "installedpackages": {
          "pfblockerngsouthamerica": {
            "config": [
             ....

Hint: use jq to parse the response JSON and obtain the config only, as such:-

cat /tmp/faux-config-get-output-from-curl.json | jq .data.config > /tmp/config.json

config_patch

Example Request

cat > /tmp/config_patch.json <<EOF
{
  "system": {
    "dnsserver": [
      "8.8.8.8",
      "8.8.4.4"
    ],
    "hostname": "newhostname"
  }
}
EOF

curl \
    -X POST \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    --header "Content-Type: application/json" \
    --data @/tmp/config_patch.json \
    "https://<host-address>/fauxapi/v1/?action=config_patch"

Example Response

{
  "callid": "5b3b506f72670",
  "action": "config_patch",
  "message": "ok",
  "data": {
    "do_backup": true,
    "do_reload": true,
    "previous_config_file": "/cf/conf/backup/config-1530613871.xml"
  }

config_reload

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=config_reload"

Example Response

{
  "callid": "5831226e18326",
  "action": "config_reload",
  "message": "ok"
}

config_restore

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=config_restore&config_file=/cf/conf/backup/config-1479545598.xml"

Example Response

{
  "callid": "583126192a789",
  "action": "config_restore",
  "message": "ok",
  "data": {
    "config_file": "/cf/conf/backup/config-1479545598.xml"
  }
}

config_set

Example Request

curl \
    -X POST \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    --header "Content-Type: application/json" \
    --data @/tmp/config.json \
    "https://<host-address>/fauxapi/v1/?action=config_set"

Example Response

{
  "callid": "5b3b50e8b1bc6",
  "action": "config_set",
  "message": "ok",
  "data": {
    "do_backup": true,
    "do_reload": true,
    "previous_config_file": "/cf/conf/backup/config-1530613992.xml"
  }
}

function_call

Example Request

curl \
    -X POST \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    --header "Content-Type: application/json" \
    --data "{\"function\": \"get_services\"}" \
    "https://<host-address>/fauxapi/v1/?action=function_call"

Example Response

{
  "callid": "59a29e5017905",
  "action": "function_call",
  "message": "ok",
  "data": {
    "return": [
      {
        "name": "unbound",
        "description": "DNS Resolver"
      },
      {
        "name": "ntpd",
        "description": "NTP clock sync"
      },
      ....

gateway_status

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=gateway_status"

Example Response

{
  "callid": "598ecf3e7011e",
  "action": "gateway_status",
  "message": "ok",
  "data": {
    "gateway_status": {
      "10.22.33.1": {
        "monitorip": "8.8.8.8",
        "srcip": "10.22.33.100",
        "name": "GW_WAN",
        "delay": "4.415ms",
        "stddev": "3.239ms",
        "loss": "0.0%",
        "status": "none"
      }
    }
  }
}

interface_stats

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=interface_stats&interface=em0"

Example Response

{
  "callid": "5b3a5bce65d01",
  "action": "interface_stats",
  "message": "ok",
  "data": {
    "stats": {
      "inpkts": 267017,
      "inbytes": 21133408,
      "outpkts": 205860,
      "outbytes": 8923046,
      "inerrs": 0,
      "outerrs": 0,
      "collisions": 0,
      "inmcasts": 61618,
      "outmcasts": 73,
      "unsuppproto": 0,
      "mtu": 1500
    }
  }
}

rule_get

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=rule_get&rule_number=5"

Example Response

{
  "callid": "583c279b56958",
  "action": "rule_get",
  "message": "ok",
  "data": {
    "rules": [
      {
        "number": 5,
        "rule": "anchor \"openvpn/*\" all",
        "evaluations": "14134",
        "packets": "0",
        "bytes": "0",
        "states": "0",
        "inserted": "21188",
        "statecreations": "0"
      }
    ]
  }
}

send_event

Example Request

curl \
    -X POST \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    --header "Content-Type: application/json" \
    --data "[\"interface reload all\"]" \
    "https://<host-address>/fauxapi/v1/?action=send_event"

Example Response

{
  "callid": "58312bb3398bc",
  "action": "send_event",
  "message": "ok"
}

system_reboot

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=system_reboot"

Example Response

{
  "callid": "58312bb3487ac",
  "action": "system_reboot",
  "message": "ok"
}

system_stats

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=system_stats"

Example Response

{
  "callid": "5b3b511655589",
  "action": "system_stats",
  "message": "ok",
  "data": {
    "stats": {
      "cpu": "20770421|20494981",
      "mem": "20",
      "uptime": "1 Day 21 Hours 25 Minutes 48 Seconds",
      "pfstate": "62/98000",
      "pfstatepercent": "0",
      "temp": "",
      "datetime": "20180703Z103358",
      "cpufreq": "",
      "load_average": [
        "0.01",
        "0.04",
        "0.01"
      ],
      "mbuf": "1016/61600",
      "mbufpercent": "2"
    }
  }
}

system_info

Example Request

curl \
    -X GET \
    --silent \
    --insecure \
    --header "fauxapi-auth: <auth-value>" \
    "https://<host-address>/fauxapi/v1/?action=system_info"

Example Response

{
    "callid": "5e1d8ceb8ff47",
    "action": "system_info",
    "message": "ok",
    "data": {
        "info": {
            "sys": {
                "platform": {
                    "name": "VMware",
                    "descr": "VMware Virtual Machine"
                },
                "serial_no": "",
                "device_id": "719e8c91c2c43b820400"
            },
            "pfsense_version": {
                "product_version_string": "2.4.5-DEVELOPMENT",
                "product_version": "2.4.5-DEVELOPMENT",
                "product_version_patch": "0"
            },
            "pfsense_remote_version": {
                "version": "2.4.5.a.20200112.1821",
                "installed_version": "2.4.5.a.20191218.2354",
                "pkg_version_compare": "<"
            },
            "os_verison": "FreeBSD 11.3-STABLE",
            "cpu_type": {
                "cpu_model": "Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz",
                "cpu_count": "4",
                "logic_cpu_count": "4 package(s)",
                "cpu_freq": ""
            },
            "kernel_pti_status": "enabled",
            "mds_mitigation": "inactive",
            "bios": {
                "vendor": "Phoenix Technologies LTD",
                "version": "6.00",
                "date": "07/29/2019"
            }
        }
    }
}

Versions and Testing

The FauxAPI has been developed against the following pfSense versions

FauxAPI has not been tested against 2.3.0 or 2.3.1. Additionally, it is apparent the pfSense packaging technique changed significantly prior to 2.3.x so it is unlikely it will be backported to anything prior to 2.3.0.

Testing is reasonable but does not achieve 100% code coverage within the FauxAPI codebase. Two client side test scripts (1x Bash, 1x Python) that both demonstrate and test all possible server side actions are provided. Under the hood FauxAPI, performs real-time sanity checks and tests to make sure the user supplied configurations will save, load and reload as expected.

Shout Out: Anyone that happens to know of any test harness or test code for pfSense please get in touch - I'd very much prefer to integrate with existing pfSense test infrastructure if it already exists.

Releases

v1.0 - 2016-11-20

v1.1 - 2017-08-12

v1.2 - 2017-08-27

v1.3 - 2018-07-02

v1.4 - 2020-05-31

FauxAPI License

Copyright 2016-2020 Nicholas de Jong

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.