The intention for this project/rpm is to provide a common tool to extract tmos applications. Taking a bigip.conf and extracting all it's configuration items to a form that can easily be searched and different functions performed, all with the intent of migrating the application to something like AS3.
Latest performance stats can be found here: testingStats
check out the: CHANGE LOG for details of the different releases
look into using a main stream parser like
HashiCorp HCL has a very similar laguage structure
Deeper app parsing
Object one-line output
tmsh list net self one-line--one-line, remove line returns from each array item, then combine the array items with line returns to provide a single string like beforeCommand line interface
Exploring the idea of fully jsonifying the entire config
a "sanitize" function to remove sensitive information and change IPs/names so configs can be shared as part of the process
an option to include the default profiles
Identify advanced configurations
The initial parsing of the files seems to be the heaviest operation so far:
The current performance of a 6MB config file with almost 300vs, 223k lines, and over 13k tmos objects fully processed in about 20 seconds seems pretty good for this stage. I think it would be acceptable for full processing to take up to a couple of minutes (with bigger configurations), given appropriate feedback with status/progress bars. This would provide the user with information that a long running process is still "processing", without just sitting blank.
Furthermore, the stats being included for the different files, sizes, and processing times should provide adequate information to quickly identify what will be the next bottleneck (whether it's file size, object count, or if we can rely on VS count...?)
If/when we get to the point that we do need to increase performance, I have the following thoughts:
The unpacking of UCS and kqviews
loading of file content -> streaming
parsing the stream
Things to keep in mind as we ponder performance numbers:
My point here is that we have a general idea of how big the configs can get and that we are not looking to support 100k Virtual servers and 1m+ objects, or file sizes pushing several hundred Mb or even Gb. This tells me that we should be able to accomplish the goals envisioned with the project without any red flags or big unknowns that could push the architecture or development path in a completely different direction (custom C/rust parser engine kinda stuff)
The whole thing is written in TypeScript and heavigly documented with JSDoc. If you use vscode with this setup, you will see all the jsdoc information as you hover over different variables and functions.
Even though this was written in TypeScript, it can be used in any node project. This project is compiled to JS and the TS typing are there as needed
Right now, the main piece is the BigipConfig class in the ltm.ts file. This class is what takes in the bigip.conf at initiation. From there different functions can be called to get different information
The logger class is a simple logger implementation that collect all the logs througout processing and provides a way to return them as part of the response back. This is mainly to provide some feedback into what is happening without having to straight to debugging node directly.
Trying not to pollute the command line with a bunch of errors, So we catch and log all errors to the logger and the logs can be returned to the user as needed
The regex class provides a way to structure the different regexs needed to parse the different pieces of information. I feel that the json tree is not only a good way to hold and access all the regex's, but also an easy way to update pieces of the tree depending on tmos configuration variances of different code versions. In short, if pools are reference differently in v17, a flag that updates that single regex can easily be configured with mininmal impact to everything else.
Some example modifications have been documented in the function
jsonified configI have the idea that the TMOS config loosely represents a json structure. The parent tmos objects look like names json objects and everything else can end up being a regular object attribute as a "key": "value" pair.
If you check out the output of the configObjects class var, you will see a browsable json tree of the parent objects. What this could mean, is that the entire config could be jsonified, then searched for the needed data without the need for breaking things down with the regex tree. This could make data search and extraction very quick and efficient. The main downside to that, is that it is going to reguire some good js/json foo to be able to search and extract the necessary information. I feel like we are halfway there...
the test file bigip.conf_1.test.ts abuses the mocha testing suite to run the code as I have been developing. It's my intent to move more in the direction of test driven development, but I need to think and discuss with others about different approches I have been looking at.
We load a config from a file or get it through some other means, like api POST...
const devCloud01 = fs.readFileSync(path.join(__dirname, "artifacts", "devCloud01_9.17.2020.conf"), "utf-8");Then we create and initialize the class as follows:
const devCloud = new BigipConfig();UPDATE: with the parsing of archives, no data is provided at instantiation, now a .load('ucs/qkview/file/path') function is provided to load the data. After loading, one can execute the .parse() function to parse the configs or parse will happen automatically with the .explode() function (which just returns most of the necessary data)
The .apps() function returns an array of objects if no parameter is supplied. However, you can feed it a list of apps (vs names) for it to return configurations for
[{
"name": "vs_name",
"config": "extracted_app_config",
"map": "src->dst IPs"
}]The .logs() function returns a log of the extraction process. It is recommended to call this to get the logs if any one of the previous functions do not return a value
The top of the main class also describes some of the different ways I have the the initial breakdown of the parent objects. These are public definitions on the class which can be access directly: const x = devCloud.configMultiLevelObjects
/**
* tmos config as nested json objects
* - consolidated parant object keys like ltm/apm/sys/...
*/
public configObjects: goodBigipObj = {};Everything is in a json tree for easy manipulation
Check out the .log and .json outputs in the ./tests/artifacts directory
Please see Install and Usage below
Aside from trying to incorporat this rpm into another nodejs project (not recommneded in this early stage), there is the command line utility.
Corkscrew comes with a simple command line utility. This utility takes in a bigip.conf and produces the output of the "explode" function, which pretty much runs through all current functionality. This output is in json formate and is intented to be parsed with tools like "jq"
Make sure you have node installed with NPM
ted@thanos:~$ node --version && npm --version
v10.19.0
6.13.4Run the following command to download the rpm
npm install -g https://github.com/f5devcentral/f5-corkscrew.git
latest branch as of 10.26.2021 is v0.9.0 ->
npm install -g https://github.com/f5devcentral/f5-corkscrew.git#v0.9.0
Run corkscrew to confirm installation status
ted@thanos:/mnt/c/Users/bunoo/f5-corkscrew/src/tests$ node --version
v10.19.0
ted@thanos:/mnt/c/Users/bunoo/f5-corkscrew/src/tests$ npm --version
6.13.4
ted@thanos:/mnt/c/Users/bunoo/f5-corkscrew/src/tests$ corkscrew
cli.js <command>
Commands:
cli.js explode <file> explode bigip.conf to apps
Options:
--help Show help [boolean]
--version Show version number [boolean]
--no_sources supress config file sources bigip.conf, bigip_base.conf output [boolean]
--no_file_store supress filestore files output [boolean]
--no_command_logs no cli output [boolean]
--no_conversion_logs no extraction parsing logs [boolean]
A command is requiredSince the package is not published to NPM and it is installed directly from the repo, I'm not sure how to track any of the version information so you can easily update when a new version is released.
It seems the only route at this time is to uninstall the package, make sure the directory is deleted, and re-install the packge, which should get the latest version
The npm install should link the corkscrew cli scrit with global search paths, so it should be able to be called from anywhere node/npm knows about
example: corkscrew explode <./path/to/bigip.conf>
NOTE: It is highly recommended to utilize a tool like "jq" to be able to parse the output as needed
NOTE: It is also recommended to pipe the ouput of the explode command to a file so parsing/processing happens once.
https://shapeshed.com/jq-json/
Execute corkscrew and pipe the results to a file
user@workstation:/tests$ corkscrew explode ./path/to/bigip.conf >> demo.jsonHow to list app names (vs)
ted@thanos:/tests$ echo demo.json | jq .config.apps[].name
"/Common/app1_t80_vs"
"/Common/app1_t443_vs"
"/Common/app2_t80_vs"
"/Common/app2_t443_vs"
"/Common/app3_t8443_vs"
"/Common/app4_t80_vs"
"/Common/forwarder_net_0.0.0.0"Get app by array number
ted@thanos:/mnt/c/Users/bunoo/f5-corkscrew/src/tests$ cat demo.json | jq .config.apps[2]
{
"name": "/Common/app2_t80_vs",
"config": "ltm virtual /Common/app2_t80_vs {\n creation-time 2020-09-17:08:50:22\n destination /Common/192.168.2.21:80\n ip-protocol tcp\n last-modified-time 2020-09-17:08:51:07\n mask 255.255.255.255\n profiles {\n /Common/http { }\n /Common/tcp { }\n }\n rules {\n /Common/_sys_https_redirect\n }\n serverssl-use-sni disabled\n source 0.0.0.0/0\n translate-address enabled\n translate-port enabled\n}\n"
}How to list keys of the parent object
ted@thanos:/tests$ echo demo.json | jq keys
[
"config",
"dateTime",
"id",
"logs",
"stats"
]How to show the details of a nested object
ted@thanos:/tests$ echo demo.json | jq .stats
{
"configBytes": 57711,
"lineCount": 1125,
"objectCount": 153,
"objects": {
"virtuals": 7,
"profiles": 6,
"policies": 2,
"pools": 7,
"irules": 5,
"monitors": 6,
"nodes": 10,
"snatPools": 1
},
"parseTime": 5.4783,
"appTime": 6.7522,
"packTime": 1.39
}Examples on how to fetch individual json object details
ted@thanos:/tests$ echo demo.json | jq .id
"3a07cc36-781c-4183-8fb8-4858a5bab6a7"
ted@thanos:/tests$ echo demo.json | jq .dateTime
"2020-10-08T18:43:29.732Z"At a minimum JSDoc practices should be followed to document code and function. Heavey comments will really help the integration/merge process.
Please include some sort of tests for any new features or functionality. It's really the only way to develop on this since there is no main app to run the functions
For support, please open a GitHub issue. Note, the code in this repository is community supported and is not supported by F5 Networks.
Please refer to the F5 DevCentral Community Code of Conduct.
Copyright 2014-2020 F5 Networks Inc.
Before you start contributing to any project sponsored by F5 Networks, Inc. (F5) on GitHub, you will need to sign a Contributor License Agreement (CLA).
If you are signing as an individual, we recommend that you talk to your employer (if applicable) before signing the CLA since some employment agreements may have restrictions on your contributions to other projects. Otherwise by submitting a CLA you represent that you are legally entitled to grant the licenses recited therein.
If your employer has rights to intellectual property that you create, such as your contributions, you represent that you have received permission to make contributions on behalf of that employer, that your employer has waived such rights for your contributions, or that your employer has executed a separate CLA with F5.
If you are signing on behalf of a company, you represent that you are legally entitled to grant the license recited therein. You represent further that each employee of the entity that submits contributions is authorized to submit such contributions on behalf of the entity pursuant to the CLA.