Current file-based reconfiguration is a huge pain:
it's hard to add/update single server group or vhost if there are already thousands of server groups and/or vhosts - we have to traverse them all to understand what differs. File configuration update always means full configuration update and we need to understand how it differs from the previous configuration.
There is no good solution for a configuration update - if a part of live reconfiguration update fails, then it's hard to rollback previous changes.
File configuration isn't suitable for a distributed environment: apply the same configuration change to all the nodes or configure a node remotely. If we update a configuration remotely (e.g. via REST API), then we have to update the configuration file somehow and it's painful to keep comments and formatting.
Automatic configuration generation and validation would be a problem - the configuration language should have a grammar and allow schema-less configuration updates like Envoy.
At this point, it's clear that the whole configuration, maybe except quite a few properties (e.g. path to TDB files) necessary to start Tempesta, must be dynamically reconfigurable. Rgardles whether a particular option is dynamically reconfigurable or not, the whole configuration must be stored in TDB persistent tables, so depends on #516. No current file configuration is required.
Thus we have to move to TDB-handled configuration from the file.
Reconfiguration
We need to provide gRPC/Flatbuffer (the most efficient one) API for the configuration updates. The API must be implemented by a rest module which processes the HTTP request and fetches the reconfiguration string which is dispatched (as part of parsed HTTP message via kernel-user space transport, #77) to a user space daemon tfw_mgr.
The rest module must provide strong authentication of all the ingress requests. Initially it could be done based on ingress physical interface and IP addresses checking specified in the configuration.
The tfw_mgr daemon parses Flatbuffer and calls libtdb API to update appropriate table in TDB. The updates must be done in transactional context (#516) to be able to rollback the change on any problem. TDB must also implement events notification (callbacks), also required fro #516. TL programs must be compiled also in the user space using a separate parser. The daemon can also make any compilation of user friendly configuration into low-level representation (see for example https://github.com/tempesta-tech/tempesta/issues/1105).
Different TDB tables must be used for the configuration handling. At least separate tables for vhosts and server groups are required since these are the place for multiple configuration entities. Moreover, currently server groups are stored in sg_hash - a perfect candidate for a migration to TDB HTrie.
The control flow for the configuration update is following (everything works in process context of tfw_mgr):
tfw_mgr starts a transaction and execute all the updates on all the tables
tfw_mgr commits transaction and TDB calls trx_prepare hooks on table updates
trx_prepare hooks call routines of all the modules owning updated configuration tables and the modules do all necessary preparatin (e.g. memory allocations) and validation of the changes. They do not affect current configuration.
Next trx_commit hook is called in all trx_prepare returned 0 and all the modules apply changes to their configuration.
If some of the trx_prepare hooks fails, then trx_rollback hooks are called which free all allicated resources.
The user space configuration functionality must be implemented in a separate library libcfg, whcih is called by tfw_mgr.
Basically, it seems we don't need transactions from Tempesta DB and just can update all the entries in RCU faschion. But the reconfiguration process on it's own really needs something like 2PC protocol (described at the above).
libcfg
The main configuration parsing and TDB actions (using libtdb) must be done in libcfg. The library is used by the configuration daemon tfw_mgr and CLI tool tfw.
Possible request to load new filter rules could look like
POST /rest/filter?load HTTP/1.1
Host: tempesa
Content-Type: application/x-www-form-urlencoded
Content-Length: N
block=IP%3D1%2E1%2E1%2E1%26URL%7Efoo&block=URL%3Dfoo.com
This request adds two blocking rules:
block clients with IP=1.1.1.1 (IP%3D1%2E1%2E1%2E1) and URL containing (~) foo;
block all requests with URL equal (=) to foo.com
PUT, DELETE and other methods also should be used.
TL (#102) programs must be stored in separate table, probably depending on their context (e.g. req or resp processing contexts).
GET /rest/stats?perf HTTP/1.1
Host: tempesa
System statistics must be returned by Tempesta FW in JSON document.
Interoperability
Migration from the file
Sometimes file configuration might be handy and it makes sense to move to the new configuraion scheme smoothly. So tempesta.sh (using a helper script) should
parse the configuration file
remove options supporting the new interface
run Tempesta with minimal configuration
executes the script to translate the config file to JSON and execute the REST API for configuration
run tfw tool to update the configuration via libcfg
File visibility
tfw should implement more user friendly configuration interface than tdbq working with raw TDB data representation. In particular, it must implement show option which print the whole configuration with all necessary identation is single text representation.
Current tfw_init() logic remains the same, but tfw_start() abandons tfw_cfg_parse() - instead of reading and parisng the file, Tempesta reads TDB configuration tables and execute trx_prepare and trx_commit hooks on all modules and tables. Next, either if we loaded full configuration or all the tables are empty, Tempesta waits for TDB events (callback, #516) on configuration table. On the initial run, when all the tables are empty, tfw loads configuration from the configuration file, JSON, or interactive user commands.
Historically there wasn't vhost and location directives and srv_group directive included some options that actually describe web service behaviour. This options are about HTTP processing features:
It seems reasonable to move options from the first group to vhost settings.
Need to review all the configuration directives and stabiles them. Future additions must extend but not change configuration format. Tempesta's upgrade to the newer version should be possible without any issues.
.htaccess
tdbfscan provide directory based per-vhost configuration with full user/group access validation (thanks to OS filesystem inteface), so we can provide configuration overriding for non-priveledged users, which is available in Apache HTTPD only at the moment and which seems the most influential reason for people to stay with the server.
Introduction
Current file-based reconfiguration is a huge pain:
At this point, it's clear that the whole configuration, maybe except quite a few properties (e.g. path to TDB files) necessary to start Tempesta, must be dynamically reconfigurable. Rgardles whether a particular option is dynamically reconfigurable or not, the whole configuration must be stored in TDB persistent tables, so depends on #516. No current file configuration is required.
Thus we have to move to TDB-handled configuration from the file.
Reconfiguration
We need to provide gRPC/Flatbuffer (the most efficient one) API for the configuration updates. The API must be implemented by a
rest
module which processes the HTTP request and fetches the reconfiguration string which is dispatched (as part of parsed HTTP message via kernel-user space transport, #77) to a user space daemontfw_mgr
.The
rest
module must provide strong authentication of all the ingress requests. Initially it could be done based on ingress physical interface and IP addresses checking specified in the configuration.The
tfw_mgr
daemon parses Flatbuffer and callslibtdb
API to update appropriate table in TDB. The updates must be done in transactional context (#516) to be able to rollback the change on any problem. TDB must also implement events notification (callbacks), also required fro #516. TL programs must be compiled also in the user space using a separate parser. The daemon can also make any compilation of user friendly configuration into low-level representation (see for example https://github.com/tempesta-tech/tempesta/issues/1105).Different TDB tables must be used for the configuration handling. At least separate tables for vhosts and server groups are required since these are the place for multiple configuration entities. Moreover, currently server groups are stored in
sg_hash
- a perfect candidate for a migration to TDB HTrie.The control flow for the configuration update is following (everything works in process context of
tfw_mgr
):tfw_mgr
starts a transaction and execute all the updates on all the tablestfw_mgr
commits transaction and TDB callstrx_prepare
hooks on table updatestrx_prepare
hooks call routines of all the modules owning updated configuration tables and the modules do all necessary preparatin (e.g. memory allocations) and validation of the changes. They do not affect current configuration.trx_commit
hook is called in alltrx_prepare
returned0
and all the modules apply changes to their configuration.trx_prepare
hooks fails, thentrx_rollback
hooks are called which free all allicated resources.The user space configuration functionality must be implemented in a separate library
libcfg
, whcih is called bytfw_mgr
.Basically, it seems we don't need transactions from Tempesta DB and just can update all the entries in RCU faschion. But the reconfiguration process on it's own really needs something like 2PC protocol (described at the above).
libcfg
The main configuration parsing and TDB actions (using libtdb) must be done in
libcfg
. The library is used by the configuration daemontfw_mgr
and CLI tooltfw
.The configuration parsing is done in user space, se
nft
also should be called from the table to tread multi-layer firewalling rules (see https://github.com/tempesta-tech/tempesta/issues/862#issuecomment-442868737).Examples
Possible request to load new filter rules could look like
This request adds two blocking rules:
IP%3D1%2E1%2E1%2E1
) and URL containing (~
) foo;=
) tofoo.com
PUT, DELETE and other methods also should be used.
TL (#102) programs must be stored in separate table, probably depending on their context (e.g.
req
orresp
processing contexts).System statistics must be returned by Tempesta FW in JSON document.
Interoperability
Migration from the file
Sometimes file configuration might be handy and it makes sense to move to the new configuraion scheme smoothly. So
tempesta.sh
(using a helper script) shouldtfw
tool to update the configuration vialibcfg
File visibility
tfw
should implement more user friendly configuration interface thantdbq
working with raw TDB data representation. In particular, it must implementshow
option which print the whole configuration with all necessary identation is single text representation.It also must show current active configuration (https://github.com/tempesta-tech/tempesta/issues/862#issuecomment-442880298).
Initialization
Current
tfw_init()
logic remains the same, buttfw_start()
abandonstfw_cfg_parse()
- instead of reading and parisng the file, Tempesta reads TDB configuration tables and executetrx_prepare
andtrx_commit
hooks on all modules and tables. Next, either if we loaded full configuration or all the tables are empty, Tempesta waits for TDB events (callback, #516) on configuration table. On the initial run, when all the tables are empty,tfw
loads configuration from the configuration file, JSON, or interactive user commands.Revision of configuration consistency
There are claims regarding consistency of current configuration options, e.g. https://github.com/tempesta-tech/tempesta/issues/862#issuecomment-423214152 :
Historically there wasn't
vhost
andlocation
directives andsrv_group
directive included some options that actually describe web service behaviour. This options are about HTTP processing features:While others describe the group of servers as single load-balaned array of server connections and 'low-level' connection and load balancer properties:
It seems reasonable to move options from the first group to vhost settings.
Need to review all the configuration directives and stabiles them. Future additions must extend but not change configuration format. Tempesta's upgrade to the newer version should be possible without any issues.
.htaccess
tdbfs can provide directory based per-vhost configuration with full user/group access validation (thanks to OS filesystem inteface), so we can provide configuration overriding for non-priveledged users, which is available in Apache HTTPD only at the moment and which seems the most influential reason for people to stay with the server.
References
F5 BIG-IP configuration on top of IMDB
Nginx Crossplane