This is an experimental compiler for P416 and P414 files. For publications and more, see our homepage.
An older version of the compiler is also available.
Find out more about the P₄ language.
To start working with the compiler, simply download the bootstrap-t4p4s.sh
script and execute it in the following way.
The script installs all necessary libraries (DPDK, P4C, P4Runtime
and more) and T₄P₄S itself, and sets up environment variables.
wget https://raw.githubusercontent.com/P4ELTE/t4p4s/master/bootstrap-t4p4s.sh
chmod +x bootstrap-t4p4s.sh
. ./bootstrap-t4p4s.sh
Notes.
⚠ The purpose of the script is to setup a convenient environment for you if you're just starting out. Therefore, you only need to execute it once. If you already have a working environment, you don't need to run the script again, and you probably shouldn't.
git clone --recursive https://github.com/P4ELTE/t4p4s
git pull --recurse-submodules
bootstrap-t4p4s.sh
, the previously installed content will be put in a backup directory.Without the .
at the beginning of the line, the environment variables will not be usable immediately.
. ./t4p4s_environment_variables.sh
The script is intended to work on recent Debian based systems, e.g. the latest LTS edition of Linux Mint or Ubuntu.
INSTALL_STAGE2_DPDK=no INSTALL_STAGE3_PROTOBUF=no INSTALL_STAGE4_P4C=no . ./bootstrap-t4p4s.sh
P4Runtime
features.INSTALL_STAGE5_GRPC=no . ./bootstrap-t4p4s.sh
Setting the option DPDK_APPS=no
lets you install DPDK without its applications. The SLIM_INSTALL=yes
option further quickens installation by not compiling many of the device drivers of DPDK.
DPDK_APPS=no . ./bootstrap-t4p4s.sh SLIM_INSTALL=yes . ./bootstrap-t4p4s.sh
You may customise the build process further. As the P₄ library is rather slow to compile to begin with, CFLAGS
isn't passed to it, but you may use the specific P4C_CFLAGS
variable. Here is a sample configuration.
DPDK_APPS=no CFLAGS="-march=native -mtune=native -O2" MESONFLAGS="-Db_pch=true -Ddebug=false -Doptimization=2 -Dstrip=true" P4C_CFLAGS="" . ./bootstrap-t4p4s.sh
To see all possible options (including available stages), run the script the following way.
./bootstrap-t4p4s.sh showenvs
If you happen to have some of the dependencies locally checked out, you can speed up the installation process by letting the script clone them locally.
LOCAL_REPO_CACHE=/my/cache/dir . ./bootstrap-t4p4s.sh
At this stage of development, T₄P₄S will not compile and run all P₄ programs properly. In particular, header stacks are not supported currently.
Overriding defaults.
To increase efficiency, the script runs jobs on all cores of the system in parallel. Should you experience any problems (for example, your system may run out of memory), you can override the number of jobs.
MAX_MAKE_JOBS=4 . ./bootstrap-t4p4s.sh
By default, the script runs downloads in parallel. You can force it to work sequentially.
PARALLEL_INSTALL=no . ./bootstrap-t4p4s.sh
The script installs sensibly recent versions of DPDK, P4C and PROTOBUF by default. To use the most recent versions, set INSTALL_TOOL_VSN
to newest
.
INSTALL_TOOL_VSN=newest . ./bootstrap-t4p4s.sh
You can also individually override the version/date settings.
DPDK_VERSION=20.05 . ./bootstrap-t4p4s.sh DPDK_VERSION=20.05 DPDK_FILEVSN=20.05.0 . ./bootstrap-t4p4s.sh P4C_COMMIT_DATE=2020-11-01 . ./bootstrap-t4p4s.sh
The script uses clang
, clang++
and lld
by default if they are installed unless overridden. It also uses ccache
.
T4P4S_CC=gcc T4P4S_CXX=g++ T4P4S_LD=bfd . ./bootstrap-t4p4s.sh
You can test the correct installation of the compiler by running the following simple example ./t4p4s.sh %l2fwd
.
This should result in the following message T₄P₄S switch running ./examples/p4_14/l2fwd.p4_14 exited normally
.
The example runs a single test case offline (so no network card is needed to test it).
In the t4p4s.sh
script, options control the process of compilation and execution.
The options are collected in the following phases.
colours.cfg
, lights.cfg
, the command line, examples.cfg
and opts_${ARCH}.cfg
are processed.
colours.cfg
describes the available colours for output highlighting.lights.cfg
describes which colours are used in the terminal and in the switch output for highlighting.examples.cfg
sets options for each example.opts_${ARCH}.cfg
sets architecture specific options.${ARCH}
is dpdk
.p4
.lights.cfg
) contain an option definition on a single line.examples.cfg
) contain an example identifier and then any number of options on a line.
examplename@testcasename
, or if @testcasename
is not given, @std
is used by default.;
not preceded by a number, until the end of the line) are ignored.The format of option definitions is the following.
-
(dash), _
(underscore) and .
(dot).myopt
with the value myval
.
myopt=myval
myopt
that takes the default value on
.
myopt
myopt=foo bar
myopt
is ignored: it is considered not to be defined.
^myopt
myopt
with the value val
only if mycondopt
is defined at this time.
mycondopt->myopt=val
myopt
with the value val
only if mycondopt
is not defined at this time.
^mycondopt->myopt=val
myopt
with the value val
only if mycondopt
is defined at this time, and its value is condval
.
mycondopt=condval->myopt=val
+=
++=
can take the place of =
. Instead of setting the option, they append to the current value: +=
with a space separator, ++=
with a newline.
myopt=foo
and then myopt+=bar
is equivalent to myopt=foo bar
For convenience, there are some abbreviations.
Option given | Equivalent to |
---|---|
@myvariant | variant=myvariant |
:myexample | example=myexample |
::myexample | example=myexample dbg |
%myexample=mytestcase | example=myexample variant=test testcase=mytestcase |
%myexample | example=myexample variant=test testcase=test |
%%myexample=mytestcase | example=myexample variant=test verbose dbg testcase=mytestcase |
The script returns an exit code of 0
if the execution was successful, and a non-zero value otherwise.
The script creates build/<example-name>
.
build
, srcgen
and Makefile
contain compilation artifacts, including the created switch executable.log
.
controller.log
is the log output from the most recent controller execution..txt
extension is a regular textual log.lit.txt
extension contains ANSI colour codes. Invoking cat
on it, or using an appropriate viewer like SublimeANSI will show coloured output.last.txt
and last.lit.txt
../t4p4s.sh :l2fwd
examples
automatically, but you can also specify it manually
./t4p4s.sh ./examples/p4_14/l2fwd.p4_14 model=v1model
./t4p4s.sh :l2fwd p4
./t4p4s.sh :l2fwd c
./t4p4s.sh :l2fwd run
p4 c run
) are active
./t4p4s.sh :l2fwd
p4 c run
order)
./t4p4s.sh :l2fwd p4 c
./t4p4s.sh :l2fwd c p4
./t4p4s.sh :l2fwd p4=on c=on run=on
^
suppresses it
./t4p4s.sh :l2fwd ^run
./t4p4s.sh :l2fwd ctrcfg=my_ctr_opts.txt
./t4p4s.sh :l2fwd bw
./t4p4s.sh :l2fwd bw=terminal
./t4p4s.sh :l2fwd bw=switch
./t4p4s.sh :l2fwd verbose
./t4p4s.sh :l2fwd dbg
./t4p4s.sh :l2fwd dbg=1
./t4p4s.sh :l2fwd dbg stats
stats=1
./t4p4s.sh :l2fwd dbg stats=1
./t4p4s.sh :l2fwd noeal
./t4p4s.sh :l2fwd silent
gdb
)
./t4p4s.sh :l2fwd autodbg
./t4p4s.sh %l2fwd=test
, ./t4p4s.sh %l2fwd=payload
examples/test-l2fwd.c
./t4p4s.sh %l2fwd=payload strict
examples.cfg
sets the required number of hugepages for each example1024 MB
of hugepages
./t4p4s.sh %l2fwd hugemb=1024
./t4p4s.sh %l2fwd hugepages=1024
t4p4s.sh
not to modify the current number of hugepages (may cause problems if it is less than required for the example)
./t4p4s.sh %l2fwd hugepages=keep
t4p4s.sh
to adjust the number of hugepages exactly to the requested amount (by default, the hugepage count is never decreased)
./t4p4s.sh %l2fwd hugeopt=exact
EXAMPLES_CONFIG_FILE="my_config.cfg" ./t4p4s.sh my_p4 @test
EXAMPLES_CONFIG_FILE="my_config.cfg" COLOUR_CONFIG_FILE="my_colors.txt" P4_SRC_DIR="../my_files" ARCH_OPTS_FILE="my_opts.cfg" ./t4p4s.sh %my_p4 dbg verbose
./t4p4s.sh showenvs
showenvs
is not the first argument, it prints the argument values after they have been fully computed/substituted.
./t4p4s.sh %l2fwd showenvs
./t4p4s.sh :l2fwd ctr=l2fwd
gnome-terminal
is used, as the more general x-terminal-emulator
does not seem to work properly.
./t4p4s.sh %my_p4 ctrterm
extern void log(string s);
to your P₄ file, calls to log("My message")
will produce a line in the debug output.
./t4p4s.sh %my_p4 x_log
./t4p4s.sh %my_p4 recompile
./t4p4s.sh %my_p4 hint=all
./t4p4s.sh %my_p4 hint=nopath
./t4p4s.sh %my_p4 hint=noext
./t4p4s.sh %my_p4 hint=nofile
meson
. See more details here.
MESON_BUILDTYPE=release ./t4p4s.sh %my_p4
clang
, you may make use of thin-lto
.
./t4p4s.sh %my_p4 lto
meson
like this.
./t4p4s.sh %my_p4 mopt+=-Db_ndebug=if-release mopt+=-Doptimization=3 mopt+=-Ddebug=false
upx
. This is a reasonable compromise between compression ratio and speed.
./t4p4s.sh %my_p4 upx
upx
, you can pass options to the command. Note that --brute
and --ultra-brute
will probably break the executable.
UPX_OPTS=-1 ./t4p4s.sh %my_p4
./t4p4s.sh %my_p4 upx=--best
./t4p4s.sh :l2fwd vsn=14
./t4p4s.sh :l2fwd-gen model=v1model
, ./t4p4s.sh :l2fwd-gen model=psa
T4P4S_TEST_1
that is available during P₄ preprocessing.
./t4p4s.sh %my_p4 p4testcase=1
As described above, you can run individual test cases. To see detailed output about compilation and execution, use the following options.
./t4p4s.sh %%l2fwd=payload
To run all available test cases, execute ./run_tests.sh
.
You can also give this script any number of additional options.
./run_tests.sh verbose dbg stats
As its name implies, run_tests.sh
runs each test case in the offline (nicoff
, meaning no NIC present) mode.
You may set the PREFIX
and POSTFIX
environment variables to make the script start t4p4s.sh
with a different setup for the test case.
For example, the following command tests whether the test cases compile in the online (nicon
) mode, but it doesn't execute them.
PREFIX=: POSTFIX="" ./run_tests.sh ^run
Once the test cases are run, the script prints a summary of successful and failed test cases,
grouped by the types of failures.
You may indicate which tests are to be skipped by listing them in a file.
See the default skip file, tests_to_skip.txt
, for further details.
SKIP_FILE="my_skip_file" ./run_tests.sh verbose dbg stats
The batch file processes all P4 files from the folder examples
and its subfolders (including symlinked ones) by default. You can override it like this.
START_DIR=my_examples/testcases/ ./run_tests.sh verbose dbg stats
The script can generate a HTML and JSON output of the test into the build/all-run-logs
folder in the following way:
HTML_REPORT="yes" ./run_tests.sh
There is also a possibility to ask the run_test to run multiple times a test if it not fails. There are cases that is not deterministic, so with this option you can test a case multiple times to find the problem.
RUN_COUNT=3 ./run_tests.sh
Currently, this area of T₄P₄S is under rewrite. Stay tuned.
For more details on how to work with HLIR attribures, see the readme of the hlir16
submodule.
The compiler uses the .py
files inside the hardware_indep
directory to generate Python code (saved with the extension .desugared.py
under build/util/desugared_compiler
), then executes the code to produce .c
files. Under src/utils
, files with the extension .sugar.py
are also primarily used as code generators. The files are written with some syntactical sugar, which is described in the following.
hardware_indep
have access to the global variable hlir16
, which is the root of the representation.
generated_code
global variable that starts out with an empty text. Usually, you do not want to manipulate it directly.PyExpr
stands for a Python expression.
#[ (insert generated code here)
: the code will be textually added to generated_code
#[ ... $my_var ...
: the textual value of the Python variable my_var
is inserted here#[ ... ${PyExpr} ...
: the code is evaluated, then its result will be inserted as text#= PyExpr
: the expression is evaluated, its result is inserted textually
#[ ${Python expression}
#{
and #}
: the same as #[
, except that code between the two will be indented one level
#{
markers will have a proper corresponding #}
marker$${PyExpr}
highlights the evaluated text using the default colour (T4LIGHT_default
)
$$[mycolourname]{PyExpr}
uses T4LIGHT_mycolourname
as the colour of highlighting; these colours are defined in lights.cfg
and must be listed in ALL_COLOURS
of t4p4s.sh
$$[mycolourname]{PyExpr}{text}
is the same as above, but text
(which is just plain text) also appears in the highlighted part$$[mycolourname][text1]{PyExpr}{text}
is the same as above, but text1
(which is just plain text) also appears in the highlighted partT4LIT(some text)
or T4LIT(my header instance's name,hdrinst)
.sugar.py
files, but are used in hardware_indep
as well.
gen_
are considered helper functions in which the above markers are usable.
generated_code
variable that starts out empty, and they will return it at the end.gen_
part) that calls the function.uint8_t /* codegen@123*/
means that the text uint8_t
was generated by executing code on or around line 123 in codegen.sugar.py
(in the directory src/utils
).... // actions@123
file_sugar_style
and the class SugarStyle
(in compiler.py
), see the end of codegen.sugar.py
for usage examples.It is hardcoded in the current prototype to create an OpenSSL-based virtual crypto device in DPDK in order to support encryption and decryption extern functions. The PMD for this virtual device is not compiled in DPDK by default.
To enable the OpenSSL crypto PMD, edit dpdk-19.02/config/common_base by changing
CONFIG_RTE_LIBRTE_PMD_OPENSSL=n
to
CONFIG_RTE_LIBRTE_PMD_OPENSSL=y
and do a rebuild on DPDK.
You can also activate a separate crypto node to run the commands with parameter crypto_node=openssl
. If you run a crypto node, you have to configure an extra core that only will do the external job.
If you want to test with a constant time external function, you can set crypto_node=fake
and set for example the time with fake_crypto_time=5000
that sets the external function to run until 5000 clock ticks.
An example call of async mode:
./t4p4s.sh :l2fwd-gen cores=4 ports=3x2 async_mode=pd crypto_node=fake fake_crypto_time=3000