[RFC] Python representation of events & post-processing interpreter

[atenart]

This is aimed at v1.4, to leave time to experiment with the feature and make it production ready. Having said that the current state is IMHO in a good shape and already allows to work with events in Python at post-processing time.

Events are parsed all at once and exported in the events variable. This is an object we control that provides some custom methods but also acts as a list of events. Those events are a Python representation of Retis events, with their own methods, and also act as a dict. Sub-sections and following types are not all supported, some like the skb tracking one are and export custom methods (e.g. tracking_id() or match()). For a few examples of what can be done, see demo.py.

There are two modes of operation:

• The interactive interpreter: retis python
• The execution of a script: retis python --exec foo.py

Current limitations or open questions:

• Enum with data cannot currently be converted to Python objects, but pyo3 has currently a PR opened and it seems like this is moving forward nicely. Because of this not all sub-sections were converted as a Python object and none got automatic get implementation (see pyo3 get_all) which would allow to use them as dict. The later could be done for some by adding an extra derive macro, but for this RFC that seemed enough. Because of this all objects implement a raw() method to get a raw list or dict representation.
• Events are parsed at startup time, meaning a very large set of events might be an issue. IMHO by default the current behavior is fine (otherwise each access to events would mean parsing the whole set) but we could introduce a lazy mode for very large sets, so the user can for example iter on events w/o having all of them in memory at once. Or automatically choose based on the events size and issue an INFO msg.
• On top of this we could, similarly to profiles, distribute scripts for specific post-processing cases. It also opens the question of moving our current post-processing commands to Python (not saying we should do it, but it's a possibility). This, combined with profiles, would allow things like: retis -p csum python. Or we could parse the sub-command and use it as a Python script if not a built-in one, eg. retis csum mapping internally to retis python --exec /etc/retis/python/csum.py.
• Some extra care is needed to look at how the Python interpreter is being called and how if this could be improved (eg. configured better). But note that current implementation made the release binary to grow to from 6.8M to 7.7M.

Fixes #35.

[vlrpl]

This is aimed at v1.4, to leave time to experiment with the feature and make it production ready. Having said that the current state is IMHO in a good shape and already allows to work with events in Python at post-processing time.

Events are parsed all at once and exported in the events variable. This is an object we control that provides some custom methods but also acts as a list of events. Those events are a Python representation of Retis events, with their own methods, and also act as a dict. Sub-sections and following types are not all supported, some like the skb tracking one are and export custom methods (e.g. tracking_id() or match()). For a few examples of what can be done, see demo.py.

There are two modes of operation:

* The interactive interpreter: `retis python`

* The execution of a script: `retis python --exec foo.py`

Current limitations or open questions:

* Enum with data cannot currently be converted to Python objects, but `pyo3` has currently a PR opened and it seems like this is moving forward nicely. Because of this not all sub-sections were converted as a Python object and none got automatic get implementation (see pyo3 `get_all`) which would allow to use them as dict. The later could be done for some by adding an extra derive macro, but for this RFC that seemed enough. Because of this all objects implement a `raw()` method to get a raw list or dict representation.

* Events are parsed at startup time, meaning a very large set of events might be an issue. IMHO by default the current behavior is fine (otherwise each access to `events` would mean parsing the whole set) but we could introduce a lazy mode for very large sets, so the user can for example iter on events w/o having all of them in memory at once. Or automatically choose based on the events size and issue an INFO msg.

* On top of this we could, similarly to profiles, distribute scripts for specific post-processing cases. It also opens the question of our current post-processing commands (not saying we should do it, but it's a possibility). This, combined with profiles, would allow things like: `retis -p csum python`. Or we could parse the sub-command and use it as a Python script if not a built-in one, eg. `retis csum` mapping internally to `retis python --exec $HOME/.config/retis/python/csum/py`.

* Some extra care is needed to look at how the Python interpreter is being called and how if this could be improved (eg. configured better). But note that current implementation made the release binary to grow to from 6.8M to 7.7M.

The first comment is this is awesome. About the size, it's good to point out, but the gain is so good that it is largely justified. Will take some time to play with it and get back on this.

[amorenoz]

Sharing some thoughts I've already shared in private.

I'm all for having python bindings but I'm not 100% (only) embedding a python interpreter in retis is the right way to go. Apart from requiring extra dependencies (python-devel), it will likely not work with some of the huge python echosystem: the many python virtual environment managers out there, existing visual REPLs such as IPython and Jupyter Notebooks, etc, as well as other python interpreters.

Although this approach would be great for small experiments, I think it'll limit the number of bigger python applications that could be built on top of retis events.

My proposal would be to (also?) build a python library with embedded binary bits coming from retis. This would allow users to just import a library and parse events (one by one if desired) as well as call event helpers implemented in rust.

Based on the current work I've hacked a quick&dirty experiment (available here) that allows parsing events with IPython:

❯ ipython
/usr/lib/python3.12/site-packages/IPython/core/interactiveshell.py:889: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn(
Python 3.12.2 (main, Feb 21 2024, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from pyretis import *

In [2]: r = PyEventReader("out.json")

In [3]: e = r.next()

In [4]: e
Out[4]: {'skb': {'tcp': {'ack_seq': 2525605617, 'dport': 43226, 'doff': 8, 'flags': 24, 'seq': 3008269705, 'sport': 8080, 'window': 260}, 'ip': {'v4': {'flags': 2, 'offset': 0, 'tos': 0, 'id': 5456}, 'saddr': '127.0.0.1', 'len': 54, 'ttl': 64, 'daddr': '127.0.0.1', 'ecn': 0, 'protocol': 6}, 'packet': {'capture_len': 66, 'packet': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 69, 0, 0, 54, 21, 80, 64, 0, 64, 6, 39, 112, 127, 0, 0, 1, 127, 0, 0, 1, 31, 144, 168, 218, 179, 78, 141, 137, 150, 137, 174, 241, 128, 24, 1, 4, 254, 42, 0, 0, 1, 1, 8, 10, 204, 104, 155, 204, 204, 104, 143, 221], 'len': 68}}, 'common': {'timestamp': 83093637144625, 'task': {'comm': 'node', 'pid': 4373, 'tgid': 4373}}, 'kernel': {'symbol': 'tcp_v4_rcv', 'probe_type': 'kprobe'}}

In [5]: e.show()
Out[5]: '83093637144625 [node] 4373 [k] tcp_v4_rcv\n  127.0.0.1.8080 > 127.0.0.1.43226 ttl 64 tos 0x0 id 5456 off 0 [DF] len 54 proto TCP (6) flags [P.] seq 3008269705:3008269707 ack 2525605617 win 260'

In [6]: 

[atenart]

Rebased on top of main and using a released version of PyO3.

[atenart]

Closing in favor of the upcoming Python support & lib PR.