dslx: investigate adding support for mini nettests

[bassosimone]

This issue is about experimenting with adding support for mini nettests to dslx. The current richer input prototype defines mini nettests as portions of a nettest that run within a larger nettest. For example, the IM nettests can all be implemented in terms of mini nettests.

In the current prototype, for example, we can express Facebook Messenger as follows:

        {
            "run": "nettest:run@v1",
            "with": {
                "nettest_name": "facebook_messenger",
                "report_id": "20230406T142431Z_facebookmessenger_IT_30722_n1_nLq4AP3YQWmW8hg6",

                // Note how here the richer input contains mininettests that
                // the experiment should run. This functionality allows us to
                // tweak the behavior of experiments using the check-in API.
                //
                // The engine ignores the mininettests it does not know.
                //
                // We initially determine the results in the probe and later we
                // will move the algorithm in the fastpath.
                //
                // We will not generate the top-level keys that cannot be
                // generated given the available mininettest IDs.
                "targets": [
                    {
                        "id": "fbmessenger-stun",
                        "run": "dns-lookup@v1",
                        "with": {
                            "domain": "stun.fbsbx.com"
                        }
                    },
                    {
                        "id": "fbmessenger-b-api",
                        "run": "tcp-connect-domain@v1",
                        "with": {
                            "domain": "b-api.facebook.com",
                            "port": 443
                        }
                    },
                    {
                        "id": "fbmessenger-b-graph",
                        "run": "tcp-connect-domain@v1",
                        "with": {
                            "domain": "b-graph.facebook.com",
                            "port": 443
                        }
                    },
                    {
                        "id": "fbmessenger-b-edge-mqtt",
                        "run": "tcp-connect-domain@v1",
                        "with": {
                            "domain": "edge-mqtt.facebook.com",
                            "port": 443
                        }
                    },
                    {
                        "id": "fbmessenger-external-cdn",
                        "run": "tcp-connect-domain@v1",
                        "with": {
                            "domain": "external.xx.fbcdn.net",
                            "port": 443
                        }
                    },
                    {
                        "id": "fbmessenger-scontent-cdn",
                        "run": "tcp-connect-domain@v1",
                        "with": {
                            "domain": "scontent.xx.fbcdn.net",
                            "port": 443
                        }
                    },
                    {
                        "id": "fbmessenger-star",
                        "run": "tcp-connect-domain@v1",
                        "with": {
                            "domain": "star.c10r.facebook.com",
                            "port": 443
                        }
                    }
                ]
            }
        }

The above design defines a bunch of mini nettests by name (e.g., tcp-connect-domain@v1). I am worried that defining mini nettests in this way leads to a bunch of code to maintain on the probe side. In fact, to implement mini nettests in this fashion we need to define an input data structure and an implementation function for many combinations of the fundamental network operations, including DNS lookup, TCP connect, TLS handshake, QUIC handshake, HTTP round trips.

Because we have recently implemented composable measurement primitives as part of the dslx package, I would like to explore whether it is possible to implement changes to dslx such that we can express measurement primitives as a flat pipeline. A YAML example of what I mean is the following:

pipelines:
  fbmessenger_star:
    - run: core/dns_lookup
      with: 
        domain: "star.c10r.facebook.com"

    - run: fbmessenger/dns_lookup_check
      with:
        target: star

    - run: core/tcp_connect
      with:
        port: 443

    - run: fbmessenger/tcp_connect_check
      with:
        target: star

It is probably possible to make dslx more generic to support this use case. The question is how much complexity would this change entail. Therefore, I am going to sketch out a prototype first and then land it into master if it seems that this way of proceeding yields more benefits than the extra complexity it requires.

[bassosimone]

I have started sketching out this functionality in https://github.com/ooni/2023-05-richer-input/pull/1 and https://github.com/ooni/2023-05-richer-input/pull/2.

[bassosimone]

In https://github.com/ooni/2023-05-richer-input/pull/3, I have rewritten fbmessenger to use the new DSL prototype.

[bassosimone]

I have made additional progress in https://github.com/ooni/2023-05-richer-input/pull/5, as documented by the commit message.

[bassosimone]

In https://github.com/ooni/2023-05-richer-input/pull/6, I've updated fbmessenger again to sync up with https://github.com/ooni/2023-05-richer-input/pull/5.

[bassosimone]

In https://github.com/ooni/2023-05-richer-input/pull/7, I added a significantly improved DSL implementation. I think with https://github.com/ooni/2023-05-richer-input/pull/7 we can say the original objective of this issue has been met.

(This may also be a good moment as anyone else to say hello to my friend @Lymphatus)

[bassosimone]

With the merge of https://github.com/ooni/2023-05-richer-input/pull/8, this work is really done.

I'm going to close this issue.