Recirculation does not always copy metadata

[edgar-costa]

This applies to resubmit and probably to clone (Did not try).

Metadata fields are not copied to the recirculated packet unless you copy the entire metadata struct or you create a tuple of individual fields. For example if you have:

struct metadata_t {
   bit<8> count;
}
struct metadata { 
    metadata_t meta1;
    metadata_t meta2;
}

The metadata will be copied in the following cases:

recirculate(meta);
recirculate({meta.meta1.count, meta.meta2.count});

However, I remember (and just tested) that with older versions of bmv2/p4c copying only some structs from the main metadata worked. And the following does not work anymore:

recirculate(meta.meta1);
recirculate({meta.meta1, meta.meta2});

Is this a bug or it's an intended behaviour ? I personally think that the previous way made it quite easier, for example when you want to copy an entire structure that has 10s of fields.

[mihaibudiu]

You will get 10 fields in both cases. But in each case they will include the 5 you care about.

[mihaibudiu]

But, as I said, if we want fine-grained control we can have 2 different annotations. In v1model there is only one user metadata definition for all blocks.

[mihaibudiu]

For PSA we have to take a look. The way it's done now is just fine too.

[mihaibudiu]

We can just leave PSA as it is. It seems to support everything that is needed. It is also easier to understand how it works.

[mihaibudiu]

So I take back my comment on 10 fields above, that applies to v1model.

[jafingerhut]

Created PR https://github.com/p4lang/p4c/pull/1693 that removes standard_metadata from the P4_16 test program that currently preserves it across resubmit/recirculate/clone3 operations. It appears that I wrote the program so that the only thing it is relying on being preserved are the packet contents, and that the standard_metadata.instance_type is not preserved, but filled in as it should be to indicate whether the packet was recirculated, resubmitted, cloned, or none of those (e.g. a new packet arriving on a port).

I can look into creating a separate test case that relies for correct operation on preserving one or more user-defined metadata fields. Perhaps I will make it a P4_14 program, since at least then I know how to write the source code to follow the spec, and should continue working no matter how we change these operations for P4_16+v1model

[mihaibudiu]

I will hopefully submit a PR soon which does this using annotations on the user metadata structure - at least in P4-16. I still have to work on the translation p4-14 -> p4-16. It would be nice to have a program with an stf file that we run to make sure we haven't broken the resubmit again in the future.

[mihaibudiu]

I think this is the same bug as https://github.com/p4lang/p4c/issues/1694

[jafingerhut]

P4_16 + v1model does not use an intrinsic_metadata header, and yet resubmit still works in BMv2 JSON today for the v1model-special-ops.p4 and stf test program, so I do not understand how they can be the same bug?

[mihaibudiu]

If you are lucky the compiler does not use temporaries for the data you want to recirculate, so it works as you expect. I hope we are talking about the same problem. Isn't "standard_metadata" the same as what p4-14 calls "intrinsic metadata"?

[jafingerhut]

The example P4_14 program for issue #1694 can be compiled with today's p4c, and if you examine the BMv2 JSON file, there is a normal, un-mangled standard_metadata_t header_type, and a mangled intrinsic_metdata_t header_type. These are separate header_types in BMv2's implementation. I do not know the history here, but both standard_metadata_t and intrinsic_metadata_t contain fields that are "defined by the architecture", but for some probably-historical implementation reason they were put into separate headers. Similarly for queueing_metadata_t, which is yet another separate header containing additional fields for use in P4_14 programs. You can see documentation for P4_14 programmers using simple_switch here, which I wrote some of, but Antonin reviewed it all, so hopefully it is correct according to today's BMv2: https://github.com/p4lang/behavioral-model/blob/master/docs/simple_switch.md

The root cause of #1694 appears to me not to be that the compiler is introducing temporaries, but that p4c formerly used to generate BMv2 JSON files with not only standard_metadata_t header_type's, but also intrinsic_metadata_t and, if needed, queueing_metadata_t. Now the latter 2 are treated as if they are normal user-defined metadata header_type's instead.

[cc10512]

Sorry to join the party so late, and more over to throw a monkey wrench into the whole discussion. In my view, recirc and resubmit are such fundamental operations that it seems they have to have proper support in the language, rather than annotations. Sure, we can define externs for them (functions or objects) that capture the required semantics, just as we did in PSA. However, there were numerous discussions on how do we preserve the "call-by-name" semantics of P4-14, and the only solution we came up with is to leave it to programmers to invoke these extern functions in the deparser. Note that I would very much prefer to have the language support for call-by-reference for field lists. I know it is not an easy change and will require significant discussion for acceptance, but I think we should at least consider it.

Computing the union of all the fields or even recirculating the entire set is not a realistic option. Think about a scenario where you recirculate to implement a state machine that transitions from one type of processing to another and needs to consume data. If you have to maintain the whole set for every packet that is recirculated, you're likely to not make progress. The support for multiple field lists for both recirculation and resubmit is not negotiable -- any solution we devise has to do it, and it is better to have the language support to distinguish between the different instances. Otherwise we're just making life more difficult for both users, who will be confused on what data they get because they expect only a subset, and for us because we need to do magic to disentangle the different instances (based on ... not sure what).

So, I am afraid this is one of those cases where we have to accept that we made a mistake in simplifying the language a bit too much and go back and re-think a better design.

@jnfoster, you've been very quiet on this thread. Will you please chime in?

[hesingh]

With so many comments, it's hard to figure out all goals and non-goals of this work. It would be good to write a short doc and attach it to this issue. For example, have we agreed that ingress and egress user metadata is disjoint, because then we need to find a solution. Maybe this goal is no longer a goal.

I have already said, I don't like using Annotations.

[mihaibudiu]

@hesingh The goal is to support recirculation in v1model.

Having references would not solve anything. In architectures like Tofino the hardware does not support call-by-reference. Compiling references (in general) for such an architecture may be impossible. Field lists were used only in some very specific ways, and they match very closely the way BMv2 is implemented, that's why they worked fine for P4-14. The only type of architecture that would really benefit from having references is a software switch.

Even for the case of PSA, there are no shared resources between ingress and egress, so there is no way one can pass a reference between these two. So having references would not help at all.

I think that copy-in/copy-out actually matches quite closely the capabilities of hardware switches, that are deeply pipelined and have lots of parallel non-shared resources.

All our difficulties are confined to v1model. I think that the psa solution is actually very adequate. We should consider v1model a legacy architecture that should be maintained, but not one that has to be evolved.

[jafingerhut]

Well, the named field references as allowed by P4_14 field lists, with the entire field_list name used as an argument to resubmit, recirculate, and clone operations, is a pretty weak kind of named reference, compared to what they can be used to do in other programming languages. They cannot be used to implement call-by-reference as far as I can tell, because there is no way to pass them as arguments -- granted P4_14 only has arguments for actions, not controls.

They are fairly compile-time static things in P4_14 -- the names of field_lists cannot be stored in run-time variables, nor can references to field names be stored in run-time variables.

I am not saying I have a proposal to add similar things to P4_16 -- just that the limitations on them as they exist in P4_14 are much more severe than any other programming language I know of.

[jnfoster]

I am indeed weighing into this conversation late.

As a strawman, let's consider the simplest possible design, which is motivated by what many hardware targets really do. We could arrange things so that recirculate, resubmit, etc. don't actually do anything special to the headers and user-defined metadata -- they merely set a flag in the intrinsic metadata that causes the packet to take a non-standard processing path in the architecture, and indicates that it has done so. In this view, any data that needs to be "wormholed" along with the packet, needs to be handled explicitly by the programmer using the deparser and parser. For example, the programmer could define a header type to hold the recirculated data, make sure it is valid at the end of the pipeline in question so it will be emittted by the deparser, and then check for the flag and extract it in the parser.

What's non-optimal about this solution?

1. We lose the ability to "recirculate" data from anywhere in the pipeline. I'm not sure if this is a feature or a bug. Consider a program that calls recirculate(meta) and later in the same pipeline executes meta.f = 0. Which value of meta will be used? If the latter, it would be better in my opinion not to provide the illusion of being able to call recirculate anywhere since its effects take place later.

2. The programmer has to go to the trouble of defining a header type to store the recirculated data, this type needs to be flexible to store all different kinds of data that might be recirculated (maybe unions or overlays help), and most critically, they need to handle it explicitly in the deparser/parser! This is a bit silly since the data is really just being internally routed across the P4 device, and the compiler could very well handle the (de)serialization code automatically, just like it does for intrinsic metadata.

I agree with Calin that if this is an important feature in practical programs, relying on annotations to get a non-standard semantics (if that's what is being proposed) seems like the wrong strategy -- we should add new constructs that make common uses of this use case convenient.

If my analysis is right, could people weigh in on which of the two issues above is most critical? To recap: (i) being able to invoke recirculate and friends from anywhere in a control or (ii) automating the encoding/deparsing/parsing of this data? Or both!

If we understand the requirements, and have some example programs, then I think we can make progress towards a good solution.

And to focus attention on the issues, let's declare v1model and P4_14 translator a non-goal for the moment. We need to fix that too, but let's get the P4_16 design right first.

[jafingerhut]

I don't see anything wrong with your analysis. I think your strawman proposal sounds fairly close to what PSA proposes for these operations, except that it uses structs instead of headers for containing the data to be carried with a packet when it is recirculated, resubmitted, or cloned. It does require explicit assignments in the deparser and parser to make sure this data gets copied to the right output parameter in the deparser, and copied from the right input parameter in the parser.

Here is some sample P4_16 + PSA arch code for copying the data from a parser input parameter named recirc_meta into a place the user chose to put it in, during parsing: https://github.com/p4lang/p4-spec/blob/master/p4-16/psa/examples/psa-example-recirculate.p4#L94-L109

and here is the portion of the egress deparser code that copies the desired data to be recirculated into the PSA-defined output parameter: https://github.com/p4lang/p4-spec/blob/master/p4-16/psa/examples/psa-example-recirculate.p4#L191-L193

There are similar example programs psa-example-resubmit.p4 and psa-example-clone-to-port.p4 that show similar code examples for resubmit and clone cases, and also psa-example-bridged-metadata.p4 which gives a demo of PSA's explicit method for doing the corresponding thing for indicating which metadata is carried from ingress to egress for normal unicast and multicast packets, which is not P4_14's "everything is carried along if you use it in egress". These are in the directory https://github.com/p4lang/p4-spec/tree/master/p4-16/psa/examples

I am not saying that PSA is the only way to go -- there are design choices there that could be made differently, certainly. The design choices in PSA are, I believe, based on an explicit embracing of P4_16's "no spooky action at a distance", i.e. all function/control/action/extern calls take effect immediately, with the values the parameters have at the time of the call, not some later time.

I wouldn't be surprised if we could add some kind of early pass to a P4_16 compiler that adds field lists and delayed evaluation of the values of fields in the field list, and even auto-generated code similar to PSA for marshaling and demarshaling metadata to be carried with packets. The 'output' language of this compiler pass could be something like today's P4_16+PSA arch. The 'input' language could be something that has more P4_14-like constructs that make it more convenient for programmers, assuming that is what is desired.

[mihaibudiu]

The annotations do not change the semantics of p4-16, they merely describe how the environment of he program behaves.

Mihai

---

From: Nate Foster notifications@github.com Sent: Sunday, February 3, 2019 8:49:37 AM To: p4lang/p4c Cc: Mihai Budiu; Mention Subject: Re: [p4lang/p4c] Recirculation does not always copy metadata (#1669)

I am indeed weighing into this conversation late.

As a strawman, let's consider the simplest possible design, which is motivated by what many hardware targets really do. We could arrange things so that recirculate, resubmit, etc. don't actually do anything special to the headers and user-defined metadata -- they merely set a flag in the intrinsic metadata that causes the packet to take a non-standard processing path in the architecture, and indicates that it has done so. In this view, any data that needs to be "wormholed" along with the packet, needs to be handled explicitly by the programmer using the deparser and parser. For example, the programmer could define a header type to hold the recirculated data, make sure it is valid at the end of the pipeline in question so it will be emittted by the deparser, and then check for the flag and extract it in the parser.

What's non-optimal about this solution?

1. We lose the ability to "recirculate" data from anywhere in the pipeline. I'm not sure if this is a feature or a bug. Consider a program that calls recirculate(meta) and later in the same pipeline executes meta.f = 0. Which value of meta will be used? If the latter, it would be better in my opinion not to provide the illusion of being able to call recirculate anywhere since its effects take place later.

2. The programmer has to go to the trouble of defining a header type to store the recirculated data, this type needs to be flexible to store all different kinds of data that might be recirculated (maybe unions or overlays help), and most critically, they need to handle it explicitly in the deparser/parser! This is a bit silly since the data is really just being internally routed across the P4 device, and the compiler could very well handle the (de)serialization code automatically, just like it does for intrinsic metadata.

I agree with Calin that if this is an important feature in practical programs, relying on annotations to get a non-standard semantics (if that's what is being proposed) seems like the wrong strategy -- we should add new constructs that make common uses of this use case convenient.

If my analysis is right, could people weigh in on which of the two issues above is most critical? To recap: (i) being able to invoke recirculate and friends from anywhere in a control or (ii) automating the encoding/deparsing/parsing of this data? Or both!

If we understand the requirements, and have some example programs, then I think we can make progress towards a good solution.

And to focus attention on the issues, let's declare v1model and P4_14 translator a non-goal for the moment. We need to fix that too, but let's get the P4_16 design right first.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fp4lang%2Fp4c%2Fissues%2F1669%23issuecomment-460067799&data=02%7C01%7Cmbudiu%40vmware.com%7Cd5ee401b669f4f40205808d689f795f8%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636848093812160430&sdata=FzxATIigAkVz6110RbMYZF3ffq4KnkDuQ9NxeAE%2F0ug%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASf5sl1GqXkbSO4eaLAYr0OAZlvEL6naks5vJxMhgaJpZM4aDL16&data=02%7C01%7Cmbudiu%40vmware.com%7Cd5ee401b669f4f40205808d689f795f8%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636848093812160430&sdata=ATrOQKGpz4QSJTKtAKXp%2FIP1hHEKc9hcsc2DMqh1kVU%3D&reserved=0.

[jnfoster]

Sure, in a narrow sense. They do change the way an operation supported on a specific standard architectures behaves... one that Calin is arguing is central enough to many programs that we should give it first-class support.

On Sun, Feb 3, 2019 at 1:02 PM Mihai Budiu notifications@github.com wrote:

The annotations do not change the semantics of p4-16, they merely describe how the environment of he program behaves.

Mihai

---

From: Nate Foster notifications@github.com Sent: Sunday, February 3, 2019 8:49:37 AM To: p4lang/p4c Cc: Mihai Budiu; Mention Subject: Re: [p4lang/p4c] Recirculation does not always copy metadata (#1669)

I am indeed weighing into this conversation late.

As a strawman, let's consider the simplest possible design, which is motivated by what many hardware targets really do. We could arrange things so that recirculate, resubmit, etc. don't actually do anything special to the headers and user-defined metadata -- they merely set a flag in the intrinsic metadata that causes the packet to take a non-standard processing path in the architecture, and indicates that it has done so. In this view, any data that needs to be "wormholed" along with the packet, needs to be handled explicitly by the programmer using the deparser and parser. For example, the programmer could define a header type to hold the recirculated data, make sure it is valid at the end of the pipeline in question so it will be emittted by the deparser, and then check for the flag and extract it in the parser.

What's non-optimal about this solution?

1. We lose the ability to "recirculate" data from anywhere in the pipeline. I'm not sure if this is a feature or a bug. Consider a program that calls recirculate(meta) and later in the same pipeline executes meta.f = 0. Which value of meta will be used? If the latter, it would be better in my opinion not to provide the illusion of being able to call recirculate anywhere since its effects take place later.

2. The programmer has to go to the trouble of defining a header type to store the recirculated data, this type needs to be flexible to store all different kinds of data that might be recirculated (maybe unions or overlays help), and most critically, they need to handle it explicitly in the deparser/parser! This is a bit silly since the data is really just being internally routed across the P4 device, and the compiler could very well handle the (de)serialization code automatically, just like it does for intrinsic metadata.

I agree with Calin that if this is an important feature in practical programs, relying on annotations to get a non-standard semantics (if that's what is being proposed) seems like the wrong strategy -- we should add new constructs that make common uses of this use case convenient.

If my analysis is right, could people weigh in on which of the two issues above is most critical? To recap: (i) being able to invoke recirculate and friends from anywhere in a control or (ii) automating the encoding/deparsing/parsing of this data? Or both!

If we understand the requirements, and have some example programs, then I think we can make progress towards a good solution.

And to focus attention on the issues, let's declare v1model and P4_14 translator a non-goal for the moment. We need to fix that too, but let's get the P4_16 design right first.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub< https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fp4lang%2Fp4c%2Fissues%2F1669%23issuecomment-460067799&data=02%7C01%7Cmbudiu%40vmware.com%7Cd5ee401b669f4f40205808d689f795f8%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636848093812160430&sdata=FzxATIigAkVz6110RbMYZF3ffq4KnkDuQ9NxeAE%2F0ug%3D&reserved=0>, or mute the thread< https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASf5sl1GqXkbSO4eaLAYr0OAZlvEL6naks5vJxMhgaJpZM4aDL16&data=02%7C01%7Cmbudiu%40vmware.com%7Cd5ee401b669f4f40205808d689f795f8%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636848093812160430&sdata=ATrOQKGpz4QSJTKtAKXp%2FIP1hHEKc9hcsc2DMqh1kVU%3D&reserved=0

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/p4lang/p4c/issues/1669#issuecomment-460073740, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwi0pdXo4Cn8jqakrzKPEcqR8uJppJ7ks5vJyQwgaJpZM4aDL16 .

[jafingerhut]

Nate, I think you mentioned this during the language design WG meeting earlier today (2019-Feb-04), but wanted to point out that the P4_14 language spec explicitly says that you can do resubmit(meta), but it is not the current values of the fields of meta that are preserved, but the final values those fields have at the end of ingress processing. Thus later assignments during ingress processing to meta.f are kept.

[jafingerhut]

An update on the status of this issue.

The recirculate, resubmit, and clone operations should all work as of the latest versions of p4c and behavioral-model simple_switch, but only if you give them an empty list of metadata fields to preserve.

As stated in this issue and follow-up discussions, p4c and simple_switch will in some cases correctly preserve a non-empty list of fields given to these operations, but due to the way this is implemented, it is an accident that it sometimes works, not by design. Some optimization passes in p4c can make changes to the intermediate form of the code, such that the generated BMv2 JSON compiler output files does not preserve the metadata fields the P4 program specified in the source code.

There has been a fair amount of discussion in the P4 language design work group on the issue, and several possible approaches considered to change the v1model architecture definition so that these operations could be done in a way that consistently preserves such metadata fields. None of them have desirable enough properties that they are considered worth implementing, at least not yet. Someone may still come up with another approach considered good enough, but we do not have one now.

The long term goal is to implement the PSA architecture for P4_16, which uses a different way to specify the preserved metadata for these operations, and also to implement automatic translation of P4_14 programs to P4_16+PSA architecture programs that have the same behavior as the original program, or at least differ only in well-documented ways.

In the mean time, this change will likely be merged into p4c soon: https://github.com/p4lang/p4c/pull/2000

If it is merged, p4c will issue a warning if the source program uses any of these operations with a non-empty list of fields to preserve. There is no such warning if the list of fields to preserve is empty, as that works fine. The output BMv2 JSON files will not be altered by that change. If a particular source P4 program happens to work with today's p4c and simple_switch, they will continue to work after that change, but with the warning printed. If a particular source P4 program happens not to work with today's p4c and simple_switch, it will still not work after that change, and with the warning printed.

Note that only the P4_16 v1model architecture is affected by this. Other P4_16 architectures may choose to implement similar operations in a different way than the v1model architecture does within p4c and simple_switch -- check with your P4 tool vendor.

[jafingerhut]

As one more reference to add for those who may want to learn of proposals considered, here is a write-up of several of the ideas, with code examples, and a discussion of some of the known advantages and disadvantages of each: https://github.com/jafingerhut/p4-guide/blob/master/v1model-special-ops/README-resubmit-examples.md

[vitor-l]

Hello, @jafingerhut!

I'm joining this discussion a little bit late, but I would like to know if it is still necessary to use empty list of metadata fields when using recirculate with clone. If so, how am I supposed to save any data in the P4-16?

[jafingerhut]

@vitor-l The compiler will let you use non-empty lists of metadata fields to preserve on all of the operations resubmit, recirculate, and clone. It will warn that this is not supported, but even while warning, it may just happen to implement it correctly. It might also implement it incorrectly. It is a case by case basis, and the most precise advice I have on telling whether it compiled in a way that will preserve metadata correctly is given in this documentation: https://github.com/p4lang/behavioral-model/blob/master/docs/simple_switch.md#hints-on-distinguishing-when-metadata-is-preserved-correctly

[jafingerhut]

Long term there appear to be some people now actively working to get the open source Portable Switch Architecture (PSA) implementation fully implemented, and it should have a reliable way to preserve metadata for these operations that matches well with the P4_16 language.

[castilhorosa]

Hello guys!

I'm also joining in the discussion a little late.. I'm trying to use the PSA architecture to recirculate packets keeping metadata, according to psa spec, just to avoid the problems aforementioned. When I try to compile the program though, I get the following error: "target bmv2-psa not yet implemented.". Is there any update in fully support PSA on BMv2 since May 2019, as was asked here: http://lists.p4.org/pipermail/p4-dev_lists.p4.org/2019-May/003943.html ?

[jafingerhut]

There has been active work on enhancing the open source PSA implementation over the last several months by several volunteers. The status of some features that are working, with test P4 programs that are passing their tests, can be found here: https://github.com/jafingerhut/p4-guide/tree/master/psa-test-info

Unfortunately, while recirculation, resubmit, unicast, multicast, and ingress-to-egres cloning are all working, they are working only when you do not attempt to preserve user-defined metadata. That feature is still yet to be implemented, and requires changes to the p4c back end code, and likely also to the behavioral-model bmv2 code.

The list of tests I mentioned above should be kept up to date as more features are working, if you want to check there, but feel free to ask again. We will also be making an announcement when a few more features are working, on the p4-dev email list and a few other places, and again when the work has been completed (or at least almost all of them).

[castilhorosa]

Thanks @jafingerhut . I've been trying to find a temporary solution to solve my problem. Once is not possible yet to keep user-defined metadata when recirculating packets in PSA, is it possible to copy what I want into a header, make that header valid, recirculate the packet with such header by emitting it in egress deparser, and then recovery it in ingress parser? Or the only way I can copy data into a header is by using the exctract() function in ingress/egress parser? I'm assuming that P4 allows me to add new headers into packets at runtime. Correct me if I'm wrong. If this naivly approach works, I could use the information from the header and make it invalid afterwards. I was not able to find that information in PSA spec.

[jafingerhut]

@castilhorosa Yes, you should be able with today's PSA implementation to recirculate packets, but add new headers to the packet before it recirculates. Such new headers could be added in the egress deparser. They could alternate be added in the ingress deparser if you wished, as long as you realize they would be visible to your egress parser code, too.

You can also have ingress parser code that does something like this, to parse packets differently when they are recirculated, versus newly arriving packets:

transition select (istd.packet_path) {
    PSA_PacketPath_t.RECIRCULATE: parse_recirculated_packet;
    default: parser_non_recirculated_packet;
}

[mihaibudiu]

It looks like this is one of these wontfix issues.