What is the intended behavior of the VERIFY_AND_SAVE action of SetForwardingPipelineConfig ?

[jafingerhut]

Version 1.2.0 of the specification says this about the VERIFY_AND_SAVE action of the SetForwardingPipelineConfig message in this section: https://p4.org/p4runtime/spec/v1.2.0/P4Runtime-Spec.html#sec-setforwardingpipelineconfig-rpc

VERIFY_AND_SAVE: saves the config if the P4Runtime target can realize it. The forwarding state in the target is not modified. However, any subsequent Read / Write requests must refer to fields in the new config. Returns an INVALID_ARGUMENT error if the forwarding config is not provided or if the provided config cannot be realized.

Suppose a device currently has P4 program P1 loaded and operating in it. A client has successfully added many table entries into multiple tables, and the device is humming along happily executing program P1 with that configuration.

P1 has table T1 with actions T1A1 and T1A2

Now a client sends SetForwardingPipelineConfig with action VERIFY_AND_SAVE with program P2, where P2 is a program that the device is capable of loading and executing. The server returns success.

From my reading of the text above, the device should still be executing P1, with the same set of table entries as before the SetForwardingPipelineConfig message was processed successfully.

P2 also has table T1, with actions:

• T1A1 that is the same as in P1
• T1A2 is different between P2 and P1 - in P2 it has an additional parameter t1a2_new_param with type bit<8>
• T1A3 is a new action for table T1 that is in P2, but appears nowhere in P1

If a client now attempts to add a table entry T1 with action T1A1, and the device has space to add that new entry, should it immediately be added into the device's table, and thereby change the packet processing behavior of the device?

If a client now attempts to add a table entry to T1 with action T1A3. Should it be successful, because program P2 can add it? Or should it fail because it does not exist in P1? If it succeeds, should the packet processing behavior of the device change in any way?

If a client now attempts to add a table entry to T1 with actin T1A2, should it be an error if no parameter value t1a2_new_param is provided, because program P2 needs that parameter value? If the client does provide a value t1a2_new_param for the action, should the device's packet processing behavior be changed immediately, adding the action with all parameters except for t1a2_new_param?

Because of the questions above, and others I could make up for other kinds of changes between P1 and P2, it seems to me that the job of the server is drastically simplified if, when in this state where it has successfully processed a SetForwardingPipelineConfig message with action VERIFY_AND_SAVE, the server rejects all kinds of run-time configuration messages except another SetForwardingPipelineConfig message, perhaps with restrictions on which kinds of SetForwardingPipelineConfig messages it can succeed on in this state.

If that restriction isn't intended, is there a description somewhere in the spec that answers what to do in all of the example scenarios described above?

[antoninbas]

As soon as the SetForwardingPipelineConfig with action VERIFY_AND_SAVE succeeds, program P1 is out of the picture from a P4Runtime perspective. You can no longer affect packet processing in any way until the COMMIT action. For example, you cannot react to a port down event to update some forwarding table. Every subsequent Write is valid if and only if it is valid for program P2.

The 2-step update (VERIFY_AND_SAVE -> COMMIT) is meant to let you build your new forwarding state (think of it in terms of memory contents) without affecting packet processing. This can be done over several seconds if you need to push a lot of table entries to the P4Runtime server (even if you only have one Write with a bunch of entries). Once you are ready, you perform the COMMIT action and the device should try to "swap" {P1 + entries} with {P2 + entries} as "atomically" as possible. I gave a talk around this feature and its implementation on Tofino in 2018: https://www.youtube.com/watch?v=qlwIlw0dxJk

[jfingerh]

Thanks for that info. I may look over the P4Runtime API spec and see if anything might be changed slightly there to make what you say above more explicit there.

Suppose a client did VERIFY_AND_SAVE, followed by many attempted additions of table entries to the new, but not yet active, P4 program. Suppose the client software wished to operate in a way such that if any of the table additions failed, it wanted to "undo" the VERIFY_AND_SAVE operation, i.e. go back to a state where it could perform operations on the currently running P4 program, and abandon the new one it gave in the VERIFY_AND_SAVE operation. Is that possible using operations in the current P4Runtime API spec?

[antoninbas]

Of course the client can just implement this manually and "undo" its actions using DELETE write RPCs. But I agree it would be convenient to be able to use the VERIFY_AND_SAVE action again to start again. The spec does not say anything about that, so it is possible that some targets implement this behavior already. IIRC correctly, bmv2 will fail with a "swap already in progress" error message. However, it does seem that it should not be too hard to implement (on bmv2 or any other target). If a target only builds state in software in reaction to write RPCs that occur between a VERIFY_AND_SAVE action and a COMMIT action, clearing this state again should not be an issue. If a target starts building state in hardware (e.g. because available memory allows it and there exists some versioning system in the hardware to have multiple datapath versions, but only one active for any given packet), it may be more involved.

So in a nutshell, I think it would be good to have and we should probably propose this change when we resume P4 API WG meetings.

[jafingerhut]

At the 2020-Sep-11 P4 API working group meeting, I volunteered for two things:

(1) Create a PR for the spec that clarifies the currently intended VERIFY_AND_SAVE behavior, which is that it puts the server into a different "state", where write requests and read requests all go to an initially empty state for the new program (that isn't committed to the hardware yet), and those read and write requests do not affect the currently operating data plane at all. It seems that perhaps a simple state diagram might be useful here, even if it only has 2 states, to show all of the ways that the server can change between those states.

I think this current issue has a good title for this, and rather than creating a new issue for it, will use this one, and link to any PR for proposed spec clarifications from here.

(2) Create a separate issue for a proposed new command ABORT, which has been done: https://github.com/p4lang/p4runtime/issues/314

[jafingerhut]

Now that I think of writing the clarification for this issue, I have a question for others about the current intended behavior:

It seems clear that if there is a single client session that does a successful VERIFY_AND_SAVE operation, that all previously sent read and write requests from that same client that haven't completed ought to refer to the currently operating data plane, and be allowed to finish operating on that state, even if they haven't finished yet at the time the server receives the VERIFY_AND_SAVE operation.

Any new read/write requests received from the same client after the server accepts and processes (with a successful result) a VERIFY_AND_SAVE operation, should refer to the new P4 program provided in the VERIFY_AND_SAVE call.

Should that be the case for all clients? That is, if there is a second and third client that did not do the VERIFY_AND_SAVE operation, they need to cooperate with the client performing the VERIFY_AND_SAVE so that they stop doing read/write requests on the current program before the VERIFY_AND_SAVE is sent to the server?

And if they do not stop sending such read/write requests, then after the server processes the VERIFY_AND_SAVE command with successful result, all clients read/write requests will now operate on the new P4 program state (the one not in the data plane yet)?

Or is it the intent that somehow some clients can continue operating on the current P4 program in the data plane, whereas other clients only operate on the new P4 program state that is not in the data plane?

[jafingerhut]

After a successful VERIFY_AND_SAVE operation, is it intended that the server continues sending out digests, timeout notifications, and PacketIn messages to client(s) for the current program in the data plane?

Should the server still accept PacketOut messages from clients for the current data plane?

If yes, then the clients are in a state where they cannot write/read table entries for the current data plane, but they are receiving these other kinds of notifications about the current data plane program?

This question is out of scope for clarifying the existing P4Runtime API spec, I think, since it sounds like a change to me, but did folks consider providing the capability for write and read requests to refer to 'currently operating P4 program' versus 'next P4 program' as the target of the operation? (where 'next P4 program' means 'the P4 program provided by the last successful VERIFY_AND_SAVE operation')

[antoninbas]

So here is my interpretation of the spec:

1. after a successful VERIFY_AND_SAVE, the client will not receive any more messages on the stream, save for those related to arbitration, until after COMMIT: no packet-ins, no digests.
2. after a successful VERIFY_AND_SAVE, the client should not attempt to send messages on the stream, save for those related to arbitration. If a packet-out message is received by the stream, it should be dropped, and an error may be logged / reported to the client.
3. regarding other clients: First only a single client can have write access at a given time. For clients with read-access only, I agree that things are not optimal as: they don't get notified by P4Runtime of the pipeline change (but may be notified by some distributed control plane logic) and their only recourse is periodic GetForwardingPipelineRequest RPCs. One thing that is not covered by the spec I believe is a change in leadership after a VERIFY_AND_SAVE has been initiated. There is no mechanism baked into P4Runtime right now that would enable the new leader to figure out that a pipeline change is ongoing and that a COMMIT is still required. So maybe this is information that GetForwardingPipelineRequest should be able to return to the new leader. Once again it is possible that in practice, there is some distributed data base shared across clients that would be able to provide that information.

The rationale for 1 & 2 is simply that the P4Runtime implementation is expected to support a single P4 pipeline version at any given time. So if that version does not match the one currently being implemented by the device (as is the case between VERIFY_AND_SAVE and COMMIT), things like Packet IO are not possible (metadata handling). I would actually be in favor of supporting operations on both versions of the pipeline, but that's a big decision which would potentially introduce more complexity, so I think it requires a discussion across both ends of the spectrum (people who write controllers and vendors who need to implement this support).

[adibrastegarnia]

Any plans to clarify the state changes for  SetForwardingPipelineConfig RPC? It is not clear to me what happens if control plane calls VERIFY_AND_SAVE and then calls VERIFY_AND_COMMIT rather than COMMIT (Is target just accepting COMMIT when the previous action is VERIFY_AND_SAVE?)