direct mapped (counters and meters) state when entries are added and removed

[cc10512]

What is the status of direct meters and counters when entries get added and removed?

[jafingerhut]

A couple of thoughts.

When a new table entry is added for a table that has a DirectCounter, I am assuming the corresponding counter state for that entry is initialized to 0. Removing a table entry causes the corresponding counter state to be 'deallocated' and available for reuse, just as the table entry resources are. I am assuming that P4Runtime API attempts to read DirectCounter state will return no useful data, and perhaps an error status, if you attempt to read such state for a table search key that isn't currently in the table (corrections welcome on that point if someone is planning otherwise).

The only subtle point I can think of regarding this is: What if the control plane cares about getting the last value of the counter, including all matches of the corresponding table entry up until the moment it was removed?

If the control plane only cared about getting "a recent value of the counters" before the table entry is removed, it can read the counter state, then remove the table entry, and if there were a few (or millions) of counter updates in between those times, the control plane doesn't care.

If the control plane wants to guarantee that it can atomically delete the table entry and get the most up to date value of the counter up until the instant the table entry was removed, that sounds like it requires some special driver/runtime support to get it right, and perhaps even a special P4Runtime API call that hasn't been defined yet.

For DirectMeter, I doubt the control plane cares about the transient state of the meter (e.g. number of bytes that the meter will color green/yellow before it starts returning red) except for very timing-sensitive debug purposes, and would be a little surprised if someone wants a P4Runtime API to read this transient state. If so, then the DirectCounter "atomic table_delete & read counters" issue above isn't applicable for DirectMeter at all.

There is an issue for DirectMeter where the control plane may want to specify what the default meter parameters, e.g. committed/peak rates and burst sizes, are for a newly added table entry, after which the control plane could change those values after creating the table entry. Perhaps someone might want an operation to atomically "create table entry and initialize meter parameters", but that sounds like something that isn't in P4Runtime API yet, and maybe not even planned.

[antoninbas]

The ability to specify default meter rates for a direct meter when adding an entry is definitely planned, as I mentioned here: https://github.com/p4lang/PI/issues/177. Adding the initial direct resource configuration to the TableEntry message will let the control plane express that it wants this to be done atomically. What is yet TBD is whether a target unable to apply this atomically at the packet level should return an error or "do its best".

As for the "read counters & delete", we could consider adding something like this if someone can provide a real-world use case where it would be valuable. I am a little doubtful that hardware would support that.

[jafingerhut]

I don't know whether anyone cares enough to push for the 'atomic read counters & delete table entry' use case, either. I only mentioned it because without that operation, there doesn't seem to be much required for DirectCounter other than the operations already proposed in the P4Runtime API, i.e. independent table delete and counter read operations.

In case somebody did want such an operation, an implementation technique for that operation could be:

• mark a TCAM/hash-table/etc. table entry in the data plane as 'invalid' before actually deleting it, so it no longer can be matched by any packets
• if necessary, wait a little bit until all 'in flight' counter updates take effect (implementation-specific exactly how long to wait, or to take advantage of any 'flush' operation to guarantee in flight updates have taken effect)
• read the final value of the counter state
• deallocate the TCAM/hash-table/etc. table entry resources, making it available for reuse by a newly added table entry

TCAM/hash-table/etc. valid bits are pretty normal stuff, which is why I suspect it would more typically be a driver software issue than a hardware issue, but wouldn't be surprised if some hardware left out just the right knob to make it all work. Plus it would almost certainly be a lower performance operation than removing a table entry without going through the extra steps.

[antoninbas]

Your implementation technique makes a lot of sense to me. Still pretty sure no one would be interested in an exact count. Would not be very hard to modify P4Runtime to support this, but I do not think it is worth it at this stage.

[cc10512]

@jnfoster where do we stand on this issue?

[jnfoster]

I was happy with what @jafingerhut proposed:

When a new table entry is added for a table that has a DirectCounter, I am assuming the corresponding counter state for that entry is initialized to 0. Removing a table entry causes the corresponding counter state to be 'deallocated' and available for reuse, just as the table entry resources are.

The "read and delete" operation would be cool, and OpenFlow supposedly supported it for counters, although I wonder whether implementations did it on the nose.